Repository navigation
tests: gate standalone smokes in CI; fix or delete the 44 failing on main #3649
Description
Problem
Homeboy CI runs PHPUnit only. The 373 standalone tests/*-smoke.php files are never executed in CI, so they rot silently. On main at dcad49078 (v0.195.0), 44 of 373 fail (329 pass).
This is the shared cause of a long tail of one-off "repair the X smoke" issues. Fixing them one by one does not stop new ones from appearing.
1.0 requirement
Every standalone smoke either:
- passes and runs as a CI gate, or
- is deleted because PHPUnit already covers the contract, or is folded into PHPUnit.
The WordPress.org submission checklist requires all gates to be green with no skipped checks, so this blocks #2424.
Failing on main (exit code, first failure line)
Reproduce: for f in tests/*-smoke.php; do php "$f" >/dev/null 2>&1 || echo "$f"; done (PHP 8.5, after composer install).
| Smoke | Exit | First failure |
|---|---|---|
action-scheduler-group-registration-smoke |
1 | FAIL: drain defensively ensures AS group before claiming |
agent-bundle-portable-update-smoke |
1 | FAIL: existing stale flow payload normalizes step key |
agent-bundle-upgrade-planner-smoke |
255 | PHP Fatal error: Uncaught Error: Call to undefined function DataMachine\Engine\AI\Actions\add_option() in inc/Engine/AI/Actions/PendingActi |
agents-api-persisted-agent-registry-smoke |
1 | FAIL persisted rows are visible through wp_get_agents() |
ai-completion-assertion-packet-smoke |
255 | PHP Fatal error: Uncaught Error: Undefined constant "DataMachine\Core\Steps\AI\DAY_IN_SECONDS" in tests/ai-completion-assertion-packet-smok |
ai-error-status-propagation-smoke |
255 | PHP Fatal error: Uncaught Error: Call to undefined function DataMachine\Core\Steps\apply_filters() in inc/Core/Steps/StepTypeMetadata.php:2 |
ai-required-handlers-smoke |
1 | no failure line; non-zero exit |
bootstrap-service-provider-composition-smoke |
1 | FAIL: runtime order preserves \DataMachine\Api\Email::register(); |
bulk-create-pipeline-workflow-smoke |
1 | FAILED: 2 bulk workflow assertions failed. |
command-registry-canonical-roots-smoke |
1 | FAIL: provider does not register exactly 31 commands. |
composable-file-atomic-write-smoke |
1 | FAIL: defined FS_CHMOD_FILE child failed. |
content-format-abilities-smoke |
255 | PHP Fatal error: Uncaught Error: Call to undefined function DataMachine\Abilities\Content\wp_slash() in inc/Abilities/Content/UpsertPostAbi |
email-multi-inbox-forward-smoke |
1 | no failure line; non-zero exit |
exclude-keywords-smoke |
255 | PHP Fatal error: Uncaught Error: Trait "DataMachine\Abilities\Fetch\FetchHttpGetTrait" not found in inc/Abilities/Fetch/FetchRssAbility.php |
external-login-primitives-smoke |
255 | PHP Fatal error: Uncaught Error: Class "DataMachine\Core\Database\Chat\Chat" not found in inc/Core/Auth/PrincipalSessionPromoter.php:44 |
fetch-no-content-log-severity-smoke |
255 | PHP Fatal error: Uncaught Error: Call to undefined function DataMachine\Core\Steps\apply_filters() in inc/Core/Steps/StepTypeMetadata.php:3 |
flow-config-cli-round-trip-smoke |
255 | PHP Fatal error: Uncaught Error: Class "DataMachine\Core\AbilityResult" not found in inc/Cli/Commands/Flows/FlowsCommand.php:1160 |
flow-formatter-loaded-settings-display-smoke |
255 | PHP Fatal error: Uncaught Error: Class "DataMachine\Abilities\AbilityRegistration" not found in inc/Abilities/HandlerAbilities.php:84 |
flow-update-handler-config-params-smoke |
255 | PHP Fatal error: Uncaught InvalidArgumentException: task_type is required. in inc/Core/Steps/Settings/SettingsHandler.php:67 |
job-outcome-metrics-smoke |
255 | PHP Fatal error: Uncaught Error: Class "AgentsAPI\AI\WP_Agent_Run_Result_Envelope" not found in inc/Core/RunResultEnvelope.php:117 |
media-diagnostics-core-boundary-smoke |
255 | PHP Fatal error: Uncaught Error: Class "DataMachine\Engine\AI\System\Tasks\JobReaperTask" not found in inc/Engine/AI/System/SystemAgentServ |
named-mailbox-security-contract-smoke |
1 | FAIL: every email ability input schema advertises the auth_ref mailbox selector |
packet-reconciliation-transaction-smoke |
255 | PHP Fatal error: Uncaught Error: Class "DataMachine\Core\Steps\StepTypeMetadata" not found in inc/Engine/Actions/Handlers/StepLifecycleHand |
phpunit-file-isolation-smoke |
1 | PHP Warning: proc_open(): posix_spawn() failed: No such file or directory in tests/phpunit-file-isolation-smoke.php on line 20 |
poster-template-smoke |
1 | PHP Warning: imagettfbbox(): Could not find/open font in inc/Abilities/Media/GDRenderer.php on line 791 |
processed-items-revision-conventions-smoke |
1 | [FAIL] processed-items docs include revision-key conventions |
react-queue-mode-contract-smoke |
1 | FAIL: PUT URL targets /queue/mode |
recover-stuck-cli-output-smoke |
1 | [FAIL] summary computes actionable total |
retention-engine-data-shedding-smoke |
255 | [FAIL] Jobs promotes handler_slug column + populates it from engine_data |
run-artifact-egress-policy-smoke |
255 | PHP Fatal error: Uncaught Error: Class "DataMachine\Engine\Agents\AgentSubagentGraph" not found in inc/Engine/Bundle/AgentBundleManifest.ph |
runtime-task-ability-contract-smoke |
1 | FAIL: runtime task ability file is loaded |
runtime-vocabulary-smoke |
1 | FAIL runtime source uses host-neutral vocabulary |
setup-schema-runtime-smoke |
255 | PHP Fatal error: Uncaught Error: Call to undefined function delete_option() in inc/setup/schema.php:139 |
step-exception-failure-contract-smoke |
255 | PHP Fatal error: Uncaught Error: Call to undefined function DataMachine\Core\wp_json_encode() in inc/Core/StepResult.php:126 |
step-execution-result-error-smoke |
255 | PHP Fatal error: Uncaught Error: Call to undefined function DataMachine\Core\wp_json_encode() in inc/Core/StepResult.php:126 |
successful-child-routing-smoke |
255 | PHP Fatal error: Uncaught Error: Call to undefined function DataMachine\Core\Steps\apply_filters() in inc/Core/Steps/StepTypeMetadata.php:2 |
test-handler-raw-output-smoke |
255 | PHP Warning: require_once(/var/www/extrachill.com/wp-includes/class-wp-error.php): Failed to open stream: No such file or directory in test |
tool-source-registry-smoke |
255 | PHP Fatal error: Uncaught Error: Class "DataMachine\Engine\AI\Tools\HostToolPolicy" not found in inc/Engine/AI/Tools/ToolSourceRegistry.php |
upsert-handler-result-handoff-smoke |
255 | PHP Fatal error: Uncaught Error: Class "WP_Abilities_Registry" not found in inc/Engine/AI/Tools/Sources/AbilityToolSource.php:69 |
webhook-custom-verifier-mode-smoke |
255 | PHP Fatal error: Uncaught Error: Class "DataMachine\Abilities\AbilityRegistration" not found in inc/Abilities/Flow/WebhookTriggerAbility.ph |
webhook-gate-single-winner-smoke |
255 | PHP Warning: Undefined array key "status" in tests/webhook-gate-single-winner-smoke.php on line 174 |
workflow-persistent-install-smoke |
1 | FAILED: 9 workflow persistent install assertions failed. |
workflow-spec-contract-smoke |
255 | PHP Fatal error: Uncaught Error: Class "DataMachine\Abilities\AbilityRegistration" not found in inc/Abilities/StepTypeAbilities.php:52 |
wp-ai-client-request-timeout-smoke |
1 | FAIL: RequestBuilder dispatch succeeds with wp-ai-client test double |
Supersedes
Per-smoke repair issues whose smokes still fail on main are closed in favor of this one: #2908, #3145, #3010, #3201, #3279, #3320, #3387, #3422, #3446, #3610, #3648.
AI disclosure: Claude (via the Pi coding agent in a Roadie session) ran every standalone smoke on a clean origin/main worktree, collected the results, and drafted this issue at Chris's direction.
Activity
chubes4 commentedon Oct 25, 2026
#3669 (merged as 5c189ea) repaired 4 of these:
action-scheduler-group-registration,agent-bundle-upgrade-planner,ai-completion-assertion-packet, andai-error-status-propagation. All 4 had stale stubs. After that merge, CI shards still report ~40 failing smokes on main. Thehomeboy-test / Aggregatejob also fails on every PR withInvalid Test shard evidence: shard-1 sidecar provenance does not match its immutable manifest, separately from the smoke failures.chubes4 commentedon Oct 25, 2026
Re-ran every smoke on main
5003c4d(v0.198.2): 42 of 371 still fail. I split them by root cause and every failing smoke falls under one of these issues:require_oncelists (13)CI aggregation is tracked separately in Extra-Chill/homeboy-action#552.
Cooking all 6 as cook-batch
wf-cook-batch-e2eb961550af. Each one is gated on: both audits pass, no smoke that passes on main starts failing, and at least one baseline smoke gets fixed.chubes4 commentedon Oct 25, 2026
Status: all 42 failing smokes now have PRs: #3676 (#3670), #3677 (#3671), #3678 (#3672), #3679 (#3673), #3680 (#3674), #3681 (#3675). None of them touches the same file as another, and none adds a failure. Each was verified against the controller and lab baselines.
The red
Aggregatejob is a consequence, not a separate bug. Shards with failing smokes writeinvalid_evidence(Extra-Chill/homeboy-extensions#2978), and the aggregator then reports it misleadingly (Extra-Chill/homeboy-action#552). Once these PRs land, the smokes pass and that path stops triggering.test: load standalone smokes through composer autoload via shared smo…
test: add missing WordPress stubs to shared smoke-wp-stubs (#3677)
test: update workflow/flow smokes to the step_type workflow contract (#…
test: derive bootstrap/runtime contract smokes from current service c…
test: make host-dependent smokes hermetic and resolve toolchain from …
test: repair email, queue, retention and docs contract smokes (#3680)
test: fix two smokes broken by combining the #3649 repair PRs
test: fix two smokes broken by combining the #3649 repair PRs (#3682)
chubes4 commentedon Oct 26, 2026
#3682 merged. Every smoke regression from this sweep is fixed. The remaining CI-only failures (PHP 8.2 artifact perms in
job-artifact-hashable, plus 1 pre-existing unnamed PHPUnit test) are tracked in #3683.chubes4 commentedon Oct 26, 2026
#3684 merged, which closes #3683. Its CI run was the first fully green data-machine Test run (all 4 shards plus Aggregate). Every failure from this sweep (42 smokes on main, plus the CI-only PHP 8.2 and PHPUnit failures) is now fixed.
chubes4 commentedon Oct 26, 2026
Closing: CI already runs every standalone smoke in the shards. Both halves of this issue are done, and the last two failures were fixed in #3684, so data-machine Test is fully green.