Repository navigation
Draft: agent-oriented test selection and automatic parallel coordination - #77
Draft
simulieren wants to merge 32 commits into
Draft
simulieren wants to merge 32 commits into
simulieren wants to merge 32 commits into
Conversation
Experimental. Adds an opt-in loop for running only the tests a change can affect, with machine-readable output for agents and CI. - Selection caches: --prepare-caches records which tests live in which file and which project modules each test file imports. Filters prune files before import. - Change selection: --changed (or --changed=auto from git) runs only test files whose imports reach the edited paths, with a full-suite fallback. - Agent reporting: --agent / --agent-file emit bounded JSON lines with locations, compaction, selection metadata and timings. - japa-agent: prepares missing caches, runs changes, writes a delta and loop metrics. - Baselines: --allow-known-failures and --update-baseline; the coordinator merges worker failures once. - Automatic parallel coordination and timing history, with cache locking and --workers=auto for narrow selections. Runs against the published @japa/core 10.4.0. Runner version is unchanged. Checks: 109/109 tests, 176/176 core tests in the fork, lint and typecheck.
- japa-agent now resolves --cache-dir from either its own options or the forwarded command and uses it for cache detection, preparation and metrics. Previously it always checked the default directory and could prepare on every run. - Remove the process-wide JAPA_SPEED_COLLECT_FAILURES mutation; the flag is passed per worker, so a later runParallel call does not inherit it. - Remove unused overheadFor(). - Add AGENT_LOOP.md covering flags, exit codes, JSON output, AdonisJS usage, safety rules and limits; link it from the README. - Drop duplicated runner tests uncovered during the port (109 -> 103). Checks: 103/103 tests, lint (targeted), typecheck, tsdown and declarations.
…igure AdonisJS resolves providers, preloads and config before configure() runs, so the recorder never saw those modules' own imports and the dependency map could never be marked complete. --changed then always ran the full suite. recordImportsFromStart() registers the resolve hook early; recordDependencies reuses its edges instead of registering a second hook.
…eteness Type-only imports and specifiers inside JSDoc never reach Node's resolver, so files that had them were reported as unrecorded and the dependency map stayed incomplete on TypeScript apps.
--agent-file writes agent JSON to a file and leaves stdout alone, so there is no mixed-output reason to drop the app's reporters. Apps that write their own result files or run guard reporters keep them; --agent still replaces.
A worker that died mid-run was reported as 'Unable to collect worker timings; use this runner fork', and its exit code was overwritten with 1. Name the exit code or signal instead, and keep the misconfiguration hint for reports that exist but cannot be parsed.
Each entry stored a size/mtime stamp for every file its spec imported. On an app whose specs reach ~1,400 modules each, the index grew past 100 MB, every write stat-ed each import once per spec, and six parallel workers queued on the cache lock to rewrite it. Workers outlived the app's teardown exit timer and died before writing their coordinator report. Entries now keep one digest of their imports' stamps, computed from the dependency map with each path stat-ed once per write. Parallel workers skip the index write, as they already skip the dependency map.
The runner switches its default reporter to dot when isRunningInAIAgent() is true, so the config and planner suites failed whenever an agent ran them.
japa-agent now writes its report through --agent-file only, so the command's configured reporters still run. JAPA_WORKERS sets the worker count for package scripts, whose extra arguments reach the test command instead.
--changed used to fall back to the full suite silently. It now says why, worded as the fix (no map yet, an incomplete map and recordImportsFromStart(), paths no spec imports, a Node without module.registerHooks), and the agent summary carries it as selection.fallbackReason. --prepare-caches warns when modules were imported before recording started, naming examples. japa-parallel warns about more workers than CPUs, and a missing worker report names the two causes seen in practice: a teardown process.exit() timer and an app shutdown waiting on an unreleased connection.
A timed-out test that kept a transaction open was the cause in the app; the teardown exit timer is a possible one, never observed. Order the message and the docs that way.
…ze, run lock The dependency walk no longer passes through packages. A framework package loads most of an app at boot, so following those edges made every test depend on every booted file and --changed always selected the whole suite. Project files a package loads before the first test file are recorded as the boot set; a change to one that no test reaches still runs everything. selection.resolvers add files a test reaches without importing them (a controller behind a route). selection.heuristic, off by default, guesses tests by file name and prefix rules for paths nothing reaches, and the summary marks the run as heuristic. japa-agent and --workers=auto size the pool from recorded durations of the planned files, so a small selection stays in one process. Concurrent pools in one project now queue on a run lock instead of sharing worker resources. The map format moves to version 2; version 1 maps are discarded.
Naming suites counted as a partial run, so an app with a browser suite could never record a baseline from 'unit functional'. Entries of suites that did not run now stay as they are, and the coordinator says why it skipped a write. Also documents the selection rules, resolvers, heuristic and run lock.
The wrapper's new= count compares with the previous report, so a wide run after a narrow one printed new=87 while the summary said one new failure against the baseline. Print both, labelled.
A new spec changed alongside a heuristic match was dropped: the name guess picked a recorded spec with the same stem, and the selection plan then kept only listed files. A changed spec the map has not recorded now selects itself (no full-suite fallback), and the plan restricts only recorded files, so a spec pulled since the map was written still runs. Adds fast-check as a dev dependency for a property test: over random graphs, boot sets and change sets, a narrowed selection always contains every test that reaches a changed path, and an exact one contains nothing else.
Selection could skip an affected test: - selection.global: path prefixes whose change runs everything even when a test imports them (config, bootstrap, route files). - Files a package loads after boot and no resolver names are recorded as bridges (map version 3); a change to one runs everything. - A resolver that throws warns and marks the map incomplete. - Only a complete run decides completeness, so it can be revoked. - heuristic and global come from the current config; parallel workers whose config differs from the map's policy run their whole shard. - Parallel runs keep test files the map never recorded, like serial runs. - The completeness check counts `#` subpath imports. Verdicts could read green or promote data wrongly: - The timing listener no longer swallows an uncaught error before Japa's exceptions manager watches (global setup); a late error withdraws the worker report. - Merged agent summaries need a summary from every worker, keep processError, and stay knownOnly when another shard passed. - --failed-new skips a title only when every failure recorded under it is a baseline failure of the same file; the retry cache stores file + title. - The selection signature includes title, group and tag filters. - japa-agent pool runs forward --update-baseline and say when it was skipped. - Auto worker sizing prices only the named suites. - A run lock wait is abortable, and an empty lock file left by a dead process is recovered.
TypeScript erases a value import used only as a type, and alias imports are where that is common: five such files in an AdonisJS app marked its whole map incomplete, so --changed always ran the full suite.
A shared dependency was re-read and re-scanned for every test file that reaches it: 277k visits over 3.2k unique files on a large app. Memoize the per-file result for the duration of one write.
…s alone on a pool - japa-agent writes the command's output to <report>.log and prints a digest on stderr: outcome, counts, selection, and the failures to act on with message and line. --verbose streams the output as before. - The agent summary's baseline lists up to 20 newFailures; merged parallel summaries carry them and report the slowest worker's durationMs. - The parallel coordinator has no config, so changed paths the map never saw are handed to workers as pending instead of forcing the full suite. A worker runs its whole shard only when one is not a configured test file.
git diff --name-only prints repository-root paths, so in a monorepo every modified file under the app came through as apps/web/... and matched nothing, and --changed=auto ran the full suite. --relative scopes the diff to the cwd, matching what ls-files --others already printed.
An automatic pool could take 8 workers on a 12-core machine, each booting the app, on top of the database those workers share. JAPA_MAX_WORKERS caps the automatic count without forcing a pool for small selections the way JAPA_WORKERS does. Workers now start at nice 10, so the rest of the machine stays responsive. The digest log test relied on npm test's NODE_DEBUG to give the log content; it now sets it itself.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Experimental branch against
5.x. Draft, not proposed for merge as-is. No package has been published.This PR contains only
@japa/runnercode and tests. It also carries the automatic parallel coordinator and timing history those features build on, because upstream5.xdoes not have them.@japa/coreis the published 10.4.0; no core fork is included. It adds an opt-in path for running only the tests a change can affect, with machine-readable output for agents and CI. Default behaviour is unchanged.--prepare-cachesrecords file contents;--tests,--groups,--tagsprune files before import--changed/--changed=autoruns only affected test files: imports, a resolver hook, an opt-in labelled heuristic, a full-suite fallback--cache-dirkeeps both caches in a tracked directory--allow-known-failuresexits 0 for known-only failures;--update-baselinerecords them--agent/--agent-fileemit bounded JSON lines with locations and timingsjapa-agentprepares caches, runs changes, logs the command's output and prints a short digest of what to fix--workers=autoandjapa-agentsize the pool from recorded durations, capped byJAPA_MAX_WORKERS; workers run at lower priority; concurrent pools queue on a run lockHow it works
flowchart TD A["Agent edits code"] --> B["japa-agent"] B --> C{"Caches present?"} C -- "no" --> D["--prepare-caches<br/>import files, write caches"] C -- "yes" --> E["--changed=auto"] D --> E E --> F{"Changed paths known?"} F -- "no" --> G["Full suite"] F -- "yes" --> H["Select dependent test files"] H --> I["Planner prunes files that cannot match"] I --> J["Import selected files"] G --> J J --> K["Run tests"] K --> L["Agent JSON + delta + metrics"] L --> M{"Only known failures?"} M -- "yes" --> N["exit 0"] M -- "no" --> O["exit 1"] K -. "setup or planning error" .-> P["exit 2"]The two caches answer different questions: the selection index knows what a file
contains, and the dependency map knows what a file imports.
flowchart LR subgraph prepare["--prepare-caches"] A["Import every test file"] --> B["Selection index<br/>file → titles, groups, tags"] A --> C["Dependency map<br/>test file → transitive imports"] end subgraph filter["Filtered or changed run"] D["--tests / --groups / --tags"] --> E{"Index fresh?"} E -- "yes" --> F["Skip files that cannot match"] E -- "no" --> G["Always import the file"] H["--changed"] --> I{"Path in dependency map?"} I -- "yes" --> J["Select dependent files"] I -- "no" --> K["Run the full suite"] end B --> E C --> IUsage
Usage with AdonisJS
AdonisJS runs Japa through
Ignitor.testRunner()inbin/test.ts. One line is required:Adonis loads providers, preloads and config before
configure(), so without it thedependency map never completes and
--changedalways runs the full suite (Japa now warnsabout this, see Warnings).
Ace forwards the flags as they are, no
--needed:An automatic pool takes one worker per ~10 s of recorded test time, up to 8 (or
JAPA_MAX_WORKERS). The ceiling keeps the small-selection behaviour: a fixedJAPA_WORKERSstarts a pool even for one file,JAPA_MAX_WORKERSstill runs it in oneprocess. Workers start at nice 10, so a pool yields the CPU to an editor or a dev server.
Prefer
--agent-fileto--agentin an app with its own reporters (result files, guards):--agentreplaces every reporter so stdout is pure JSON,--agent-fileadds the agentreporter beside them.
Parallel isolation
Japa supplies
JAPA_WORKER_IDandJAPA_WORKER_COUNT. The app owns database, Redis,search and port isolation. Set them in
bin/test.tsbefore the Ignitor boots: Adonis neveroverwrites a variable already in
process.env, and by the time a bootstrap file runs theenv service has already read
.env.What broke a real app under
japa-parallel, and how it showed up:app.terminate()waits in the database provider for the open transaction; the worker exits without a report (observed, traced to the query)setTimeout(process.exit, 1000)run()resolvesserver.ws: falseunder testmail.fake(), sinon stubs, fake clocks left behindsetTimeoutfor 6 s in a lease test)--prepare-cachesimports files without running tests and skips global setup hooks, so itis safe to run before a database reset.
Warnings
Wrong or missing setup no longer fails silently. Japa writes these to stderr, so agent
JSON on stdout stays parseable, and the agent summary carries the same text as
selection.fallbackReason.--changed is running the full suite: no dependency map yet--prepare-caches... the dependency map is incompleterecordImportsFromStart()at the top of the entrypoint, then--prepare-caches... N changed path(s) no test file imports (a, b, c)selection.heuristic. A new spec selects itself and does not cause this... this Node has no module.registerHooksThe dependency map is incomplete: N file(s) imported other files before Japa could record them--prepare-caches; same fix[japa-parallel] N workers on M CPUs--workers=autoworker N: exited with code C before writing its reportTrying it in another repository
The packages are not published, so install both tarballs into a disposable
checkout. The runner depends on the unpublished core prerelease; installing the
runner alone will not resolve it.
The runner resolves the published
@japa/core@10.4.0; no core workspace tarball isneeded for this branch.
Then prime the caches once:
Try the loop:
Verify that selection is honest before trusting it: run the same filter once with
the index and once with
JAPA_NO_SELECTION_INDEX=1. The test identities and exitcodes must match. The agent summary reports
selection.fallback: truewhen afilter could not narrow the import list.
Restore the app afterwards with
npm ci(or whatever restores its lockfile), whichremoves the
--no-savepackages.Troubleshooting
--changedneedsmodule.registerHooks; on older Node it safely falls back tothe full suite.
--testsand--prepare-cachesstill work.--cache-dir(ornode_modules/.cache/japa) and are safe todelete.
from the app; Japa only supplies
JAPA_WORKER_IDandJAPA_WORKER_COUNT.--update-baselineonly writes from a full run. Filtered runs warn and leavethe file unchanged.
Safety rules
spec the map has not recorded selects itself instead of widening the run.
--changedfalls back to the full suite for any unknown path.Benchmarks
Correctness was checked before timing.
Environment
Apple M2 Max, 12 logical CPUs, 96 GiB RAM, Node 24.19.0. Docker VM limited to 12
CPUs and 7.65 GiB. Application boot runs through
@poppinss/ts-exec; the app isa large production-style AdonisJS application with a known-failure baseline.
Runs are sequential. Where a range is shown it is separate runs, not a median
unless stated. The machine was shared, so absolute wall times moved by several
seconds between sessions.
Full suite
The six-worker median is 27.768 s, about 8.9× faster than the serial
reference and about 1.7× faster than the previous retained four-worker run.
The three six-worker runs were consecutive and complete. The 8-worker run was
rejected as an improvement: wall time tracked host load, not worker count.
A later session measured ~48–49 s for both this branch and unmodified code, with
load averages of 4–10. That session is labelled load-affected, not a regression.
Worker ceiling
Every worker boots the app and opens its own database connection pool, so past a
point more workers add CPU, not speed. One sample each, unit suite only, 8,595 tests,
same failures:
JAPA_MAX_WORKERS=4)The app now sets the ceiling to 4 in its scripts. The 8-worker run also kept the
Docker VM serving Postgres at about 3 cores.
Preparing caches
--prepare-caches, before skipping setup--prepare-caches, skipping setupPreparation executes no tests. The cost is roughly 4.8 s of boot plus a 3.3 s
import pass; application setup was only ~0.6 s with a prepared database.
Selecting one test
A second pair measured 3,342 ms vs 860 ms of imports and 8.08 s vs 5.83 s of wall
time. The four planned files are the matching file plus three files that build
titles at run time and therefore cannot be pruned by a title filter.
Wall time still includes roughly 4.8–4.9 s of application boot. The import phase
is the stable, load-independent comparison.
Selecting by change
--filesFailure baseline
--update-baseline, serial--allow-known-failuresThe 6-worker update included four load-sensitive failures in addition to the
stable baseline. Each passed in isolation; they are treated as flakes, not
regressions.
Boot attribution
bootstrap-loaded, default loaderbootstrap-loaded,ts-execbootstrap-loaded,tsxNODE_COMPILE_CACHEwarmmakeSyncRequest)compileSourceTextModule)load()timeThe loader path, not provider boot or TypeScript transformation, dominates. The
tsxresult is a lead only: direct-loader samples were noisy (4.5–6.0 s) and thefull identity suite was not run under it, so it is not adopted here.
Correctness and regression checks
Limitations
tsxloader has not passed the full identity suite.Cross-app validation
Tested against three open-source AdonisJS apps. Each needed one migration change
because Japa 5 uses Node's
fs.glob:batosai/adonis-starter-kitoguzhankrcb/OTIadocasts/adocastsFeature checks on those apps:
--prepare-caches--force-exit--tests--changed=<path>--changed=auto--agent-filesummaryjapa-agentwrapper + delta--update-baseline/--allow-known-failures--failed --failed-newThe Adocasts failures were placeholder SMTP/service settings, not selection or
reporting. Browser suites were not run because Playwright browsers were not
installed.
Full functional suite
The complete functional suite: 431 files, 5,038 tests. Every run collected
all 5,038 identities with no missing or duplicate tests.
Eight workers was the fastest setting, roughly 2.6–3.1× faster than serial,
with noticeable run-to-run variance.
Failure comparison is noisy because this suite is not green in the lab even
serially (125 failures). Across the three 8-worker runs:
three.
The failure differences are consistent with load/order-sensitive tests, not a
selection or reporting mismatch. Identities were identical in every run.
Node versions
@poppinss/ts-execneedsregisterHooks(22.15+)The runner's own fallback was checked on 22.14 with the compiled build:
--changed=autoruns without crashing whenregisterHooksis absent and fallsback to the full suite. Practical floor for TypeScript/Adonis apps is Node
22.15; Node 24 is recommended.
Precise
--changedselection (map version 2)Measured in the AdonisJS app below, the version 1 map made
--changeduseless: everyreachable source file selected all 1,079 test files. The dependency walk followed edges
into framework packages, and the framework loads most of the app at boot (routes,
providers, preloads, every module). Every test that imports
@adonisjs/coretherefore"depended" on everything.
selection.resolversselection.heuristiclease.ts->lease.spec.ts) and by prefix rules. The summary says"heuristic": true.Selection per changed file, 2,400 app source files and 1,110 test files:
The rest are env files and core models that most tests really do import, plus boot code.
A shared test-helper barrel still widens some files: following only the re-exports a
test imports would cut the p90 from 748 to 435 test files but leave the median at 15,
so it is not in this PR.
Pool sizing and run lock
--workers=autoandjapa-agentsize the pool from recorded durations of the plannedfiles: one worker per ~10 s of test time, at most 8. A small selection stays in one
process instead of paying for extra application boots.
japa-agent --workers=auto|N(default auto): one worker runs the command directly asbefore, more go through the coordinator.
japa-parallelin the same project waits for the first (withRunLock, stalewhen the owning process is gone). Worker N of two pools would otherwise share the app's
database N.
JAPA_PARALLEL_LOCK=0opts out.--update-baselinefrom a run of named suites (unit functionalin an app that alsohas
browser) now updates those suites' entries and leaves the others alone. Before,naming suites counted as a partial run and the baseline could never be written. The
coordinator now says why it skipped a write.
japa-agentno longer mixes the previous-report delta into its output; the digestcounts against the baseline (see below).
Fixes from running this in a large AdonisJS app
Integrating the branch into the app the benchmarks came from turned up these. Each has a
test that fails without the change.
--changedalways ran the full suite on AdonisJS: providers and preloads load beforeconfigure(), so their imports were never recorded and the map never completedrecordImportsFromStart(), called at the top of the test entrypoint, registers the resolve hook earlyimport typeand specifiers inside JSDoc made recorded files look unrecorded--agent-filedropped every configured reporter, so app result files and guard reporters stopped silently--agentreplaces reporters;--agent-fileadds the agent reporter.japa-agentnow uses--agent-filealoneconfig_managerandplannerspecs failed under any AI agent, because the default reporter becomesdotisRunningInAIAgent()npm run x -- --workers=6reaches the test command, not the coordinatorJAPA_WORKERSenv var forjapa-parallel--changedfell back to the full suite with no explanation, and broken setups looked like a slow but working loopselection.fallbackReasonin the agent summaryAdonisJS entrypoint:
Controllers reached only through lazy route imports were the open case here; the
selection.resolvershook above closes it (a controller change selects the specs thatrequest its routes).
Result in that app
Node 24.19, one 12-core machine, after the fixes above. Single runs, not medians.
node ace test unit)--workers=auto(8)Unit is stable in parallel. The functional suite's parallel-only failures vary from run to
run (10 to 33), and the causes found so far are in the application, not the runner: one
Mailpit inbox that every worker clears, Redis pub/sub channels (a cache bus, Transmit) that
ignore the per-worker Redis database, and tests that rely on rows accumulated in the
long-lived serial database.
--allow-known-failureswith a recorded baseline keeps thosefrom reading as new failures.
A typical
--changedselection now runs about 14 test files (median over 2,246 source files, withselection.globalset).Fixes from an external review
A review of the branch reported 15 findings. Each fix below has a test that fails
without it, except the last row, which has no reliable reproduction.
selection.global: path prefixes whose change runs everything (config, bootstrap, route files). Treating every boot file this way would have run the full suite for 1,271 of 1,522 boot files in the app above, most of them services a module loads at bootdebugand the map stayed completelatest.complete === true || ...)selection.heuristicwas read from the map, not the config--changedstill dropped test files the map never recordedprocessErrorwas dropped,knownOnlywas lost when another shard passedokneeds a summary from every worker,processErroris kept,knownOnlymeans every failing shard failed on known tests--failed-newmatched baseline entries by title alone--testsfilter could report a false "fixed"japa-agenton a pool dropped--update-baselineTwo findings were partly declined. Counting
#alias imports in the completeness checkflagged five files in the app whose value imports TypeScript erased (used only as types)
and made the whole map incomplete, so it was reverted and documented as a limit. Imports
inside test bodies are recorded by full runs only;
--prepare-cachesdoes not run tests.Agent output
Measured on the AdonisJS app above with a throwaway failing unit spec. Before these
changes
test:agentwrote 2.6 MB (36,717 lines) to the terminal: ANSI colours, eachworker's migrator and seeder logs, and Japa's own stack frames. An agent that reads its
command output paid for all of it.
japa-agentsends the command's output to<report>.logand prints a digest on stderr.--verbosestreams the output as beforebaseline.newFailureslists up to 20 failures the baseline does not know, merged across workersdurationMsis the slowest worker's, not the sumThree selection bugs turned up while measuring it, each with a test:
pending; a worker runs its whole shard only when one is not a configured test file. The same probe now runs 1 file in 8 sgit diff --name-onlyprints repository-root paths (apps/web/app/x.ts) whilels-files --othersprints cwd-relative ones, so--changed=autoran everything for any editgit diff --relative. Changes outside the app directory are no longer seen--prepare-cachesspent about 3.3 s per process in the completeness check, re-reading a shared file once per test that reached it (277,046 reads of 3,193 files)--prepare-cachesin the app went from 19 s to about 15 s; the map is still completeKnown limits
module.registerHooks. On older Node the map is unavailable and--changedruns the full suite.@poppinss/ts-exectransforms on the off-thread loader, and the per-import round trip took 2.2 s. An in-threadmodule.registerHooksloader with a content-hash cache took about 1.1 s off each process. That is an app-side change, not part of this PR.tsxloader showed a promising bootstrap reduction but has not been validated across the full suite.package.jsonstill advertises Node >=18.16. Withts-exec, the practical floor is 22.15; aligningenginesis a release decision.Checks
npm run lintnpm run typechecknpm run buildnpm testandquick:test), including a 2,000-run fast-check property for--changed(globals, bridges, policy)Scope
Runner source and tests only. No application source, credentials, benchmark experiment logs, lab harnesses, planning documents, or private artifacts are included. The branch is experimental and expected to be split or reworked before review.