Skip to content

⚡ Async file access, concurrent discovery, and coverage gaps - #23

Merged
mkarlesky merged 3 commits into
next_versionfrom
perf/async-coverage
Oct 23, 2026
Merged

mkarlesky merged 3 commits into
next_versionfrom
perf/async-coverage

Conversation

@mkarlesky

@mkarlesky mkarlesky commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Third PR in the lint-and-coverage series, stacked on #22. It covers two parts of the triage report:

  • the performance findings (async file access, concurrent discovery), which also fix two concurrency bugs found by stress testing;
  • unit tests for the uncovered behavior in the pure modules.

Adapter extraction (PR D) and fake-Ceedling integration tests (PR E) follow separately.

Async file access

  • Discovery, configuration, watcher, and report paths no longer block the extension host thread.
  • EngineFileSystem is fully async.
  • projectsFromSettings() and configurationFiles() return promises.
  • The no-restricted-properties warnings on sync fs in src/ drop to zero, apart from the documented /proc reads in ceedlingProcess.ts.

Concurrent discovery

  • Projects load concurrently across directories and one at a time within a directory (mapGrouped in the new src/concurrency.ts).
  • Load outcomes apply in settings order, so error messages don't depend on timing.
  • Listings run concurrently across directories, and test sources are read concurrently.
  • Tree order is unchanged.

A lock per project directory (fixes a real race)

The race: Every Ceedling command that loads a project rewrites the vendored Unity and CMock files under build/vendor/ in place, dumpconfig included. Discovery's dumpconfig ran outside the old engine-wide mutex. So a rediscovery during a build in the same directory could leave gcc reading a half-written unity.h. The S2 stress run below reproduced this against real Ceedling as spurious TEST_CASE compile errors.

The fix:

  • The engine-wide mutex is replaced by KeyedLock, one lock per project directory.
  • dumpconfig, listings, runs, debug builds, and clean/clobber all hold their directory's lock.
  • Different directories proceed independently, so builds in separate projects may now overlap.
  • The async-mutex dependency is removed.

One shared read limit (fixes EMFILE)

All source reads share one Limiter capped at 16. Per-call bounds weren't enough: 600 simultaneous reparses plus two rediscoveries exhausted a 64-descriptor limit (S3).

Stress gate (one-off scripts, not committed)

All three ran against the final code with 0 violations and 0 unhandled rejections:

Run Setup Checks
S1 Compiled engine against a simulated Ceedling with random latency, failures, and cancellation. 60 seeds × 300 overlapping operations, up to 20 in flight. No two project-loading commands in one directory at once; project args and cwd on every command; exactly one result per leaf and none after a run ends; no commands after dispose; no live processes or temp files at the end.
S2 Real Ceedling 1.0.0, 1.1.10, and 1.2.0 in the madsciencelab images. 150 s of overlapping discovery, runs, debug builds, clean, and reparses across four projects (two sharing a directory), with random cancellation. Every completed run's results match the fixture's expected outcomes; no errored results; no leftover processes (via /proc); no leftover temp files.
S3 A 3,000-file × 20-case project on real disk under ulimit -n 64. Two concurrent discoveries plus 600 reparses. Full file count, listed order, case count; no logged errors.

Tests

New unit tests:

  • concurrency.test.ts covers mapGrouped, KeyedLock, and Limiter.
  • A "concurrent discovery" suite in ceedlingEngine.test.ts covers directory grouping, error ordering, lock interplay between rediscovery and builds, the shared read limit, and read order.

Results: 311 unit tests and 4 integration tests pass, and lint reports 0 findings.

Coverage of the pure modules

File Statements before → after Branches before → after
ceedlingEngine.ts 94.7% → 100% 90.0% → 98.1%
ceedlingProcess.ts 80.3% → 99.0% 97.4% → 96.2%¹
problemMatching.ts 100% 80.6% → 100%
runPlan.ts 100% 98.2% → 100%
unityTestDiscovery.ts 96.6% → 99.0% 89.3% → 93.2%
All files (adapters included, still at 0%) 69.9% → 72.9% 91.4% → 96.9%

¹ The new injection point adds branches.

What the new tests reach:

  • Engine: unparseable and missing dumpconfig output, a failed test listing, discovery warnings, an invalid file prefix for pretty labels, ignored and unmapped results, development-mode logging, a cancelled debug build, undeletable reports and temp files, project lookups, cancel and dispose, the shell and command settings, and the real filesystem and runner defaults.
  • Process: killProcessTree against a real process tree on every OS, and its fallback when tree-kill fails. The fallback is reached through an injected platform and treeKill.

One behavior change: a TEST_RANGE with no bracketed range now gives a discovery warning, as a TEST_MATRIX with no lists already did. Unity's generator fails on both (undefined method 'map' for nil). This was checked in the Ceedling 1.2.0 image.

What stays uncovered:

  • ?? fallbacks on captures that always participate;
  • a race guard for a process exiting while /proc is being read;
  • the engine's path.win32 selection. Unit tests always inject platform: 'linux', so this only runs on a real Windows host. The fake-Ceedling integration tests (PR E) run on the Windows CI leg and will cover it.

Docs

  • docs/Development.md: the module table gains concurrency.ts. Discovery describes what overlaps, the shared read limit, and dumpconfig's two forms. A new paragraph explains why commands in one project directory run one at a time.
  • CHANGELOG.md:
    • Changed: faster discovery, and commands in separate project directories may now run at the same time.
    • Added: the warning for a TEST_RANGE() or TEST_MATRIX() that Ceedling can't build.
    • The vendored-header race gets no entry: 2.0.1 held its mutex around files:test, so it only ever existed on next_version.

Upstream

Ceedling's non-atomic, in-place re-vendoring on every project-loading command is reported as ThrowTheSwitch/Ceedling#1361. This PR works around it.

🤖 Generated with Claude Code

mkarlesky and others added 2 commits October 23, 2026 13:59
… directory

File access no longer blocks the extension host thread. EngineFileSystem is fully async,
and projectsFromSettings() and configurationFiles() take and return promises.

Discovery overlaps the Ceedling starts it can:
- Projects load concurrently across directories and one at a time within a directory
  (mapGrouped). Outcomes apply in settings order, so errors read the same however loads
  interleave.
- Listings run concurrently across directories, and test sources are read concurrently.

The engine-wide mutex becomes a lock per project directory (KeyedLock). Every Ceedling
command that loads a project rewrites the vendored Unity and CMock files in its build
directory in place. Discovery's dumpconfig ran outside the old mutex, so a rediscovery
during a build in the same directory could leave gcc a half-written unity.h. A real Ceedling
stress run reproduced that as TEST_CASE compile errors. dumpconfig, listings, runs, debug
builds, and clean/clobber now hold their directory's lock. Different directories proceed
independently, so builds in separate projects may overlap. async-mutex is gone.

Every source read shares one limit (Limiter, 16). A stress run with 600 simultaneous reparses
alongside two rediscoveries hit EMFILE under a 64-descriptor limit with per-call bounds; it
runs clean with the shared limit.

Stress-tested before commit, all against the final code:
- S1: the compiled engine against a simulated Ceedling with random latencies, failures, and
  cancellation; 60 seeds x 300 overlapping operations; invariants on locking, project args,
  reporter results, live processes, and temp files.
- S2: real Ceedling 1.0.0, 1.1.10, and 1.2.0, overlapping discovery, runs, debug builds,
  clean, and reparses with random cancellation; checks results, leftover processes, and
  temp files.
- S3: a 3,000-file project under a 64-descriptor limit.
All ran with 0 violations and 0 unhandled rejections.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…thout a range

Unit tests now reach the error and edge paths that only real use exercised before:
- Engine: unparseable and missing dumpconfig output, a failed test listing, discovery warnings,
  an invalid file prefix for pretty labels, ignored and unmapped results, development-mode
  logging, a cancelled debug build, undeletable reports and temp files, project lookups,
  cancel and dispose, the shell and command settings, and the real filesystem and runner
  defaults.
- Process: killProcessTree against a real process tree on every OS, and its fallback when
  tree-kill fails. The fallback is reached through an injected platform and tree-kill.
- Problem matching, run plans, and Ruby float ranges: their remaining branches.

Unity's generator fails on a TEST_RANGE with no bracketed range, just as on a TEST_MATRIX with no
lists. Discovery now warns about both. Previously the range case passed silently.

Statement coverage of the pure modules is now 99-100%. What's left is unreachable `??`
fallbacks, a race guard, and the engine's Windows path selection. The fake-Ceedling
integration tests on the Windows CI leg are the place to cover that last one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mkarlesky
mkarlesky marked this pull request as ready for review October 23, 2026 18:11
Development.md: the module table gains concurrency.ts. Discovery describes what overlaps and
the shared read limit, and dumpconfig's stdout and temporary-file forms. A new paragraph
explains why Ceedling commands in one project directory run one at a time, and the running
step names the directory lock in place of the old mutex.

CHANGELOG: faster discovery, commands in separate project directories overlapping, and the
warning for a TEST_RANGE or TEST_MATRIX that Ceedling can't build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mkarlesky
mkarlesky changed the base branch from lint/fixes to next_version October 23, 2026 19:21
@mkarlesky
mkarlesky merged commit eba60bc into next_version Oct 23, 2026
3 checks passed
@mkarlesky
mkarlesky deleted the perf/async-coverage branch October 23, 2026 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant