Skip to content

✅ Integration tests against a fake Ceedling, and the final review - #25

Merged
mkarlesky merged 6 commits into
next_versionfrom
test/fake-ceedling-integration
Oct 23, 2026
Merged

mkarlesky merged 6 commits into
next_versionfrom
test/fake-ceedling-integration

Conversation

@mkarlesky

@mkarlesky mkarlesky commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Fifth and last PR in the lint-and-coverage series, stacked on #24. It adds integration tests that drive the real extension in VS Code against a fake Ceedling. It also carries the series' final pass: a last lint, and a thorough review of every change in #21–#25, with its findings fixed here.

Fake Ceedling

  • The fake: tests/fixtures/fake-ceedling/ceedling.js is a Node script. It answers version, dumpconfig --stdout, files:test, test: tasks, clean and clobber as Ceedling 1.2.0 does, from the scenario file fake-ceedling.json in the project.
    • Its CppUnit reports follow the structure of the captured real reports.
    • Its compiler failure is the captured ceedling-1.0.0/compile-failure-stderr.txt.
    • It logs every invocation, so the tests check the arguments the extension passes.
  • The workspace: tests/integration/main.ts copies tests/fixtures/workspace-fake-ceedling to a temporary directory and points ceedlingExplorer.ceedlingCommand at the fake. It opens that copy alongside the existing workspace-without-project as a two-folder workspace, so builds never write into the repository.
  • The test handle: VS Code's API can't read another extension's TestItems or run results. So activate() returns CeedlingExplorerApi, which gives the tests each folder's controller. The controller adds onDidReport, whenDiscovered() and run() for them.

Scenarios (10 new; 14 integration tests in all)

  • A folder without a project gets no TestController, and Ceedling never runs there.
  • Discovery: the files from files:test, functions and cases with their pretty labels and lines, and the controller named for its folder.
  • A file run: passed, failed (message, and location at the assertion's line), ignored, and parametrized cases. The fake's log confirms --force-test-rerun and the bundled mixin.
  • A function run: --test-case=<name> reaches Ceedling, and only that function reports.
  • A function run whose filter, as a substring match, also selects another function: the engine reruns the whole file.
  • An unreported test errors, naming the likely cause.
  • A compile failure errors the file's tests and reaches the Problems panel at the captured error's line.
  • Cancellation stops the hung fake and marks the test skipped.
  • A test file created on disk is discovered, and one deleted goes away.
  • An unsaved edit adds a test to the tree, and reverting removes it.

On the Windows CI leg these run the engine with Windows paths, which unit tests can't reach.

Final pass

Lint: 0 findings. The ban on synchronous fs calls in src/ now covers every synchronous fs method.

Review: four parallel reviews covered every change in #21–#25 against next_version, by area:

  • engine and concurrency;
  • adapters and extracted modules;
  • parsing and planning modules;
  • tests, tooling, docs, and style.

Each finding was checked against the code before any fix, and each fix that changes behavior was written test-first.

Fixed

Finding Where Fix
Windows: testFileAt never matched. Discovery uppercases the drive letter for gcov, but VS Code reports c:\…, and the comparison was exact. Unsaved-edit, save, and close reparses silently did nothing. The new Windows integration leg failed on it. Introduced on next_version, never released. ceedlingEngine.ts pathKey() compares Windows paths case-insensitively.
Windows: one directory could get two lock keys. An absolute projects[].path kept a lowercase drive letter, and directory locks keyed on the raw path. Two spellings of one directory could then build at once, the corruption the lock exists to prevent. ceedlingProjectConfig.ts, ceedlingEngine.ts Every project path gets an uppercase drive letter, and locks and discovery groups key on pathKey().
A captured line or column of 0 (GCC's <command-line>:0:0:) became −1. vscode.Position then threw, dropping every diagnostic from that pattern for the run. Shipped in 2.0.1. problemMatching.ts Positions clamp at 0. Added to the CHANGELOG.
extractTestFunctionName split at the last ::. test_s("a::b") gave b"), causing a false collision warning and a whole-file rerun. Shipped in 2.0.1. ceedlingOutputParsing.ts It splits at the first ::. Added to the CHANGELOG.
A cancelled Clean or Clobber reported failure. ceedlingEngine.ts Cancellation is no failure.
Discovery in flight at disposal could show an error, and create watchers and a TestController, for a removed folder. testController.ts A disposed guard.
The test API reached every extension in production, including dispose(). main.ts It is returned only in ExtensionMode.Test.
run() could create a TestController for a folder without a project. testController.ts It does nothing there.
The fake's --test-case was an exact match, unlike Ceedling's substring match, so the engine's rerun path went untested in VS Code. fake Ceedling Substring matching, plus an integration test of the rerun.
The Clean integration test could run before discovery finished. extension.test.ts It waits for discovery.
"Never runs Ceedling in a folder without a project" was claimed but unchecked. integration Both folders run the fake, and that folder must have no log.
Removing the temporary workspace could throw on Windows and fail a passing run. tests/integration/main.ts Retries, then a warning.
CI's coverage step hid failures behind tee. ci.yml shell: bash (pipefail).
npm test compiled the test build twice; clean left out-test/ and coverage/. package.json Fixed both.
Stale docs and comments: the debug executable layout from Ceedling 1.2.0, "build lock", "the engine serializes runs", and a test helper's comment. various Corrected.
Duplicated SourceEvent union. testController.ts Imports the type.
A failed ceedling files:test emptied the project silently. The project loaded with no test files. Shipped in 2.0.1. ceedlingEngine.ts Discovery fails with a CeedlingLoadError naming each project whose listing failed and the settings to check. The engine keeps its last good projects, as for every failed discovery. Added to the CHANGELOG.

Deferred, with reasons

  • No integration coverage of the Debug profile (deferred by decision). It would need VS Code's built-in JavaScript debugger to start sessions reliably in CI on all three OSes. debugLaunch.ts is unit-tested, and the manual checklist covers debugging.
  • One unbuildable marker among valid ones (e.g. a TEST_RANGE() without a range beside a valid one) still lists the valid marker's cases. The warning explains why they go unreported, as it already does for a zero step.

Reviewed and found sound

  • Locking: no deadlocks or lock-ordering cycles; every Ceedling command except version holds its directory lock.
  • Promises: no floating promises or unhandled rejections in src/.
  • Extraction: it preserves behavior path by path.
  • Lint-pass fallbacks: none of the ?? fallbacks can fire.
  • lineAt: the binary search is equivalent to the old linear scan.
  • Concurrency tests: they fail when the lock or the shared read limit is removed. A reviewer checked this by breaking the code on purpose.
  • Docs: every symbol they name exists.

CI is green on Linux, macOS, and Windows. 356 unit tests and 14 integration tests pass.

🤖 Generated with Claude Code

mkarlesky and others added 5 commits October 23, 2026 14:53
The integration suite now opens two workspace folders: the existing one without a project, and
a temporary copy of a fixture project whose ceedlingExplorer.ceedlingCommand runs
tests/fixtures/fake-ceedling/ceedling.js. The fake answers version, dumpconfig --stdout,
files:test, test tasks, clean, and clobber from the project's scenario file. Its reports follow
the captured CppUnit reports' structure, and its compiler failure is a captured one.

Nine new tests cover the wiring between the engine and VS Code:
- a folder without a project gets no TestController;
- the discovered tree, its labels and lines;
- file and function runs with their results;
- an unreported test;
- a compile failure in the Testing view and the Problems panel;
- cancellation;
- a test file created and deleted on disk;
- an unsaved edit.

On the Windows CI leg the engine runs with Windows paths, which unit tests can't reach.

VS Code's API can't read another extension's TestItems or run results, so activate() returns
CeedlingExplorerApi, a handle giving the tests each folder's controller. The controller adds
onDidReport, whenDiscovered(), and run() for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd test gaps

Windows:
- testFileAt compared paths exactly. Discovery uppercases the drive letter for gcov, and VS Code
  reports `c:\…`, so unsaved-edit, save, and close reparses never matched a file. The new
  Windows integration leg showed it. pathKey compares Windows paths case-insensitively.
- Directory locks and discovery groups key on pathKey too. An absolute project path also gets
  an uppercase drive letter, so two spellings of one directory can't build at once.

Engine and parsing:
- A cancelled clean or clobber is no failure, so cancelling shows no error.
- extractTestFunctionName splits at the first `::`. A case argument such as "a::b" produced a
  false collision and a needless whole-file rerun.
- A captured line or column of 0, as GCC reports for `<command-line>`, places a problem at the
  start. It became -1, and vscode.Position threw, losing the pattern's diagnostics for the run.

Controller:
- A discovery in flight at disposal no longer shows an error, or creates a TestController or
  watchers, for a removed folder.
- activate() returns the test API only in test mode. run() does nothing for a folder without a
  TestController.

Tests and tooling:
- The fake's --test-case filter is a substring match, as Ceedling's is. An integration test
  covers the resulting whole-file rerun.
- Both integration folders run the fake, so the one without a project proves Ceedling never runs
  there. The Clean test waits for discovery.
- The launcher retries removing its temporary workspace.
- npm test compiles the test build once. clean also removes out-test and coverage.
- CI's coverage step uses bash's pipefail.
- The lint ban on synchronous fs covers every synchronous call.

Docs: the debug executable's layout before and from Ceedling 1.2.0, stale comments, and
just clean's description.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mkarlesky
mkarlesky marked this pull request as ready for review October 23, 2026 19:08
A failed listing gave its project no test files, so the Testing view silently showed no tests.
The listing now fails discovery like any other load problem. A CeedlingLoadError names each
project whose listing failed, in settings order, and the settings to check. The engine keeps
its last good projects for runs and watchers, as for every failed discovery. Ceedling's output
is logged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mkarlesky
mkarlesky changed the base branch from refactor/adapter-extraction to next_version October 23, 2026 19:21
@mkarlesky
mkarlesky merged commit 8558952 into next_version Oct 23, 2026
3 checks passed
@mkarlesky
mkarlesky deleted the test/fake-ceedling-integration branch October 23, 2026 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant