Validation suites against real Ceedling releases (just validate)

Justfile

@@ -56,6 +56,11 @@ lint: image
 coverage: image
     just _run "npm run test:coverage"
 
+# Validate the compiled engine against a real Ceedling release in Docker, from the host (see docs/Development.md)
+validate VERSION SUITE="all" *ARGS="": image
+    just _run "npm run -s build:tests"
+    SIDECAR_IMAGE={{image_name}} NODE_MODULES_VOLUME={{node_modules_volume}} tests/validation/run.sh {{VERSION}} {{SUITE}} {{ARGS}}
+
 # Package the extension into a .vsix
 package: image
     just _run "npm run package"

docs/Architecture.md

@@ -200,7 +200,11 @@ Ceedling builds each test file in directories named for it, under `test/out`, `g
 - **Roots.** Each test path loses a leading `+:` or `-:` and the whitespace around it. A test path with a glob (`*`, `?`, `{`, `}`) is cut back to the directory before its first glob, so `+:test/**` gives `test`. One led by a glob gives nothing. Subtractive paths count as roots too, as in Ceedling.
 - **Choice.** The longest root that equals the file’s directory, or contains it, decides. A file under no root, or directly in one, builds under its base name. Paths compare as written, with backslashes read as `/`.
 
-Ceedling itself refuses some same-named layouts. Take test paths `+:test/unit/**` and `+:extra/**`. `test/unit/test_calc.c` then builds as `test_calc`, and `extra/unit/test_calc.c` as `unit/test_calc`. With real Ceedling 1.2.0 and 1.2.1, every build in that project stops with `EXCEPTION: Ambiguous file reference 'test_calc.c'`. `testBuildName()` still names both directories correctly.
+Ceedling itself can’t build some same-named layouts ([Ceedling#1364](https://github.com/ThrowTheSwitch/Ceedling/issues/1364)). To find a test’s own source, it searches every test file for the test’s build name. When that name ends another test file’s path, the search finds both and stops with `EXCEPTION: Ambiguous file reference`, whatever path the task named.
+- With the single test path `+:test/**`, `test/test_calc.c` builds as `test_calc`, which ends `test/unit/test_calc.c`. The first file can’t be built. The second can.
+- With `+:test/unit/**` and `+:extra/**`, `test/unit/test_calc.c` builds as `test_calc` and `extra/unit/test_calc.c` as `unit/test_calc`. Each name ends both paths, so neither file can be built.
+
+`test:all` fails in both. `testBuildName()` still names every directory correctly.
 
 ### Debugging
 
@@ -292,18 +296,24 @@ Ceedling 1.2.1 resolves three problems this extension works around ([#1360](http
 
 ### Possible speedup: discovery without the lock on 1.2.1
 
-Ceedling 1.2.1 could let discovery skip the build-root lock, so a rediscovery need not wait for a long run. This is not implemented. The evidence so far is suggestive, not conclusive.
+Ceedling 1.2.1 could let discovery skip the project’s locks, so a rediscovery need not wait for a long run. This is not implemented. The evidence so far supports it.
 
-**The opportunity.** A rediscovery in a project directory waits behind any run there, because `dumpconfig` and `files:test` take the project’s locks. Before 1.2.1 they must, since both rewrite `build/vendor/`. From 1.2.1, Ceedling brings `build/vendor/` up to date only for test and release builds, writes only changed files, and replaces each whole by renaming a temporary copy.
+**The opportunity.** A rediscovery in a project directory waits behind any run there, because `dumpconfig` and `files:test` take the project’s locks. Before 1.2.1 they must, since both rewrite `build/vendor/`. From 1.2.1, Ceedling brings `build/vendor/` up to date only for test and release builds, writes only changed files, and replaces each whole by renaming a temporary copy. A prototype that skipped the locks for these two commands cut a rediscovery during a 4.6-second run from about 5.0 seconds, nearly all of it waiting, to about 1.0 second, its own cost. The validation suite `latency` measures the locked case (see [Development.md](Development.md#5-validation-against-real-ceedling)).
 
-**The evidence.** A probe on Ceedling 1.2.1.dev listed every file under `build/` before and after `dumpconfig --stdout` and `files:test`. Neither command created or changed any of the 45 files. On 1.2.0 the same commands rewrote 8 vendored files. A stress test looping both commands against six clean `test:all` builds found no failure on either version, so it could not show that skipping the lock is safe, only that the race is rare.
+**The evidence.**
+- **No writes.** `probe:syscalls` traced every filesystem call both commands make under a build root populated by a full test and coverage build. On 1.2.1 neither wrote anything. They listed `build/vendor/unity/src` and `build/vendor/cmock/src` and checked that other directories exist. On 1.2.0 the same commands opened the 8 vendored sources for writing and truncated them.
+- **No reads that matter.** Discovery’s output doesn’t depend on what `build/` holds.
+- **Directories only.** Before the first build, discovery creates the build root’s directory skeleton: 20 empty directories. Builds create the same ones, and both tolerate a directory that already exists.
+- **Stress.** The prototype passed `stress`, with two seeds and with a shared build root, and `coverage` on 1.2.1. That came to about 600 operations with no violation, about 100 of them discoveries overlapping builds, cleans, and coverage runs.
+- **The limit of `probe:overlap`.** Looping discovery against clean builds found no failed build on 1.2.0 or 1.2.1. It can’t show that the race is safe, only that it is rare.
 
 **What a change would need.**
 - A capability, e.g. `discoveryWritesNoBuildFiles` from 1.2.1, so earlier versions keep the lock.
-- `resolveConfig()` and `listTestFiles()` taking no build-root lock on that capability, while builds, clean, and clobber keep both keys. The directory key could go too, unless two projects in one directory need it for another reason.
-- A real-Ceedling stress gate in which discovery overlaps builds in the same directory and build root, with results checked exactly, before and after the change.
+- `resolveConfig()` and `listTestFiles()` running alongside builds on that capability. The directory key could go too, unless two projects in one directory need it for another reason.
+- Discovery still kept apart from `clean` and `clobber`. Discovery recreates missing directories, so a discovery during `clobber` could recreate directories `clobber` is deleting. `KeyedLock` has only exclusive keys, so this needs a shared mode. Discoveries would take a second build-root key shared, and `clean` and `clobber` take it exclusive, beside the keys they hold today. Builds would not take it.
+- The validation suites, run before and after the change on every version.
 
-**The risks.** Discovery may create directories under `build/`, which the probe did not check. A later Ceedling could start writing there again, so the capability must stay pinned to versions that have been checked. And `clean` or `clobber` overlapping a discovery would still need their locks.
+**The risks.** A later Ceedling could start writing under `build/` during discovery again, so the capability must stay pinned to versions `probe:syscalls` has checked.
 
 ## 6. Test seams
 
@@ -328,3 +338,5 @@ The integration tests run the built extension in a real VS Code, on two workspac
 - the Coverage profile, and file coverage and its details reaching VS Code.
 
 On Windows they also run the engine with Windows paths. VS Code’s API can’t read another extension’s TestItems or run results, so `activate()` returns a small handle for the tests, `CeedlingExplorerApi`. See [`tests/integration/main.ts`](../tests/integration/main.ts).
+
+The validation suites in [`tests/validation/`](../tests/validation/) run the compiled engine against real Ceedling releases in Docker, with only `configProvider`, `problemMatcher`, and the logger stubbed. The real `runner` and `fileSystem` run real Ceedling. They check what a simulation can’t: the locks under real build times and random cancellation, each version’s command lines, and real coverage reports. They run by hand, not in CI. See [Development.md](Development.md#5-validation-against-real-ceedling).

docs/Development.md

@@ -1,6 +1,6 @@
 # Development
 
-This document is for contributors changing the extension’s own code. It starts with the daily workflow and a pointer to the architecture, which [Architecture.md](Architecture.md) covers in full. Then it covers running and debugging locally, the VS Code tools this work relies on, the manual testing plan, and the Docker sidecar.
+This document is for contributors changing the extension’s own code. It starts with the daily workflow and a pointer to the architecture, which [Architecture.md](Architecture.md) covers in full. Then it covers running and debugging locally, the VS Code tools this work relies on, validation against real Ceedling releases, the manual testing plan, and the Docker sidecar.
 
 ---
 
@@ -12,7 +12,7 @@ The loop is: edit, build, F5, iterate, test, commit.
 
 Source edits happen directly on the host. They happen in your real VS Code window. Docker never touches that window.
 
-The sidecar handles install, build, test, package, and publish. It runs in Docker. See [Section 6](#6-the-sidecar).
+The sidecar handles install, build, test, package, and publish. It runs in Docker. See [Section 7](#7-the-sidecar).
 
 First time: run `just install`. Then press F5.
 
@@ -152,9 +152,63 @@ Run `test_calculator_crash.c`. `test_should_crash` dereferences a null pointer.
 
 See [tests/manual/README.md](../tests/manual/README.md) for the `project.yml` settings the parametrized and crash tests need.
 
-## 5. Manual Testing Plan
+## 5. Validation Against Real Ceedling
 
-This checklist confirms the extension’s behavior in a real VS Code, where unit and integration tests can’t reach. Run it before a release, against the final code on the integration branch, on Ceedling 1.0.0, 1.1.10, 1.2.0, and 1.2.1.
+The unit and integration tests run against a simulated Ceedling. The validation suites run the engine against real Ceedling releases instead, to catch what a simulation can’t: races at real build speeds, each version’s own behavior, and real gcovr and ReportGenerator output. They aren’t part of `just test` or CI. Run them on every supported version before a release, and after changing the engine’s locking, its Ceedling command lines, or its coverage handling.
+
+```sh
+just validate 1.2.1                   # every suite, on Ceedling 1.2.1
+just validate 1.2.0 gated             # one suite
+just validate 1.1.10 stress 120 6 7   # 120 seconds, 6 operations at once, seed 7
+just validate 1.2.1 probe:syscalls    # one probe
+```
+
+A suite prints what it ran and every violation it found, and exits 0 only when every check held. `all`, the default, runs `stress` twice, the second time with one build root, then `coverage`, `gated`, and `layout`. Trailing numbers go to `stress` and `coverage`, so `just validate 1.2.1 all 45` gives each of their overlap phases 45 seconds. That takes about three minutes per version, and the default of 60 seconds about a minute more.
+
+### How they run
+
+- `just validate` compiles the tests into `out-test/` in the sidecar, then runs [`tests/validation/run.sh`](../tests/validation/run.sh) on the host. Run it from the host, not in `just shell`, since it needs Docker. `just install` must have run, to fill the `node_modules` volume.
+- Compiling replaces `out-test/`, as `just test` and `just coverage` do, so run one `just validate` at a time. To run several versions at once, compile once, then call `tests/validation/run.sh <version> <suite>` for each.
+- `run.sh` builds `vscode-ceedling-validate:<version>` from [`sidecar/validate.Dockerfile`](../sidecar/validate.Dockerfile). The image is `throwtheswitch/madsciencelab-plugins:<version>` plus the sidecar’s Node and `strace`. Any tag of that image works as a version.
+- Each suite runs in a fresh container, with the repo and the `node_modules` volume mounted read-only. It copies its projects from fixtures into the container’s own `/ws` and builds them there, so nothing is left on the host.
+- A suite drives the compiled `CeedlingEngine` directly, with VS Code’s side stubbed: settings, the logger, and the Problems panel. Every Ceedling command it issues is real.
+
+### The suites
+
+| Suite | Checks | Versions |
+|---|---|---|
+| `stress` | Random runs of files, functions, and projects, debug builds, discoveries, reparses, and cleans overlap across four projects, two of them sharing a directory. A fifth are cancelled at a random moment. Each completed run must report each of its own tests once, with the outcome [`tests/manual/`](../tests/manual/) expects. No Ceedling process or temporary file may be left. `--shared-build-root` gives all four projects one build root. Takes seconds (60), operations at once (6), and a seed (1). | All |
+| `coverage` | Exact outcomes one run at a time: gcovr preferred over ReportGenerator, counters reset or kept, a function run, a source above the project, ReportGenerator alone. Then coverage runs overlapping plain runs, discovery, reparses, and clean. Takes seconds (60), operations at once (5), and a seed (1). | All. 1.0.0’s image has no ReportGenerator, so its checks skip. |
+| `gated` | The engine’s per-version command lines, on same-named test files: coverage tasks named by path from 1.2.1, or by bare name with a warning on 1.2.0. Also `:report_task:`, a changed `--test-case` filter, plain runs, and the debug executable’s directory. | 1.2.0 and later |
+| `layout` | Exact test build directories in a project with two test roots: each debug build’s executable, and a same-named file’s coverage counters kept through a whole-file rerun. | 1.2.0 and later. Coverage from 1.2.1, since 1.2.0’s `gcov:` tasks can’t name either file. |
+| `latency` | Measures a rediscovery during a full build kept running about 4 seconds by a slow test, for [Architecture.md](Architecture.md#possible-speedup-discovery-without-the-lock-on-121). It checks only that the build’s results are unchanged. Not in `all`. | All |
+
+A seed replays the same sequence of operations, though not their timing, so a failure may need several runs to recur.
+
+### The probes
+
+The probes run Ceedling alone, without the engine, to show the Ceedling behavior the engine’s design rests on. They report what they find without judging it. Each heading in the output states what to expect. They target 1.2.0 and later, since discovery’s writes are traced through `dumpconfig --stdout`.
+
+- `probe:fixes` shows three things. Ceedling 1.2.1’s delta builds rerun a changed `--test-case` filter ([Ceedling#1360](https://github.com/ThrowTheSwitch/Ceedling/issues/1360)). A coverage run after a counter reset still needs `--force-test-rerun`. And discovery’s commands write under `build/` before 1.2.1 ([Ceedling#1361](https://github.com/ThrowTheSwitch/Ceedling/issues/1361)).
+- `probe:syscalls` traces every filesystem call `dumpconfig --stdout` and `files:test` make under a populated build root. It runs as root, which `strace` needs.
+- `probe:overlap` loops discovery’s commands against six clean builds in the same directory, and counts the builds that fail.
+
+### Changing them
+
+- The suites live in [`tests/validation/`](../tests/validation/), as TypeScript compiled with the tests. They typecheck against the engine and lint with the other tests.
+- [`harness.ts`](../tests/validation/harness.ts) holds what the suites share:
+  - the engine and its stubs;
+  - `stage()`, which copies a fixture into `/ws`;
+  - recording reporters;
+  - `churn()` for random overlap;
+  - the leftover check;
+  - `main()`, which prints the verdict and sets the exit code.
+- Fixtures live in [`tests/validation/workspaces/`](../tests/validation/workspaces/). `stress` copies [`tests/manual/`](../tests/manual/) instead.
+- A new suite is a `.ts` file there. `run.sh` finds it by name. Add it to `all` in `run.sh` if it should gate releases.
+
+## 6. Manual Testing Plan
+
+This checklist confirms the extension’s behavior in a real VS Code, where unit and integration tests can’t reach. Run it before a release, against the final code on the integration branch, on Ceedling 1.0.0, 1.1.10, 1.2.0, and 1.2.1. Run `just validate` on each of those versions first.
 
 Every step runs against [`tests/manual/`](../tests/manual/), opened as its own folder in the Development Host.
 
@@ -185,7 +239,7 @@ Every step runs against [`tests/manual/`](../tests/manual/), opened as its own f
 - [ ] **Coverage of same-named test files.** Add `test/alpha/test_dup.c` and `test/beta/test_dup.c`, each with one test calling a different function of `src/Calculator.c`. On 1.2.1, Run with Coverage each file. Each passes and shows only its own function as executed. On 1.2.0, the same run errors with Ceedling’s ambiguous file reference error, and the Ceedling Explorer log warns about the ambiguous name. Revert after.
 - [ ] **No Cobertura report.** Use `mixins/coverage-no-cobertura.yml` in place of the gcovr mixin. Run with Coverage still runs the tests. A notification says to add `Cobertura` to `:gcov:` ↳ `:reports:`. Remove the mixin setting after, and Run with Coverage disappears.
 
-## 6. The Sidecar
+## 7. The Sidecar
 
 The sidecar is Docker for everything except F5. It never touches the debug loop.
 
@@ -227,6 +281,7 @@ There are two modes.
 | `just test` | Runs the unit suite, then the integration suite, headless via Xvfb |
 | `just lint` | Reports lint findings |
 | `just coverage` | Runs the unit suite with coverage, writing `coverage/index.html` |
+| `just validate <version> [suite]` | Runs the validation suites against a real Ceedling release, from the host. See [Section 5](#5-validation-against-real-ceedling) |
 | `just package` | Produces a `.vsix` |
 | `just publish` | Publishes to the Marketplace. Requires `VSCE_PAT` |
 | `just shell` | Opens an interactive container session |

docs/KnownIssues.md

@@ -4,9 +4,13 @@ This document lists the problems currently known, grouped by the Ceedling versio
 
 ---
 
+## Ceedling 1.2.0 and later
+
+1. Ceedling can’t build a test file whose path within its test path ends another same-named test file’s path. Every run of that file, and of the whole project, stops with Ceedling’s `Ambiguous file reference` error, which shows as an error on each test. For example, with the test path `+:test/**`, `test/test_calc.c` can’t be built beside `test/unit/test_calc.c`, though `test/unit/test_calc.c` can. Renaming one of the files avoids it. The fix is tracked in [Ceedling#1364](https://github.com/ThrowTheSwitch/Ceedling/issues/1364).
+
 ## Ceedling versions before 1.2.1
 
-1. A coverage run can’t tell apart same-named test files in different directories, e.g. `test/unit/test_foo.c` and `test/integration/test_foo.c`. Ceedling’s `gcov:` tasks before 1.2.1 take a test file’s name without its directory. Ceedling 1.2.0 then stops with an ambiguous file reference error, and earlier versions may run the other file. The extension warns in its log when a coverage run is ambiguous this way. Plain runs tell the files apart from Ceedling 1.2.0. The fix is tracked in [Ceedling#1362](https://github.com/ThrowTheSwitch/Ceedling/issues/1362).
+1. A coverage run can’t tell apart same-named test files in different directories, e.g. `test/unit/test_foo.c` and `test/integration/test_foo.c`. Ceedling’s `gcov:` tasks before 1.2.1 take a test file’s name without its directory. Ceedling 1.2.0 then stops with an ambiguous file reference error, and earlier versions may run the other file. The extension warns in its log when a coverage run is ambiguous this way. Plain runs tell the files apart from Ceedling 1.2.0. Ceedling 1.2.1 fixes this ([Ceedling#1362](https://github.com/ThrowTheSwitch/Ceedling/issues/1362)).
 
 ## Ceedling versions before 1.2.0