diff --git a/Justfile b/Justfile index a9a8161..ed8d4e9 100644 --- a/Justfile +++ b/Justfile @@ -56,6 +56,11 @@ lint: image coverage: image just _run "npm run test:coverage" +# Validate the compiled engine against a real Ceedling release in Docker, from the host (see docs/Development.md) +validate VERSION SUITE="all" *ARGS="": image + just _run "npm run -s build:tests" + SIDECAR_IMAGE={{image_name}} NODE_MODULES_VOLUME={{node_modules_volume}} tests/validation/run.sh {{VERSION}} {{SUITE}} {{ARGS}} + # Package the extension into a .vsix package: image just _run "npm run package" diff --git a/docs/Architecture.md b/docs/Architecture.md index 690289f..53c1b3c 100644 --- a/docs/Architecture.md +++ b/docs/Architecture.md @@ -200,7 +200,11 @@ Ceedling builds each test file in directories named for it, under `test/out`, `g - **Roots.** Each test path loses a leading `+:` or `-:` and the whitespace around it. A test path with a glob (`*`, `?`, `{`, `}`) is cut back to the directory before its first glob, so `+:test/**` gives `test`. One led by a glob gives nothing. Subtractive paths count as roots too, as in Ceedling. - **Choice.** The longest root that equals the file’s directory, or contains it, decides. A file under no root, or directly in one, builds under its base name. Paths compare as written, with backslashes read as `/`. -Ceedling itself refuses some same-named layouts. Take test paths `+:test/unit/**` and `+:extra/**`. `test/unit/test_calc.c` then builds as `test_calc`, and `extra/unit/test_calc.c` as `unit/test_calc`. With real Ceedling 1.2.0 and 1.2.1, every build in that project stops with `EXCEPTION: Ambiguous file reference 'test_calc.c'`. `testBuildName()` still names both directories correctly. +Ceedling itself can’t build some same-named layouts ([Ceedling#1364](https://github.com/ThrowTheSwitch/Ceedling/issues/1364)). To find a test’s own source, it searches every test file for the test’s build name. When that name ends another test file’s path, the search finds both and stops with `EXCEPTION: Ambiguous file reference`, whatever path the task named. +- With the single test path `+:test/**`, `test/test_calc.c` builds as `test_calc`, which ends `test/unit/test_calc.c`. The first file can’t be built. The second can. +- With `+:test/unit/**` and `+:extra/**`, `test/unit/test_calc.c` builds as `test_calc` and `extra/unit/test_calc.c` as `unit/test_calc`. Each name ends both paths, so neither file can be built. + +`test:all` fails in both. `testBuildName()` still names every directory correctly. ### Debugging @@ -292,18 +296,24 @@ Ceedling 1.2.1 resolves three problems this extension works around ([#1360](http ### Possible speedup: discovery without the lock on 1.2.1 -Ceedling 1.2.1 could let discovery skip the build-root lock, so a rediscovery need not wait for a long run. This is not implemented. The evidence so far is suggestive, not conclusive. +Ceedling 1.2.1 could let discovery skip the project’s locks, so a rediscovery need not wait for a long run. This is not implemented. The evidence so far supports it. -**The opportunity.** A rediscovery in a project directory waits behind any run there, because `dumpconfig` and `files:test` take the project’s locks. Before 1.2.1 they must, since both rewrite `build/vendor/`. From 1.2.1, Ceedling brings `build/vendor/` up to date only for test and release builds, writes only changed files, and replaces each whole by renaming a temporary copy. +**The opportunity.** A rediscovery in a project directory waits behind any run there, because `dumpconfig` and `files:test` take the project’s locks. Before 1.2.1 they must, since both rewrite `build/vendor/`. From 1.2.1, Ceedling brings `build/vendor/` up to date only for test and release builds, writes only changed files, and replaces each whole by renaming a temporary copy. A prototype that skipped the locks for these two commands cut a rediscovery during a 4.6-second run from about 5.0 seconds, nearly all of it waiting, to about 1.0 second, its own cost. The validation suite `latency` measures the locked case (see [Development.md](Development.md#5-validation-against-real-ceedling)). -**The evidence.** A probe on Ceedling 1.2.1.dev listed every file under `build/` before and after `dumpconfig --stdout` and `files:test`. Neither command created or changed any of the 45 files. On 1.2.0 the same commands rewrote 8 vendored files. A stress test looping both commands against six clean `test:all` builds found no failure on either version, so it could not show that skipping the lock is safe, only that the race is rare. +**The evidence.** +- **No writes.** `probe:syscalls` traced every filesystem call both commands make under a build root populated by a full test and coverage build. On 1.2.1 neither wrote anything. They listed `build/vendor/unity/src` and `build/vendor/cmock/src` and checked that other directories exist. On 1.2.0 the same commands opened the 8 vendored sources for writing and truncated them. +- **No reads that matter.** Discovery’s output doesn’t depend on what `build/` holds. +- **Directories only.** Before the first build, discovery creates the build root’s directory skeleton: 20 empty directories. Builds create the same ones, and both tolerate a directory that already exists. +- **Stress.** The prototype passed `stress`, with two seeds and with a shared build root, and `coverage` on 1.2.1. That came to about 600 operations with no violation, about 100 of them discoveries overlapping builds, cleans, and coverage runs. +- **The limit of `probe:overlap`.** Looping discovery against clean builds found no failed build on 1.2.0 or 1.2.1. It can’t show that the race is safe, only that it is rare. **What a change would need.** - A capability, e.g. `discoveryWritesNoBuildFiles` from 1.2.1, so earlier versions keep the lock. -- `resolveConfig()` and `listTestFiles()` taking no build-root lock on that capability, while builds, clean, and clobber keep both keys. The directory key could go too, unless two projects in one directory need it for another reason. -- A real-Ceedling stress gate in which discovery overlaps builds in the same directory and build root, with results checked exactly, before and after the change. +- `resolveConfig()` and `listTestFiles()` running alongside builds on that capability. The directory key could go too, unless two projects in one directory need it for another reason. +- Discovery still kept apart from `clean` and `clobber`. Discovery recreates missing directories, so a discovery during `clobber` could recreate directories `clobber` is deleting. `KeyedLock` has only exclusive keys, so this needs a shared mode. Discoveries would take a second build-root key shared, and `clean` and `clobber` take it exclusive, beside the keys they hold today. Builds would not take it. +- The validation suites, run before and after the change on every version. -**The risks.** Discovery may create directories under `build/`, which the probe did not check. A later Ceedling could start writing there again, so the capability must stay pinned to versions that have been checked. And `clean` or `clobber` overlapping a discovery would still need their locks. +**The risks.** A later Ceedling could start writing under `build/` during discovery again, so the capability must stay pinned to versions `probe:syscalls` has checked. ## 6. Test seams @@ -328,3 +338,5 @@ The integration tests run the built extension in a real VS Code, on two workspac - the Coverage profile, and file coverage and its details reaching VS Code. On Windows they also run the engine with Windows paths. VS Code’s API can’t read another extension’s TestItems or run results, so `activate()` returns a small handle for the tests, `CeedlingExplorerApi`. See [`tests/integration/main.ts`](../tests/integration/main.ts). + +The validation suites in [`tests/validation/`](../tests/validation/) run the compiled engine against real Ceedling releases in Docker, with only `configProvider`, `problemMatcher`, and the logger stubbed. The real `runner` and `fileSystem` run real Ceedling. They check what a simulation can’t: the locks under real build times and random cancellation, each version’s command lines, and real coverage reports. They run by hand, not in CI. See [Development.md](Development.md#5-validation-against-real-ceedling). diff --git a/docs/Development.md b/docs/Development.md index dc46073..c255977 100644 --- a/docs/Development.md +++ b/docs/Development.md @@ -1,6 +1,6 @@ # Development -This document is for contributors changing the extension’s own code. It starts with the daily workflow and a pointer to the architecture, which [Architecture.md](Architecture.md) covers in full. Then it covers running and debugging locally, the VS Code tools this work relies on, the manual testing plan, and the Docker sidecar. +This document is for contributors changing the extension’s own code. It starts with the daily workflow and a pointer to the architecture, which [Architecture.md](Architecture.md) covers in full. Then it covers running and debugging locally, the VS Code tools this work relies on, validation against real Ceedling releases, the manual testing plan, and the Docker sidecar. --- @@ -12,7 +12,7 @@ The loop is: edit, build, F5, iterate, test, commit. Source edits happen directly on the host. They happen in your real VS Code window. Docker never touches that window. -The sidecar handles install, build, test, package, and publish. It runs in Docker. See [Section 6](#6-the-sidecar). +The sidecar handles install, build, test, package, and publish. It runs in Docker. See [Section 7](#7-the-sidecar). First time: run `just install`. Then press F5. @@ -152,9 +152,63 @@ Run `test_calculator_crash.c`. `test_should_crash` dereferences a null pointer. See [tests/manual/README.md](../tests/manual/README.md) for the `project.yml` settings the parametrized and crash tests need. -## 5. Manual Testing Plan +## 5. Validation Against Real Ceedling -This checklist confirms the extension’s behavior in a real VS Code, where unit and integration tests can’t reach. Run it before a release, against the final code on the integration branch, on Ceedling 1.0.0, 1.1.10, 1.2.0, and 1.2.1. +The unit and integration tests run against a simulated Ceedling. The validation suites run the engine against real Ceedling releases instead, to catch what a simulation can’t: races at real build speeds, each version’s own behavior, and real gcovr and ReportGenerator output. They aren’t part of `just test` or CI. Run them on every supported version before a release, and after changing the engine’s locking, its Ceedling command lines, or its coverage handling. + +```sh +just validate 1.2.1 # every suite, on Ceedling 1.2.1 +just validate 1.2.0 gated # one suite +just validate 1.1.10 stress 120 6 7 # 120 seconds, 6 operations at once, seed 7 +just validate 1.2.1 probe:syscalls # one probe +``` + +A suite prints what it ran and every violation it found, and exits 0 only when every check held. `all`, the default, runs `stress` twice, the second time with one build root, then `coverage`, `gated`, and `layout`. Trailing numbers go to `stress` and `coverage`, so `just validate 1.2.1 all 45` gives each of their overlap phases 45 seconds. That takes about three minutes per version, and the default of 60 seconds about a minute more. + +### How they run + +- `just validate` compiles the tests into `out-test/` in the sidecar, then runs [`tests/validation/run.sh`](../tests/validation/run.sh) on the host. Run it from the host, not in `just shell`, since it needs Docker. `just install` must have run, to fill the `node_modules` volume. +- Compiling replaces `out-test/`, as `just test` and `just coverage` do, so run one `just validate` at a time. To run several versions at once, compile once, then call `tests/validation/run.sh ` for each. +- `run.sh` builds `vscode-ceedling-validate:` from [`sidecar/validate.Dockerfile`](../sidecar/validate.Dockerfile). The image is `throwtheswitch/madsciencelab-plugins:` plus the sidecar’s Node and `strace`. Any tag of that image works as a version. +- Each suite runs in a fresh container, with the repo and the `node_modules` volume mounted read-only. It copies its projects from fixtures into the container’s own `/ws` and builds them there, so nothing is left on the host. +- A suite drives the compiled `CeedlingEngine` directly, with VS Code’s side stubbed: settings, the logger, and the Problems panel. Every Ceedling command it issues is real. + +### The suites + +| Suite | Checks | Versions | +|---|---|---| +| `stress` | Random runs of files, functions, and projects, debug builds, discoveries, reparses, and cleans overlap across four projects, two of them sharing a directory. A fifth are cancelled at a random moment. Each completed run must report each of its own tests once, with the outcome [`tests/manual/`](../tests/manual/) expects. No Ceedling process or temporary file may be left. `--shared-build-root` gives all four projects one build root. Takes seconds (60), operations at once (6), and a seed (1). | All | +| `coverage` | Exact outcomes one run at a time: gcovr preferred over ReportGenerator, counters reset or kept, a function run, a source above the project, ReportGenerator alone. Then coverage runs overlapping plain runs, discovery, reparses, and clean. Takes seconds (60), operations at once (5), and a seed (1). | All. 1.0.0’s image has no ReportGenerator, so its checks skip. | +| `gated` | The engine’s per-version command lines, on same-named test files: coverage tasks named by path from 1.2.1, or by bare name with a warning on 1.2.0. Also `:report_task:`, a changed `--test-case` filter, plain runs, and the debug executable’s directory. | 1.2.0 and later | +| `layout` | Exact test build directories in a project with two test roots: each debug build’s executable, and a same-named file’s coverage counters kept through a whole-file rerun. | 1.2.0 and later. Coverage from 1.2.1, since 1.2.0’s `gcov:` tasks can’t name either file. | +| `latency` | Measures a rediscovery during a full build kept running about 4 seconds by a slow test, for [Architecture.md](Architecture.md#possible-speedup-discovery-without-the-lock-on-121). It checks only that the build’s results are unchanged. Not in `all`. | All | + +A seed replays the same sequence of operations, though not their timing, so a failure may need several runs to recur. + +### The probes + +The probes run Ceedling alone, without the engine, to show the Ceedling behavior the engine’s design rests on. They report what they find without judging it. Each heading in the output states what to expect. They target 1.2.0 and later, since discovery’s writes are traced through `dumpconfig --stdout`. + +- `probe:fixes` shows three things. Ceedling 1.2.1’s delta builds rerun a changed `--test-case` filter ([Ceedling#1360](https://github.com/ThrowTheSwitch/Ceedling/issues/1360)). A coverage run after a counter reset still needs `--force-test-rerun`. And discovery’s commands write under `build/` before 1.2.1 ([Ceedling#1361](https://github.com/ThrowTheSwitch/Ceedling/issues/1361)). +- `probe:syscalls` traces every filesystem call `dumpconfig --stdout` and `files:test` make under a populated build root. It runs as root, which `strace` needs. +- `probe:overlap` loops discovery’s commands against six clean builds in the same directory, and counts the builds that fail. + +### Changing them + +- The suites live in [`tests/validation/`](../tests/validation/), as TypeScript compiled with the tests. They typecheck against the engine and lint with the other tests. +- [`harness.ts`](../tests/validation/harness.ts) holds what the suites share: + - the engine and its stubs; + - `stage()`, which copies a fixture into `/ws`; + - recording reporters; + - `churn()` for random overlap; + - the leftover check; + - `main()`, which prints the verdict and sets the exit code. +- Fixtures live in [`tests/validation/workspaces/`](../tests/validation/workspaces/). `stress` copies [`tests/manual/`](../tests/manual/) instead. +- A new suite is a `.ts` file there. `run.sh` finds it by name. Add it to `all` in `run.sh` if it should gate releases. + +## 6. Manual Testing Plan + +This checklist confirms the extension’s behavior in a real VS Code, where unit and integration tests can’t reach. Run it before a release, against the final code on the integration branch, on Ceedling 1.0.0, 1.1.10, 1.2.0, and 1.2.1. Run `just validate` on each of those versions first. Every step runs against [`tests/manual/`](../tests/manual/), opened as its own folder in the Development Host. @@ -185,7 +239,7 @@ Every step runs against [`tests/manual/`](../tests/manual/), opened as its own f - [ ] **Coverage of same-named test files.** Add `test/alpha/test_dup.c` and `test/beta/test_dup.c`, each with one test calling a different function of `src/Calculator.c`. On 1.2.1, Run with Coverage each file. Each passes and shows only its own function as executed. On 1.2.0, the same run errors with Ceedling’s ambiguous file reference error, and the Ceedling Explorer log warns about the ambiguous name. Revert after. - [ ] **No Cobertura report.** Use `mixins/coverage-no-cobertura.yml` in place of the gcovr mixin. Run with Coverage still runs the tests. A notification says to add `Cobertura` to `:gcov:` ↳ `:reports:`. Remove the mixin setting after, and Run with Coverage disappears. -## 6. The Sidecar +## 7. The Sidecar The sidecar is Docker for everything except F5. It never touches the debug loop. @@ -227,6 +281,7 @@ There are two modes. | `just test` | Runs the unit suite, then the integration suite, headless via Xvfb | | `just lint` | Reports lint findings | | `just coverage` | Runs the unit suite with coverage, writing `coverage/index.html` | +| `just validate [suite]` | Runs the validation suites against a real Ceedling release, from the host. See [Section 5](#5-validation-against-real-ceedling) | | `just package` | Produces a `.vsix` | | `just publish` | Publishes to the Marketplace. Requires `VSCE_PAT` | | `just shell` | Opens an interactive container session | diff --git a/docs/KnownIssues.md b/docs/KnownIssues.md index 3e92c89..2da0e5f 100644 --- a/docs/KnownIssues.md +++ b/docs/KnownIssues.md @@ -4,9 +4,13 @@ This document lists the problems currently known, grouped by the Ceedling versio --- +## Ceedling 1.2.0 and later + +1. Ceedling can’t build a test file whose path within its test path ends another same-named test file’s path. Every run of that file, and of the whole project, stops with Ceedling’s `Ambiguous file reference` error, which shows as an error on each test. For example, with the test path `+:test/**`, `test/test_calc.c` can’t be built beside `test/unit/test_calc.c`, though `test/unit/test_calc.c` can. Renaming one of the files avoids it. The fix is tracked in [Ceedling#1364](https://github.com/ThrowTheSwitch/Ceedling/issues/1364). + ## Ceedling versions before 1.2.1 -1. A coverage run can’t tell apart same-named test files in different directories, e.g. `test/unit/test_foo.c` and `test/integration/test_foo.c`. Ceedling’s `gcov:` tasks before 1.2.1 take a test file’s name without its directory. Ceedling 1.2.0 then stops with an ambiguous file reference error, and earlier versions may run the other file. The extension warns in its log when a coverage run is ambiguous this way. Plain runs tell the files apart from Ceedling 1.2.0. The fix is tracked in [Ceedling#1362](https://github.com/ThrowTheSwitch/Ceedling/issues/1362). +1. A coverage run can’t tell apart same-named test files in different directories, e.g. `test/unit/test_foo.c` and `test/integration/test_foo.c`. Ceedling’s `gcov:` tasks before 1.2.1 take a test file’s name without its directory. Ceedling 1.2.0 then stops with an ambiguous file reference error, and earlier versions may run the other file. The extension warns in its log when a coverage run is ambiguous this way. Plain runs tell the files apart from Ceedling 1.2.0. Ceedling 1.2.1 fixes this ([Ceedling#1362](https://github.com/ThrowTheSwitch/Ceedling/issues/1362)). ## Ceedling versions before 1.2.0 diff --git a/package.json b/package.json index df9fad4..15906ae 100644 --- a/package.json +++ b/package.json @@ -55,6 +55,7 @@ "test:integration": "npm run -s build && rimraf out-test && tsc -p test.tsconfig.json && node ./out-test/tests/integration/main.js", "test": "npm run -s typecheck && npm run -s test:unit && npm run -s build && node ./out-test/tests/integration/main.js", "typecheck": "tsc -p tsconfig.json --noEmit", + "build:tests": "rimraf out-test && tsc -p test.tsconfig.json", "test:coverage": "rimraf out-test coverage && tsc -p test.tsconfig.json && c8 mocha --ui tdd \"out-test/tests/unit/**/*.test.js\"", "lint": "eslint ." }, diff --git a/sidecar/README.md b/sidecar/README.md index 4fa4c91..70a6d58 100644 --- a/sidecar/README.md +++ b/sidecar/README.md @@ -20,7 +20,7 @@ The image holds none of the repository. The checkout is bind-mounted when a cont ## Using it through the Justfile -Every interaction goes through the root `Justfile`. `just --list` shows its recipes: `install`, `build`, `watch`, `rebuild`, `test`, `lint`, `coverage`, `package`, `publish`, `clean`, `shell`, and `image`. Don’t run `docker` directly. The recipes handle image builds, volume mounts, and environment forwarding consistently. +Every interaction goes through the root `Justfile`. `just --list` shows its recipes: `install`, `build`, `watch`, `rebuild`, `test`, `lint`, `coverage`, `validate`, `package`, `publish`, `clean`, `shell`, and `image`. Don’t run `docker` directly. The recipes handle image builds, volume mounts, and environment forwarding consistently. Recipes run in one of two modes. @@ -33,6 +33,7 @@ Recipes run in one of two modes. - **First-time setup:** `just install`. - **Iterating:** `just build`, `just watch`, or `just test`. - **Finding what to fix:** `just lint` and `just coverage`. See [docs/Development.md](../docs/Development.md#linting-and-coverage). +- **Validating against real Ceedling:** `just validate `, from the host. It compiles the tests here, then runs them in a Ceedling release’s image with this image’s Node. See [docs/Development.md](../docs/Development.md#5-validation-against-real-ceedling). - **Releasing:** `just package` produces a `.vsix`. `VSCE_PAT= just publish` publishes it, and fails at once if `VSCE_PAT` isn’t set. - **Repeated runs:** `just shell`, then `just build`, `just test`, or plain `npm` commands as often as needed. diff --git a/sidecar/validate.Dockerfile b/sidecar/validate.Dockerfile new file mode 100644 index 0000000..6dfc8d0 --- /dev/null +++ b/sidecar/validate.Dockerfile @@ -0,0 +1,17 @@ +# A Ceedling release with the sidecar's Node added, so the validation suites in tests/validation can +# drive the compiled engine against real Ceedling. tests/validation/run.sh builds one image per +# Ceedling version, tagged vscode-ceedling-validate:. +ARG CEEDLING=1.2.1 +ARG SIDECAR=vscode-ceedling-sidecar +FROM ${SIDECAR} AS sidecar + +FROM throwtheswitch/madsciencelab-plugins:${CEEDLING} +COPY --from=sidecar /usr/local/bin/node /usr/local/bin/node +USER root +# strace: the discovery syscall probe traces Ceedling's file access +RUN apt-get update -qq && apt-get install -y -qq --no-install-recommends strace >/dev/null && rm -rf /var/lib/apt/lists/* +# /ws: where each run builds its projects, inside the container +RUN mkdir /ws && chown dev /ws +USER dev +# ReportGenerator, where the image has it, installs as a .NET global tool +ENV PATH=$PATH:/home/dev/.dotnet/tools diff --git a/tests/validation/coverage.ts b/tests/validation/coverage.ts new file mode 100644 index 0000000..24f5125 --- /dev/null +++ b/tests/validation/coverage.ts @@ -0,0 +1,101 @@ +// Coverage against real Ceedling, gcovr, and ReportGenerator. The first phase checks exact +// outcomes one run at a time: +// - gcovr's report is read when both utilities write one; +// - counters reset before each request by default, and accumulate when that's turned off; +// - gcovr leaves out a source above the project directory, outside its report root; +// - ReportGenerator's report is read when it is the only one, and covers that source at its own path. +// The second phase overlaps coverage runs with plain runs, discovery, reparses, and clean, some +// cancelled, and checks that every completed coverage run passes and yields a report. +// +// Images without ReportGenerator (Ceedling 1.0.0's) skip its checks. +// +// usage: node coverage.js [seconds=60] [maxInFlight=5] [seed=1] +import { coverageMode } from '../../src/ceedlingEngine'; +import { churn, describeOutcome, fileOf, hasCommand, hits, leftovers, logSummary, main, makeEngine, only, parseArgs, recordingReporter, seeded, stage, token, WS } from './harness'; + +main('coverage', async (checks) => { + const { numbers: [seconds, maxInFlight, seed] } = parseArgs([60, 5, 1]); + stage('tests/validation/workspaces/coverage'); + const reportGenerator = hasCommand('reportgenerator'); + const both = reportGenerator ? `${WS}/mixins/both.yml` : `${WS}/mixins/gcovr.yml`; + const settings: Record = { projects: [{ path: 'project', debugLaunchConfig: 'dbg' }], mixins: [both], coverageResetsCounters: true }; + const { engine, logs } = makeEngine(settings); + const { random, pick } = seeded(seed); + + let project = only(await engine.discover()); + const calculator = fileOf(project, 'test/test_calculator.c'), clamp = fileOf(project, 'test/test_clamp.c'); + const step = async (label: string, run: (reporter: ReturnType) => Promise, passing: string[]) => { + const reporter = recordingReporter(); + await run(reporter); + console.log(`${label}: ${reporter.outcomes.map(describeOutcome).join(' | ')}`); + for (const id of passing) checks.expect(reporter.results.get(id) === 'passed', `${label}: ${id} ${reporter.results.get(id)}`); + return reporter.last(); + }; + const ADD = 'test/test_calculator.c::test_add', CLAMP = 'test/test_clamp.c::test_clamp_low'; + + let outcome = await step('C1 file, gcovr preferred', (r) => engine.runFile('project', calculator, r, token(), coverageMode()), [ADD]); + checks.expect(outcome?.kind === 'report' && outcome.path.endsWith('gcovr/GcovCoverageCobertura.xml'), 'C1: expected the gcovr report'); + checks.expect(hits(outcome, 'src/Calculator.c', 5) === 1 && hits(outcome, 'src/Calculator.c', 10) === 0, 'C1: Calculator.c hits'); + + outcome = await step('C2 the same file again, counters reset', (r) => engine.runFile('project', calculator, r, token(), coverageMode()), [ADD]); + checks.expect(hits(outcome, 'src/Calculator.c', 5) === 1, 'C2: a reset should leave 1 hit'); + + settings.coverageResetsCounters = false; + outcome = await step('C3 the same file again, counters kept', (r) => engine.runFile('project', calculator, r, token(), coverageMode()), [ADD]); + checks.expect(hits(outcome, 'src/Calculator.c', 5) === 2, 'C3: kept counters should add up to 2 hits'); + settings.coverageResetsCounters = true; + + await step('C4 function', (r) => engine.runFunction('project', calculator, calculator.functions[0], r, token(), coverageMode()), [ADD]); + + outcome = await step('C5 file whose source is above the project', (r) => engine.runFile('project', clamp, r, token(), coverageMode()), [CLAMP]); + checks.expect(outcome?.kind === 'report' && hits(outcome, 'lib/Clamp.c', 5) === undefined, 'C5: gcovr should leave out lib/Clamp.c'); + + outcome = await step('C6 project', (r) => engine.runProjectAll('project', project.files, r, token(), coverageMode()), [ADD, CLAMP]); + checks.expect(hits(outcome, 'src/Calculator.c', 5) === 1, 'C6: Calculator.c hits'); + + if (reportGenerator) { + settings.mixins = [`${WS}/mixins/reportgenerator.yml`]; + project = only(await engine.discover()); + outcome = await step('C7 project, ReportGenerator only', (r) => engine.runProjectAll('project', project.files, r, token(), coverageMode()), [ADD, CLAMP]); + checks.expect(outcome?.kind === 'report' && outcome.path.endsWith('ReportGenerator/Cobertura.xml'), 'C7: expected the ReportGenerator report'); + checks.expect(hits(outcome, `${WS}/lib/Clamp.c`, 5) === 1 && hits(outcome, `${WS}/project/src/Calculator.c`, 5) === 1, 'C7: hits by absolute path'); + settings.mixins = [both]; + } else { + console.log('C7 skipped: this image has no ReportGenerator'); + } + + project = only(await engine.discover()); + const counts: Record = {}; + await churn(seconds, maxInFlight, async () => { + const kind = pick(['coverFile', 'coverFile', 'coverFunction', 'coverProject', 'runFile', 'discover', 'reparse', 'clean']); + counts[kind] = (counts[kind] ?? 0) + 1; + const target = pick(project.files); + const t = token(); + if (random() < 0.15) setTimeout(() => t.fire(), Math.floor(random() * 3000)); + const reporter = recordingReporter(); + try { + if (kind === 'coverFile') await engine.runFile('project', target, reporter, t, coverageMode()); + else if (kind === 'coverFunction') await engine.runFunction('project', target, target.functions[0], reporter, t, coverageMode()); + else if (kind === 'coverProject') await engine.runProjectAll('project', project.files, reporter, t, coverageMode()); + else if (kind === 'runFile') await engine.runFile('project', target, reporter, t); + else if (kind === 'discover') { project = only(await engine.discover()); return; } + else if (kind === 'reparse') { await engine.rediscoverFile('project', target.id); return; } + else { await engine.clean(t); return; } + if (t.isCancellationRequested) return; + for (const [id, result] of reporter.results) checks.expect(result === 'passed', `${kind} ${target.id}: ${id} ${result}`); + if (kind.startsWith('cover')) { + checks.expect(reporter.outcomes.length > 0, `${kind} ${target.id}: no coverage outcome`); + // After a clean, a run of only test_clamp.c covers only lib/Clamp.c, which gcovr leaves out, + // so its report has no files. + for (const o of reporter.outcomes) checks.expect(o.kind === 'report' || o.message.includes('covers no source files'), `${kind} ${target.id}: ${describeOutcome(o)}`); + } + } catch (error) { + checks.fail(`${kind} threw: ${error instanceof Error ? error.stack : String(error)}`); + } + }); + + for (const leftover of await leftovers()) checks.fail(`left behind: ${leftover}`); + engine.dispose(); + console.log(`overlap: ${seconds} s, ${maxInFlight} at once, seed ${seed}; operations ${JSON.stringify(counts)}`); + for (const line of logSummary(logs)) console.log(` log ${line}`); +}); diff --git a/tests/validation/gated.ts b/tests/validation/gated.ts new file mode 100644 index 0000000..25b4b81 --- /dev/null +++ b/tests/validation/gated.ts @@ -0,0 +1,147 @@ +// What the engine does differently per Ceedling version, on a project holding same-named test +// files, test/unit/test_dup.c and test/integration/test_dup.c. Each step records the Ceedling +// command lines the engine issued and the warnings it logged, then checks them with the results +// and coverage hits: +// - From 1.2.1, coverage tasks name the test file by path, and results and hits are exact. +// - On 1.2.0, coverage tasks can name only the bare filename, so the engine warns that Ceedling +// can't tell the two files apart, and Ceedling stops with an ambiguous file reference error. +// Only the commands and the warning are checked. +// Plain runs name the file by path, and debug builds land in the test's own build directory, +// on both. Ceedling before 1.2.0 can't build same-named test files at all, so the suite skips. +// +// usage: node gated.js +import * as fs from 'fs'; +import semver from 'semver'; +import { coverageMode } from '../../src/ceedlingEngine'; +import type { CoverageOutcome } from '../../src/testModel'; +import { atLeast, ceedlingVersion, describeOutcome, fileOf, hits, main, makeEngine, only, recordingReporter, stage, token, WS } from './harness'; + +const UNIT = 'test/unit/test_dup.c', INTEGRATION = 'test/integration/test_dup.c', FILTERS = 'test/test_filters.c'; +const ADD = `${UNIT}::test_unit_add`, LOW = `${INTEGRATION}::test_integration_clamp_low`, HIGH = `${INTEGRATION}::test_integration_clamp_high`; +const ALPHA = `${FILTERS}::test_filter_alpha`, BETA = `${FILTERS}::test_filter_beta`; + +main('gated', async (checks) => { + if (!atLeast('1.2.0', "can't build same-named test files")) return; + const version = ceedlingVersion(); + const pathTasks = semver.gte(version, '1.2.1'); + console.log(`Ceedling ${version}: coverage tasks name ${pathTasks ? 'paths' : 'bare filenames'}`); + stage('tests/validation/workspaces/gated'); + const settings: Record = { projects: [{ path: 'project', debugLaunchConfig: 'dbg' }], mixins: [`${WS}/mixins/gcovr.yml`], coverageResetsCounters: true }; + const { engine, logs } = makeEngine(settings); + + let project = only(await engine.discover()); + // Ceedling commands and warnings logged since the last call, without discovery's own commands. + let mark = logs.length; + const since = () => { + const recent = logs.slice(mark); + mark = logs.length; + const commands = recent.filter((l) => l.level === 'debug' && l.message.startsWith('ceedling: ')) + .map((l) => l.message.replace(/^ceedling: ceedling /, '').replace(/ \(cwd=.*$/, '').replace(/ --project .*$/, '')) + .filter((c) => !/^(version|dumpconfig|files:test)/.test(c)); + const warnings = recent.filter((l) => l.level === 'warn' || l.level === 'showWarning').map((l) => l.message); + return { commands, warnings }; + }; + const rediscover = async (mixin: string) => { + settings.mixins = [`${WS}/mixins/${mixin}`]; + project = only(await engine.discover()); + since(); + }; + const file = (id: string) => fileOf(project, id); + const fn = (fileId: string, id: string) => file(fileId).functions.find((f) => f.id === id)!; + + type Step = { results: Map, commands: string[], warnings: string[], outcome: CoverageOutcome | undefined }; + const step = async (label: string, work: (reporter: ReturnType) => Promise, check: (s: Step) => void) => { + const reporter = recordingReporter(); + await work(reporter); + const { commands, warnings } = since(); + console.log(`${label}\n commands: ${commands.join(' ; ')}\n results: ${[...reporter.results].map(([id, r]) => `${id.split('::')[1]}=${r}`).join(', ')}` + + (reporter.outcomes.length ? `\n coverage: ${reporter.outcomes.map(describeOutcome).join(' | ')}` : '') + + (warnings.length ? `\n warnings: ${warnings.map((w) => w.slice(0, 200)).join(' | ')}` : '')); + check({ results: reporter.results, commands, warnings, outcome: reporter.last() }); + }; + const results = (label: string, got: Map, want: Record) => { + for (const [id, result] of Object.entries(want)) checks.expect(got.get(id) === result, `${label}: ${id} expected ${result}, got ${got.get(id)}`); + for (const id of got.keys()) checks.expect(id in want, `${label}: unexpected result for ${id}`); + }; + const oneCommand = (label: string, commands: string[], expected: string) => { + checks.expect(commands.length === 1, `${label}: expected one build, got ${commands.length}`); + checks.expect(commands[0]?.startsWith(expected) === true, `${label}: expected '${expected}…', got '${commands[0]}'`); + }; + const coverTask = (id: string, extra = '') => `gcov:${pathTasks ? id : id.split('/').pop()}${extra} --force-test-rerun`; + const ambiguity = (label: string, warnings: string[]) => { + const warned = warnings.some((w) => w.includes("can't tell apart")); + checks.expect(warned !== pathTasks, `${label}: ambiguity warning ${pathTasks ? 'logged' : 'missing'}`); + }; + + await step(`G1 cover ${UNIT}`, (r) => engine.runFile('project', file(UNIT), r, token(), coverageMode()), (s) => { + oneCommand('G1', s.commands, coverTask(UNIT)); + ambiguity('G1', s.warnings); + if (!pathTasks) return; + results('G1', s.results, { [ADD]: 'passed' }); + checks.expect(hits(s.outcome, 'src/Calculator.c', 5) === 1 && !hits(s.outcome, 'src/Clamp.c', 6), 'G1: hits'); + }); + await step(`G2 cover ${INTEGRATION}`, (r) => engine.runFile('project', file(INTEGRATION), r, token(), coverageMode()), (s) => { + oneCommand('G2', s.commands, coverTask(INTEGRATION)); + ambiguity('G2', s.warnings); + if (!pathTasks) return; + results('G2', s.results, { [LOW]: 'passed', [HIGH]: 'passed' }); + checks.expect(hits(s.outcome, 'src/Clamp.c', 6) === 1 && hits(s.outcome, 'src/Clamp.c', 8) === 1, 'G2: Clamp.c hits'); + checks.expect(!hits(s.outcome, 'src/Calculator.c', 5), 'G2: counters were reset, so Calculator.c should be unexecuted'); + }); + await step('G3 cover test_integration_clamp_high', (r) => engine.runFunction('project', file(INTEGRATION), fn(INTEGRATION, HIGH), r, token(), coverageMode()), (s) => { + oneCommand('G3', s.commands, coverTask(INTEGRATION, ' --test-case=test_integration_clamp_high')); + if (!pathTasks) return; + results('G3', s.results, { [HIGH]: 'passed' }); + checks.expect(hits(s.outcome, 'src/Clamp.c', 8) === 1 && !hits(s.outcome, 'src/Clamp.c', 6), 'G3: only the high branch should be hit'); + }); + // Ceedling before 1.2.1 ignores a changed --test-case filter in a delta build (#1360). + await step('G4 cover test_integration_clamp_low, a changed filter', (r) => engine.runFunction('project', file(INTEGRATION), fn(INTEGRATION, LOW), r, token(), coverageMode()), (s) => { + oneCommand('G4', s.commands, coverTask(INTEGRATION, ' --test-case=test_integration_clamp_low')); + if (!pathTasks) return; + results('G4', s.results, { [LOW]: 'passed' }); + checks.expect(hits(s.outcome, 'src/Clamp.c', 6) === 1 && !hits(s.outcome, 'src/Clamp.c', 8), 'G4: only the low branch should be hit'); + }); + await step('G5 cover the project', (r) => engine.runProjectAll('project', project.files, r, token(), coverageMode()), (s) => { + oneCommand('G5', s.commands, 'gcov:all --force-test-rerun'); + results('G5', s.results, { [ADD]: 'passed', [LOW]: 'passed', [HIGH]: 'passed', [ALPHA]: 'passed', [BETA]: 'failed' }); + checks.expect(hits(s.outcome, 'src/Calculator.c', 5) === 1 && hits(s.outcome, 'src/Clamp.c', 6) === 1 && hits(s.outcome, 'src/Clamp.c', 8) === 1, 'G5: hits'); + }); + + await rediscover('task.yml'); + await step(`G6 cover ${UNIT} with :report_task: TRUE`, (r) => engine.runFile('project', file(UNIT), r, token(), coverageMode()), (s) => { + oneCommand('G6', s.commands, coverTask(UNIT, ' report:gcov')); + if (!pathTasks) return; + results('G6', s.results, { [ADD]: 'passed' }); + checks.expect(hits(s.outcome, 'src/Calculator.c', 5) === 1, 'G6: Calculator.c hits'); + }); + + await rediscover('reportgenerator.yml'); + await step(`G7 cover ${INTEGRATION} with ReportGenerator`, (r) => engine.runFile('project', file(INTEGRATION), r, token(), coverageMode()), (s) => { + oneCommand('G7', s.commands, coverTask(INTEGRATION)); + if (!pathTasks) return; + checks.expect(s.outcome?.kind === 'report' && s.outcome.path.endsWith('ReportGenerator/Cobertura.xml'), 'G7: expected the ReportGenerator report'); + results('G7', s.results, { [LOW]: 'passed', [HIGH]: 'passed' }); + checks.expect(hits(s.outcome, 'src/Clamp.c', 6) === 1 && hits(s.outcome, 'src/Clamp.c', 8) === 1, 'G7: Clamp.c hits'); + }); + + await rediscover('gcovr.yml'); + await step(`G8 run ${UNIT}`, (r) => engine.runFile('project', file(UNIT), r, token()), (s) => { + oneCommand('G8', s.commands, `test:${UNIT} --force-test-rerun`); + results('G8', s.results, { [ADD]: 'passed' }); + }); + for (const [label, id, want] of [['G9a', ALPHA, 'passed'], ['G9b', BETA, 'failed'], ['G9c', ALPHA, 'passed']] as const) { + const name = id.split('::')[1]; + // eslint-disable-next-line no-await-in-loop -- each run's filter must follow the one before + await step(`${label} run ${name}`, (r) => engine.runFunction('project', file(FILTERS), fn(FILTERS, id), r, token()), (s) => { + oneCommand(label, s.commands, `test:${FILTERS} --test-case=${name} --force-test-rerun`); + results(label, s.results, { [id]: want }); + }); + } + await step(`G10 debug build ${INTEGRATION}`, async (r) => { + const executable = await engine.compileForDebug('project', file(INTEGRATION), r, token()); + console.log(` executable: ${executable}`); + checks.expect(executable === 'integration/test_dup/test_dup.out' && fs.existsSync(`${WS}/project/build/test/out/${executable}`), `G10: executable ${executable}`); + }, (s) => results('G10', s.results, { [LOW]: 'passed', [HIGH]: 'passed' })); + + engine.dispose(); +}); diff --git a/tests/validation/harness.ts b/tests/validation/harness.ts new file mode 100644 index 0000000..40951c8 --- /dev/null +++ b/tests/validation/harness.ts @@ -0,0 +1,216 @@ +// What the validation suites share. Each suite drives the compiled CeedlingEngine against a real +// Ceedling release inside a vscode-ceedling-validate image (see run.sh), with VS Code's side of the +// engine stubbed: settings, the logger, and the Problems panel. A suite builds its projects in /ws, +// inside the container, from fixtures in the read-only repo, then exits 0 only when every check held. +import * as child_process from 'child_process'; +import * as fs from 'fs'; +import * as path from 'path'; +import semver from 'semver'; +import type vscode from 'vscode'; +import { CeedlingEngine } from '../../src/ceedlingEngine'; +import type { Cancellation } from '../../src/ceedlingProcess'; +import type { Logger } from '../../src/logger'; +import type { ProblemMatcher } from '../../src/problemMatcher'; +import type { CeedlingFileNode, CeedlingProjectNode, CoverageOutcome, RunReporter } from '../../src/testModel'; + +export const WS = '/ws'; +// Compiled suites live in out-test/tests/validation. +const REPO = path.resolve(__dirname, '../../..'); + +type LogCall = { level: string, message: string }; + +// An engine for the workspace at /ws. `settings` stays live: a suite changes a setting by +// assigning to it, as a user editing settings.json would. +export function makeEngine(settings: Record): { engine: CeedlingEngine, logs: LogCall[] } { + const logs: LogCall[] = []; + const record = (level: string) => (...args: unknown[]) => { logs.push({ level, message: args.map(String).join(' ') }); }; + const logger = { + trace: record('trace'), debug: record('debug'), info: record('info'), warn: record('warn'), error: record('error'), + showWarning: record('showWarning'), showError: record('showError'), dispose() { }, + } as unknown as Logger; + const problems = { match: () => [], set() { }, retain() { }, retainIds() { }, clear() { }, dispose() { } } as unknown as ProblemMatcher; + const engine = new CeedlingEngine( + { uri: { fsPath: WS, toString: () => `file://${WS}` }, name: 'ws', index: 0 } as unknown as vscode.WorkspaceFolder, + logger, + false, + { + configProvider: () => ({ get: (key: string, fallback?: unknown) => (key in settings ? settings[key] : fallback) }) as unknown as vscode.WorkspaceConfiguration, + problemMatcher: problems, + bundledMixinPath: path.join(REPO, 'assets/mixins/report-tests-log-factory.yml'), + }, + ); + return { engine, logs }; +} + +// Copies a fixture directory into /ws. `from` is relative to the repo. +export function stage(from: string, to = '.'): void { + fs.cpSync(path.join(REPO, from), path.join(WS, to), { recursive: true, filter: (source) => path.basename(source) !== 'build' }); +} + +// The suite's numeric arguments, each a positive number, else its default, and its --flags. A +// mistyped argument stops the suite, rather than letting it pass having run nothing. +export function parseArgs(defaults: readonly number[], flags: readonly string[] = []): { numbers: number[], flags: Set } { + const args = process.argv.slice(2); + const given = new Set(args.filter((a) => a.startsWith('--'))); + for (const flag of given) if (!flags.includes(flag)) throw new Error(`unknown option ${flag}`); + const values = args.filter((a) => !a.startsWith('--')); + if (values.length > defaults.length) throw new Error(`expected at most ${defaults.length} numbers, got: ${values.join(' ')}`); + const numbers = defaults.map((fallback, i) => { + if (i >= values.length) return fallback; + const value = Number(values[i]); + if (!Number.isFinite(value) || value <= 0) throw new Error(`expected a positive number, got '${values[i]}'`); + return value; + }); + return { numbers, flags: given }; +} + +export function ceedlingVersion(): string { + const output = child_process.execFileSync('ceedling', ['version'], { encoding: 'utf8' }); + const version = semver.coerce(/Ceedling\s*=>\s*(\S+)/.exec(output)?.[1]); + if (!version) throw new Error(`no Ceedling version in: ${output}`); + return version.version; +} + +// Whether the image's Ceedling is `minimum` or later. If not, says why the suite skips. +export function atLeast(minimum: string, reason: string): boolean { + const version = ceedlingVersion(); + if (semver.gte(version, minimum)) return true; + console.log(`skipped: Ceedling ${version} ${reason}`); + return false; +} + +export function fileOf(project: CeedlingProjectNode, id: string): CeedlingFileNode { + const file = project.files.find((f) => f.id === id); + if (!file) throw new Error(`discovery found no ${id} in ${project.projectKey}`); + return file; +} + +export function hasCommand(name: string): boolean { + return child_process.spawnSync('sh', ['-c', `command -v ${name}`]).status === 0; +} + +export function token(): Cancellation & { fire(): void } { + const listeners = new Set<() => void>(); + const t = { + isCancellationRequested: false, + onCancellationRequested(listener: () => void) { + listeners.add(listener); + return { dispose: () => listeners.delete(listener) }; + }, + fire() { t.isCancellationRequested = true; for (const l of [...listeners]) l(); }, + }; + return t; +} + +// Xorshift, so one seed always replays the same sequence of operations. +export function seeded(seed: number): { random: () => number, pick: (list: readonly T[]) => T } { + let state = seed >>> 0 || 1; + const random = () => { + state ^= state << 13; state >>>= 0; + state ^= state >>> 17; + state ^= state << 5; state >>>= 0; + return state / 4294967296; + }; + return { random, pick: (list) => list[Math.floor(random() * list.length)] }; +} + +// The one project a suite's workspace holds. +export function only(projects: readonly T[]): T { + if (projects.length !== 1) throw new Error(`discovery found ${projects.length} projects, not 1`); + return projects[0]; +} + +// A reporter that keeps each id's last result and every coverage outcome. +export function recordingReporter(): RunReporter & { results: Map, outcomes: CoverageOutcome[], last(): CoverageOutcome | undefined } { + const results = new Map(); + const outcomes: CoverageOutcome[] = []; + return { + results, outcomes, + last: () => outcomes.length ? outcomes[outcomes.length - 1] : undefined, + started() { }, + passed: (id) => results.set(id, 'passed'), + failed: (id) => results.set(id, 'failed'), + skipped: (id) => results.set(id, 'skipped'), + errored: (id, message) => results.set(id, `errored: ${message.split('\n')[0]}`), + appendOutput() { }, + coverage: (outcome) => outcomes.push(outcome), + }; +} + +// Hits on one line of one covered file, by the end of its path. +export function hits(outcome: CoverageOutcome | undefined, file: string, line: number): number | undefined { + if (outcome?.kind !== 'report') return undefined; + return outcome.files.find((f) => f.path.endsWith(file))?.lines.find((l) => l.line === line)?.hits; +} + +export function describeOutcome(outcome: CoverageOutcome): string { + if (outcome.kind === 'missing') return `missing: ${outcome.message.slice(0, 160)}`; + const files = outcome.files.map((f) => `${path.relative(WS, f.path)}[${f.lines.map((l) => `${l.line}:${l.hits}`).join(' ')}]`); + return `${path.basename(path.dirname(outcome.path))}/${path.basename(outcome.path)}: ${files.join(' ')}`; +} + +// Every broken expectation, plus any promise rejection nothing handled. +export class Checks { + readonly violations: string[] = []; + constructor() { + process.on('unhandledRejection', (reason) => this.fail(`unhandled rejection: ${reason instanceof Error ? reason.stack : String(reason)}`)); + } + fail(message: string): void { this.violations.push(message); } + expect(condition: boolean, message: string): void { if (!condition) this.fail(message); } +} + +// Runs `operation` repeatedly for `seconds`, with up to `maxInFlight` at once, then waits for all. +export async function churn(seconds: number, maxInFlight: number, operation: () => Promise): Promise { + const deadline = Date.now() + seconds * 1000; + const running = new Set>(); + while (Date.now() < deadline) { + while (running.size < maxInFlight) { + const next: Promise = operation().finally(() => running.delete(next)); + running.add(next); + } + // eslint-disable-next-line no-await-in-loop -- refill as each operation finishes + await Promise.race(running); + } + await Promise.allSettled([...running]); +} + +// Ceedling, test executables, or coverage tools still running once every operation has settled, +// and the engine's dumpconfig temp files still on disk. A cancelled run must leave neither. +export async function leftovers(): Promise { + await new Promise((resolve) => setTimeout(resolve, 2000)); + const processes = fs.readdirSync('/proc').filter((entry) => /^\d+$/.test(entry) && Number(entry) !== process.pid).flatMap((entry) => { + try { + const command = fs.readFileSync(`/proc/${entry}/cmdline`, 'utf8').replace(/\0/g, ' '); + return /ceedling|ruby|\.out\b|gcovr|reportgenerator/.test(command) ? [`process ${entry}: ${command.slice(0, 100)}`] : []; + } catch { + return []; // exited while being read + } + }); + const temps = fs.readdirSync('/tmp').filter((file) => file.startsWith('ceedling-dumpconfig-')).map((file) => `temp file /tmp/${file}`); + return [...processes, ...temps]; +} + +// Logged lines other than debug and trace, most frequent first. +export function logSummary(logs: readonly LogCall[], limit = 8): string[] { + const counts = new Map(); + for (const { level, message } of logs) { + if (level === 'debug' || level === 'trace') continue; + const key = `${level}: ${message.slice(0, 100)}`; + counts.set(key, (counts.get(key) ?? 0) + 1); + } + return [...counts].sort((a, b) => b[1] - a[1]).slice(0, limit).map(([line, count]) => `${count}x ${line}`); +} + +// Runs a suite, prints its verdict, and exits 0 only when it ran to the end with no violations. +export function main(name: string, suite: (checks: Checks) => Promise): void { + const checks = new Checks(); + suite(checks).then(() => { + const distinct = [...new Set(checks.violations)]; + console.log(`${name}: ${checks.violations.length} violations (${distinct.length} distinct)`); + for (const violation of distinct.slice(0, 30)) console.log(` - ${violation.slice(0, 400)}`); + process.exit(checks.violations.length ? 1 : 0); + }, (error: unknown) => { + console.log(`${name}: stopped: ${error instanceof Error ? error.stack : String(error)}`); + process.exit(1); + }); +} diff --git a/tests/validation/latency.ts b/tests/validation/latency.ts new file mode 100644 index 0000000..2e2dec6 --- /dev/null +++ b/tests/validation/latency.ts @@ -0,0 +1,41 @@ +// How long a rediscovery takes while a full build of the same project runs. The project's slow test +// keeps each build running about 4 seconds. Discovery waits for the build's locks, so it takes about +// as long as what's left of the build. A measurement for +// docs/Architecture.md's "Possible speedup"; it checks only that each build's results are unchanged. +// +// usage: node latency.js +import { main, makeEngine, only, recordingReporter, stage, token } from './harness'; + +// Each test's result, in id order, so runs compare whatever order their results arrive in. +const tally = (results: ReadonlyMap) => [...results].sort(([a], [b]) => a.localeCompare(b)).map(([id, r]) => `${id}=${r}`).join(', '); + +main('latency', async (checks) => { + stage('tests/validation/workspaces/latency'); + const { engine } = makeEngine({ projects: [{ path: 'project', debugLaunchConfig: 'dbg' }] }); + const project = only(await engine.discover()); + + const baseline = recordingReporter(); + let start = Date.now(); + await engine.runProjectAll('project', project.files, baseline, token()); + const buildMs = Date.now() - start; + start = Date.now(); + await engine.discover(); + console.log(`idle discovery ${Date.now() - start} ms; full build ${buildMs} ms`); + + for (let i = 0; i < 3; i++) { + const reporter = recordingReporter(); + const buildStart = Date.now(); + const run = engine.runProjectAll('project', project.files, reporter, token()).then(() => Date.now() - buildStart); + // eslint-disable-next-line no-await-in-loop -- each sample starts its own build, then rediscovers during it + await new Promise((resolve) => setTimeout(resolve, 300)); + const discoveryStart = Date.now(); + // eslint-disable-next-line no-await-in-loop -- as above + await engine.discover(); + const discoveryMs = Date.now() - discoveryStart; + // eslint-disable-next-line no-await-in-loop -- as above + const runMs = await run; + console.log(` discovery during a build: ${discoveryMs} ms (build ${runMs} ms)`); + checks.expect(tally(reporter.results) === tally(baseline.results), `sample ${i + 1}: results ${tally(reporter.results)}, expected ${tally(baseline.results)}`); + } + engine.dispose(); +}); diff --git a/tests/validation/layout.ts b/tests/validation/layout.ts new file mode 100644 index 0000000..617f62f --- /dev/null +++ b/tests/validation/layout.ts @@ -0,0 +1,60 @@ +// Test build directories in a project with two test roots, +:test/** and +:extra/**, each holding +// a test_calc.c. From 1.2.0, Ceedling names each test's build directory by its path below its own +// root, here unit/test_calc and integration/test_calc, and the engine must find each one exactly: +// - a debug build returns the executable it just built; +// - from 1.2.1, a coverage request whose --test-case filter also matches another test in the file +// reruns the whole file, clearing only that file's counters, so the other test_calc.c keeps its +// hits. Ceedling 1.2.0's gcov: tasks take only the bare filename, which it can't resolve here. +// Ceedling before 1.2.0 builds both files in one directory, so the suite skips. +// +// usage: node layout.js +import * as fs from 'fs'; +import semver from 'semver'; +import { coverageMode } from '../../src/ceedlingEngine'; +import { atLeast, ceedlingVersion, fileOf, main, makeEngine, only, recordingReporter, stage, token, WS } from './harness'; + +const BUILD_NAMES: Record = { 'test/unit/test_calc.c': 'unit/test_calc', 'extra/integration/test_calc.c': 'integration/test_calc' }; + +main('layout', async (checks) => { + if (!atLeast('1.2.0', 'builds same-named test files in one directory')) return; + stage('tests/validation/workspaces/layout'); + const { engine, logs } = makeEngine({ projects: [{ path: 'project', debugLaunchConfig: 'dbg' }], mixins: [`${WS}/mixins/gcovr.yml`] }); + const project = only(await engine.discover()); + const file = (id: string) => fileOf(project, id); + + for (const [id, buildName] of Object.entries(BUILD_NAMES)) { + const started = Date.now() - 1000; + // eslint-disable-next-line no-await-in-loop -- one build at a time, so each executable is this build's own + const executable = await engine.compileForDebug('project', file(id), recordingReporter(), token()); + const built = `${WS}/project/build/test/out/${executable}`; + const fresh = fs.existsSync(built) && fs.statSync(built).mtimeMs >= started; + console.log(`debug ${id}: ${executable} ${fresh ? '(just built)' : '(missing or stale)'}`); + checks.expect(executable === `${buildName}/test_calc.out` && fresh, `debug ${id}: expected a fresh ${buildName}/test_calc.out, got ${executable}`); + } + + if (semver.lt(ceedlingVersion(), '1.2.1')) { + console.log("coverage skipped: Ceedling 1.2.0's gcov: tasks can't name either test_calc.c"); + engine.dispose(); + return; + } + const pairs = [['extra/integration/test_calc.c', 'test/unit/test_calc.c', 'test_add'], ['test/unit/test_calc.c', 'extra/integration/test_calc.c', 'test_clamp']] as const; + for (const [first, second, name] of pairs) { + const mode = coverageMode(); + const reporter = recordingReporter(); + // eslint-disable-next-line no-await-in-loop -- the two runs share one coverage request, in order + await engine.runFile('project', file(first), reporter, token(), mode); + const target = file(second); + const mark = logs.length; + // eslint-disable-next-line no-await-in-loop -- as above + await engine.runFunction('project', target, target.functions.find((f) => f.id === `${second}::${name}`)!, reporter, token(), mode); + const fellBack = logs.slice(mark).some((l) => l.level === 'warn' && l.message.includes('also matched an unrelated test')); + checks.expect(fellBack, `cover ${second} --test-case=${name}: expected the filter to collide and the whole file to rerun`); + const last = reporter.last(); + const linesHit = (source: string) => last?.kind === 'report' ? last.files.find((f) => f.path.endsWith(source))?.lines.filter((l) => l.hits > 0).length ?? 0 : 0; + const failures = [...reporter.results].filter(([, result]) => result !== 'passed'); + console.log(`cover ${first}, then ${second} --test-case=${name}: Calculator.c lines hit ${linesHit('Calculator.c')}, Clamp.c lines hit ${linesHit('Clamp.c')}`); + checks.expect(linesHit('Calculator.c') > 0 && linesHit('Clamp.c') > 0, `cover ${first}, then ${second}: the other test_calc.c lost its counters`); + checks.expect(failures.length === 0, `cover ${first}, then ${second}: ${failures.map(([id, result]) => `${id} ${result}`).join(', ')}`); + } + engine.dispose(); +}); diff --git a/tests/validation/probes/fixes.sh b/tests/validation/probes/fixes.sh new file mode 100755 index 0000000..053334c --- /dev/null +++ b/tests/validation/probes/fixes.sh @@ -0,0 +1,39 @@ +#!/bin/sh +# Ceedling's own behavior behind three of the engine's choices, from the command line alone: +# P1 A delta build reruns a changed --test-case filter from 1.2.1 (#1360). Before, it reruns +# the previous filter's tests, so the engine passes --force-test-rerun with a filter. +# P2 A delta build skips a test whose coverage counters were deleted, leaving a report with +# every hit at 0, so every coverage run passes --force-test-rerun after its reset. +# P3 dumpconfig and files:test rewrite files under build/ before 1.2.1 (#1361), so discovery +# holds the project's locks. +# Expected output is in each heading; the probe reports, it doesn't judge. P3 needs Ceedling 1.2.0 +# or later, for dumpconfig --stdout. +set -u +mkdir -p /ws && cp -R /workspace/tests/validation/workspaces/gated/. /ws/ +cd /ws/project +M="--mixin /ws/mixins/gcovr.yml --mixin /workspace/assets/mixins/report-tests-log-factory.yml" +names() { grep -o '[^<]*' build/artifacts/test/cppunit_tests_report.xml 2>/dev/null | sed 's/<[^>]*>//g' | tr '\n' ' '; echo; } +counters() { find build/gcov/out -name '*.gcda' 2>/dev/null | wc -l; } +hit() { grep -o 'line number="10" hits="[0-9]*"' build/artifacts/gcov/gcovr/GcovCoverageCobertura.xml 2>/dev/null | head -1; } +reset() { find build/gcov/out \( -name '*.gcda' -o -name '*.gcov' \) -delete; rm -f build/artifacts/gcov/gcovr/*.xml; } +ceedling version | grep 'Ceedling =>' + +echo "== P1: alternating --test-case filters, no --force-test-rerun (1.2.1 and later: alpha, beta, both)" +for filter in "--test-case=test_filter_alpha" "--test-case=test_filter_beta" ""; do + rm -f build/artifacts/test/cppunit_tests_report.xml + ceedling test:test_filters.c $filter $M >/dev/null 2>&1 + printf ' %-32s -> ' "${filter:-(no filter)}"; names +done + +echo "== P2: counters deleted, then gcov: without and with --force-test-rerun (without: no counters or 0 hits)" +ceedling gcov:test_filters.c $M >/dev/null 2>&1; echo " fresh build: counters=$(counters) $(hit)" +reset; ceedling gcov:test_filters.c $M >/dev/null 2>&1; echo " reset: counters=$(counters) $(hit)" +reset; ceedling gcov:test_filters.c --force-test-rerun $M >/dev/null 2>&1; echo " reset, forced: counters=$(counters) $(hit)" + +echo "== P3: files under build/ that dumpconfig --stdout and files:test change (1.2.1 and later: none)" +snapshot() { find build -type f -exec stat -c '%i %Y %s %n' {} + | sort -k4; } +snapshot > /tmp/before; sleep 1.1 +ceedling dumpconfig --stdout $M >/dev/null 2>&1; ceedling files:test $M >/dev/null 2>&1 +snapshot > /tmp/after +echo " build files: $(wc -l < /tmp/before); changed or new: $(diff /tmp/before /tmp/after | grep -c '^>')" +diff /tmp/before /tmp/after | grep '^>' | awk '{print " ", $5}' | head -12 diff --git a/tests/validation/probes/overlap.sh b/tests/validation/probes/overlap.sh new file mode 100755 index 0000000..dc61496 --- /dev/null +++ b/tests/validation/probes/overlap.sh @@ -0,0 +1,25 @@ +#!/bin/sh +# Discovery's commands looping while clean test:all builds run in the same project directory, +# counting the builds that fail. Before 1.2.1, discovery truncates and rewrites the vendored +# framework sources mid-build (#1361), the hazard the engine's locks prevent. The rewritten files +# are identical, so a build fails only if the compiler reads one while it is empty, and trials +# have seen every build pass on 1.2.0 and 1.2.1 alike. A pass here doesn't make the race safe. +set -u +mkdir -p /ws/a && cp -R /workspace/tests/manual/. /ws/a/ && rm -rf /ws/a/build +cd /ws/a +M="--mixin /workspace/assets/mixins/report-tests-log-factory.yml" +ceedling version | grep 'Ceedling =>' +failed=0 +for i in 1 2 3 4 5 6; do + ceedling clobber $M >/dev/null 2>&1 + ( while [ ! -f /tmp/stop ]; do ceedling files:test $M >/dev/null 2>&1; ceedling dumpconfig --stdout $M >/dev/null 2>&1; done ) & + loop=$! + ceedling test:all $M > /tmp/build.$i 2>&1 + touch /tmp/stop; wait $loop; rm -f /tmp/stop + if grep -q -E "error:|EXCEPTION|could not complete" /tmp/build.$i; then + failed=$((failed + 1)); echo " build $i: failed"; grep -m3 -E "error:|EXCEPTION" /tmp/build.$i | cut -c1-160 + else + echo " build $i: ok" + fi +done +echo "builds that failed: $failed of 6" diff --git a/tests/validation/probes/syscalls.sh b/tests/validation/probes/syscalls.sh new file mode 100755 index 0000000..72d7c1b --- /dev/null +++ b/tests/validation/probes/syscalls.sh @@ -0,0 +1,23 @@ +#!/bin/sh +# Every filesystem call discovery's commands make under a build root that a full test and coverage +# build has populated. From 1.2.1 there should be no writes, which is what would let discovery +# run without the project's locks (docs/Architecture.md, "Possible speedup"). run.sh runs this +# probe as root, since strace needs ptrace. It needs Ceedling 1.2.0 or later, for dumpconfig --stdout. +set -u +mkdir -p /ws && cp -R /workspace/tests/validation/workspaces/gated/. /ws/ +cd /ws/project +M="--project project.yml --mixin /ws/mixins/gcovr.yml --mixin /workspace/assets/mixins/report-tests-log-factory.yml" +ceedling version | grep 'Ceedling =>' +ceedling test:all $M >/dev/null 2>&1; ceedling gcov:all $M >/dev/null 2>&1 +echo "build files: $(find build -type f | wc -l), directories: $(find build -type d | wc -l)" +CALLS=open,openat,creat,mkdir,mkdirat,rename,renameat,renameat2,unlink,unlinkat,rmdir,utimensat,truncate,ftruncate,chmod,fchmodat,symlink,symlinkat,link,linkat,stat,newfstatat,statx,access,faccessat,faccessat2,getdents64,chdir +for command in "dumpconfig --stdout" "files:test"; do + strace -f -qq -o /tmp/trace -e trace=$CALLS ceedling $command $M >/dev/null 2>&1 + grep '/ws/project/build\|"build' /tmp/trace > /tmp/hits + writes=$(grep -E 'O_WRONLY|O_RDWR|O_CREAT|O_TRUNC|mkdir|rename|unlink|rmdir|utimensat|truncate|chmod|symlink|link\(' /tmp/hits | grep -v ENOENT) + reads=$(grep -E 'O_RDONLY' /tmp/hits | grep -v ENOENT) + echo "== ceedling $command: $(wc -l < /tmp/hits) calls touch build/; writes: $(printf '%s' "$writes" | grep -c .); file reads: $(printf '%s' "$reads" | grep -c .)" + printf '%s\n' "$writes" | grep . | sed 's/^[0-9]* */ W /' | cut -c1-150 | head -8 + printf '%s\n' "$reads" | grep . | sed 's/^[0-9]* */ R /' | cut -c1-150 | head -8 + grep -v -E 'O_RDONLY|O_WRONLY|O_RDWR' /tmp/hits | sed -E 's/^[0-9]+ +([a-z0-9_]+)\(.*"([^"]*build[^"]*)".*\) = (.*)$/ \1 \2 = \3/' | sort | uniq -c | sort -rn | head -6 +done diff --git a/tests/validation/run.sh b/tests/validation/run.sh new file mode 100755 index 0000000..eaf0ce0 --- /dev/null +++ b/tests/validation/run.sh @@ -0,0 +1,56 @@ +#!/usr/bin/env bash +# Runs a validation suite or probe against a real Ceedling release in Docker. The suites drive the +# engine compiled into out-test/, with the sidecar's Node and node_modules volume, so build the +# tests in the sidecar first; `just validate` does both. See docs/Development.md. +# +# usage: run.sh [args...] +# a throwtheswitch/madsciencelab-plugins image tag, e.g. 1.2.1 +# stress, coverage, gated, layout, latency, all, or a probe: +# probe:fixes, probe:overlap, probe:syscalls +# [args...] passed to the suite (stress and coverage take seconds, maxInFlight, seed) +# The Justfile passes the sidecar's image and node_modules volume names; these defaults match it. +set -euo pipefail +version=${1:?usage: run.sh [args...]} +suite=${2:?usage: run.sh [args...]} +shift 2 +repo=$(cd "$(dirname "$0")/../.." && pwd) +image="vscode-ceedling-validate:$version" +sidecar_image=${SIDECAR_IMAGE:-vscode-ceedling-sidecar} +node_modules_volume=${NODE_MODULES_VOLUME:-vscode-ceedling-node-modules} +suites=(stress coverage gated layout latency) +case $suite in + all | probe:fixes | probe:overlap | probe:syscalls) ;; + *) [[ " ${suites[*]} " == *" $suite "* ]] || { echo "error: no suite or probe named '$suite'" >&2; exit 1; } ;; +esac + +if ! command -v docker >/dev/null; then + echo "error: the validation suites need Docker; run them from the host, not inside the sidecar" >&2 + exit 1 +fi +if [ ! -f "$repo/out-test/tests/validation/harness.js" ]; then + echo "error: out-test/ has no compiled suites; run 'just validate', or 'just _run \"npm run -s build:tests\"' first" >&2 + exit 1 +fi +docker build -q -t "$image" --build-arg CEEDLING="$version" --build-arg SIDECAR="$sidecar_image" -f "$repo/sidecar/validate.Dockerfile" "$repo/sidecar" >/dev/null + +# The repo and node_modules are read-only. Each run builds its projects in the container's own /ws, +# so nothing is left on the host. +container=(docker run --rm -e CEEDLING_DECORATORS=false -v "$repo":/workspace:ro -v "$node_modules_volume":/workspace/node_modules:ro) +run_suite() { + echo "=== Ceedling $version: $*" + "${container[@]}" --entrypoint node "$image" "/workspace/out-test/tests/validation/$1.js" "${@:2}" +} + +case $suite in + all) + status=0 + for one in "stress" "stress --shared-build-root" "coverage" "gated" "layout"; do + # shellcheck disable=SC2086 # each entry is a suite and its arguments + run_suite $one "$@" || status=1 + done + exit $status + ;; + probe:syscalls) "${container[@]}" --user root --entrypoint sh "$image" /workspace/tests/validation/probes/syscalls.sh ;; + probe:*) "${container[@]}" --entrypoint sh "$image" "/workspace/tests/validation/probes/${suite#probe:}.sh" ;; + *) run_suite "$suite" "$@" ;; +esac diff --git a/tests/validation/stress.ts b/tests/validation/stress.ts new file mode 100644 index 0000000..97ccbc6 --- /dev/null +++ b/tests/validation/stress.ts @@ -0,0 +1,117 @@ +// Concurrency stress. Random runs, debug builds, discoveries, reparses, and cleans overlap across +// four projects for a fixed time, a fifth of them cancelled at a random moment. Every completed +// run must report each of its tests exactly once, with the outcome tests/manual expects, and +// nothing may be left running or on disk afterward. +// +// The projects are three copies of tests/manual. The third holds two project files, so two +// projects share one directory. --shared-build-root also gives all four one build root. +// +// usage: node stress.js [seconds=60] [maxInFlight=6] [seed=1] [--shared-build-root] +import * as fs from 'fs'; +import * as path from 'path'; +import { leafIds, type CeedlingFileNode, type RunReporter } from '../../src/testModel'; +import { churn, type Checks, leftovers, logSummary, main, makeEngine, parseArgs, seeded, stage, token, WS } from './harness'; + +// The outcome of every test in tests/manual, by function name. +const EXPECTED: Record = { + test_add_should_ReturnSum: 'passed', + test_add_should_HandleSpaceBeforeParen: 'passed', + test_add_should_ReturnSumButIsDeliberatelyWrong: 'failed', + test_should_BeIgnored: 'skipped', + test_should_crash: 'failed', + test_add_ParameterizedCases: 'passed', + test_add_MatrixCases: 'passed', +}; +const expectedOf = (id: string) => EXPECTED[id.split('::')[1].replace(/\(.*$/, '')]; + +// Checks one run's reporting: results only for the run's own tests, none after the run returns, +// one per test, and in a run that wasn't cancelled, the expected outcome for every test. +function checkedReporter(checks: Checks, label: string, leaves: readonly string[]) { + const state = new Map(); + let finished = false; + const result = (kind: string) => (id: string, message?: string) => { + checks.expect(leaves.includes(id), `${label}: ${kind} result for ${id}, which the run didn't select`); + checks.expect(!finished, `${label}: ${kind} after the run returned (${id})`); + checks.expect(state.get(id) === undefined || state.get(id) === 'started', `${label}: second result for ${id}`); + state.set(id, kind === 'errored' ? `errored: ${String(message).split('\n').slice(0, 3).join(' | ')}` : kind); + }; + const reporter: RunReporter = { + started: (id) => { if (!state.has(id)) state.set(id, 'started'); }, + passed: result('passed'), failed: result('failed'), skipped: result('skipped'), errored: result('errored'), + appendOutput() { }, coverage() { }, + }; + const check = (cancelled: boolean) => { + finished = true; + for (const leaf of leaves) { + const got = state.get(leaf); + checks.expect(got !== 'started', `${label}: ${leaf} left without a result`); + if (!cancelled) checks.expect(got === expectedOf(leaf), `${label}: ${leaf} expected ${expectedOf(leaf)}, got ${got}`); + } + }; + return { reporter, check }; +} + +main('stress', async (checks) => { + const { numbers: [seconds, maxInFlight, seed], flags } = parseArgs([60, 6, 1], ['--shared-build-root']); + const sharedBuildRoot = flags.has('--shared-build-root'); + for (const project of ['a', 'b', 'c']) stage('tests/manual', project); + fs.writeFileSync(path.join(WS, 'c/ci.yml'), ':project:\n :use_test_preprocessor: :none\n'); + fs.writeFileSync(path.join(WS, 'c/other.yml'), ':unity:\n :use_param_tests: true\n'); + fs.writeFileSync(path.join(WS, 'shared-build.yml'), `:project:\n :build_root: ${WS}/shared-build\n`); + const projects = ['a', 'b', 'c/ci.yml', 'c/other.yml'].map((p) => ({ path: p, debugLaunchConfig: 'dbg' })); + const { engine, logs } = makeEngine({ projects, mixins: sharedBuildRoot ? [`${WS}/shared-build.yml`] : [] }); + const { random, pick } = seeded(seed); + const leavesOf = (files: readonly CeedlingFileNode[]) => files.flatMap((f) => leafIds(f.functions)); + + let tree = await engine.discover(); + console.log(`projects: ${tree.map((p) => `${p.projectKey} (${p.files.length} files)`).join(', ')}${sharedBuildRoot ? ', one build root' : ''}`); + checks.expect(tree.length === 4, `discovery found ${tree.length} projects`); + const counts: Record = {}; + + await churn(seconds, maxInFlight, async () => { + const kind = pick(['runFile', 'runFile', 'runFunction', 'runProject', 'debug', 'discover', 'discover', 'clean', 'reparse']); + counts[kind] = (counts[kind] ?? 0) + 1; + const project = pick(tree); + const file = pick(project.files); + const t = token(); + if (random() < 0.2) setTimeout(() => t.fire(), Math.floor(random() * 3000)); + const label = `${kind} ${project.projectKey}/${file.id}`; + try { + if (kind === 'runFile') { + const run = checkedReporter(checks, label, leavesOf([file])); + await engine.runFile(project.projectKey, file, run.reporter, t); + run.check(t.isCancellationRequested); + } else if (kind === 'runFunction') { + const fn = pick(file.functions); + const run = checkedReporter(checks, `${label}::${fn.id}`, leafIds([fn])); + await engine.runFunction(project.projectKey, file, fn, run.reporter, t); + run.check(t.isCancellationRequested); + } else if (kind === 'runProject') { + const run = checkedReporter(checks, label, leavesOf(project.files)); + await engine.runProjectAll(project.projectKey, project.files, run.reporter, t); + run.check(t.isCancellationRequested); + } else if (kind === 'debug') { + const run = checkedReporter(checks, label, leavesOf([file])); + await engine.compileForDebug(project.projectKey, file, run.reporter, t); + run.check(t.isCancellationRequested); + } else if (kind === 'discover') { + tree = await engine.discover(); + checks.expect(tree.length === 4, `discovery found ${tree.length} projects`); + } else if (kind === 'clean') { + checks.expect(await engine.clean(t) || t.isCancellationRequested, 'clean failed without being cancelled'); + } else { + await engine.rediscoverFile(project.projectKey, file.id); + } + } catch (error) { + checks.fail(`${label} threw: ${error instanceof Error ? error.stack : String(error)}`); + } + }); + + // The engine looks for a debug build's executable while it still holds the build's locks. Once + // they're released, another project's build or a clean may replace it, so only that check holds. + for (const missing of logs.filter((l) => l.message.startsWith('executableSubpath:'))) checks.fail(missing.message); + for (const leftover of await leftovers()) checks.fail(`left behind: ${leftover}`); + engine.dispose(); + console.log(`${seconds} s, ${maxInFlight} at once, seed ${seed}; operations ${JSON.stringify(counts)}`); + for (const line of logSummary(logs)) console.log(` log ${line}`); +}); diff --git a/tests/validation/workspaces/gated/mixins/gcovr.yml b/tests/validation/workspaces/gated/mixins/gcovr.yml new file mode 100644 index 0000000..d8cc6ac --- /dev/null +++ b/tests/validation/workspaces/gated/mixins/gcovr.yml @@ -0,0 +1,5 @@ +:gcov: + :utilities: + - gcovr + :reports: + - Cobertura diff --git a/tests/validation/workspaces/gated/mixins/reportgenerator.yml b/tests/validation/workspaces/gated/mixins/reportgenerator.yml new file mode 100644 index 0000000..af8adb6 --- /dev/null +++ b/tests/validation/workspaces/gated/mixins/reportgenerator.yml @@ -0,0 +1,5 @@ +:gcov: + :utilities: + - ReportGenerator + :reports: + - Cobertura diff --git a/tests/validation/workspaces/gated/mixins/task.yml b/tests/validation/workspaces/gated/mixins/task.yml new file mode 100644 index 0000000..b460392 --- /dev/null +++ b/tests/validation/workspaces/gated/mixins/task.yml @@ -0,0 +1,6 @@ +:gcov: + :report_task: TRUE + :utilities: + - gcovr + :reports: + - Cobertura diff --git a/tests/validation/workspaces/gated/project/project.yml b/tests/validation/workspaces/gated/project/project.yml new file mode 100644 index 0000000..cf17b18 --- /dev/null +++ b/tests/validation/workspaces/gated/project/project.yml @@ -0,0 +1,16 @@ +--- +:project: + :build_root: build + :use_test_preprocessor: :none + :use_backtrace: :simple +:plugins: + :enabled: + - gcov +:paths: + :test: + - "+:test/**" + :source: + - src/** + :include: + - src/** +... diff --git a/tests/validation/workspaces/gated/project/src/Calculator.c b/tests/validation/workspaces/gated/project/src/Calculator.c new file mode 100644 index 0000000..ac5ac6b --- /dev/null +++ b/tests/validation/workspaces/gated/project/src/Calculator.c @@ -0,0 +1,11 @@ +#include "Calculator.h" + +int add(int a, int b) +{ + return a + b; +} + +int subtract(int a, int b) +{ + return a - b; +} diff --git a/tests/validation/workspaces/gated/project/src/Calculator.h b/tests/validation/workspaces/gated/project/src/Calculator.h new file mode 100644 index 0000000..fa1989d --- /dev/null +++ b/tests/validation/workspaces/gated/project/src/Calculator.h @@ -0,0 +1,2 @@ +int add(int a, int b); +int subtract(int a, int b); diff --git a/tests/validation/workspaces/gated/project/src/Clamp.c b/tests/validation/workspaces/gated/project/src/Clamp.c new file mode 100644 index 0000000..12d3629 --- /dev/null +++ b/tests/validation/workspaces/gated/project/src/Clamp.c @@ -0,0 +1,10 @@ +#include "Clamp.h" + +int clamp(int value, int low, int high) +{ + if (value < low) + return low; + if (value > high) + return high; + return value; +} diff --git a/tests/validation/workspaces/gated/project/src/Clamp.h b/tests/validation/workspaces/gated/project/src/Clamp.h new file mode 100644 index 0000000..a3c8f76 --- /dev/null +++ b/tests/validation/workspaces/gated/project/src/Clamp.h @@ -0,0 +1 @@ +int clamp(int value, int low, int high); diff --git a/tests/validation/workspaces/gated/project/test/integration/test_dup.c b/tests/validation/workspaces/gated/project/test/integration/test_dup.c new file mode 100644 index 0000000..4f25a8c --- /dev/null +++ b/tests/validation/workspaces/gated/project/test/integration/test_dup.c @@ -0,0 +1,12 @@ +#include "unity.h" +#include "Clamp.h" + +void test_integration_clamp_low(void) +{ + TEST_ASSERT_EQUAL(0, clamp(-5, 0, 10)); +} + +void test_integration_clamp_high(void) +{ + TEST_ASSERT_EQUAL(10, clamp(50, 0, 10)); +} diff --git a/tests/validation/workspaces/gated/project/test/test_filters.c b/tests/validation/workspaces/gated/project/test/test_filters.c new file mode 100644 index 0000000..64bd1c9 --- /dev/null +++ b/tests/validation/workspaces/gated/project/test/test_filters.c @@ -0,0 +1,12 @@ +#include "unity.h" +#include "Calculator.h" + +void test_filter_alpha(void) +{ + TEST_ASSERT_EQUAL(1, subtract(3, 2)); +} + +void test_filter_beta(void) +{ + TEST_ASSERT_EQUAL(99, subtract(3, 2)); +} diff --git a/tests/validation/workspaces/gated/project/test/unit/test_dup.c b/tests/validation/workspaces/gated/project/test/unit/test_dup.c new file mode 100644 index 0000000..cb7cb7e --- /dev/null +++ b/tests/validation/workspaces/gated/project/test/unit/test_dup.c @@ -0,0 +1,7 @@ +#include "unity.h" +#include "Calculator.h" + +void test_unit_add(void) +{ + TEST_ASSERT_EQUAL(3, add(1, 2)); +} diff --git a/tests/validation/workspaces/latency/project/project.yml b/tests/validation/workspaces/latency/project/project.yml new file mode 100644 index 0000000..5b5aa07 --- /dev/null +++ b/tests/validation/workspaces/latency/project/project.yml @@ -0,0 +1,11 @@ +--- +:project: + :build_root: build +:paths: + :test: + - "+:test/**" + :source: + - src/** + :include: + - src/** +... diff --git a/tests/validation/workspaces/latency/project/src/Calculator.c b/tests/validation/workspaces/latency/project/src/Calculator.c new file mode 100644 index 0000000..ac5ac6b --- /dev/null +++ b/tests/validation/workspaces/latency/project/src/Calculator.c @@ -0,0 +1,11 @@ +#include "Calculator.h" + +int add(int a, int b) +{ + return a + b; +} + +int subtract(int a, int b) +{ + return a - b; +} diff --git a/tests/validation/workspaces/latency/project/src/Calculator.h b/tests/validation/workspaces/latency/project/src/Calculator.h new file mode 100644 index 0000000..fa1989d --- /dev/null +++ b/tests/validation/workspaces/latency/project/src/Calculator.h @@ -0,0 +1,2 @@ +int add(int a, int b); +int subtract(int a, int b); diff --git a/tests/validation/workspaces/latency/project/test/test_calculator.c b/tests/validation/workspaces/latency/project/test/test_calculator.c new file mode 100644 index 0000000..baeba70 --- /dev/null +++ b/tests/validation/workspaces/latency/project/test/test_calculator.c @@ -0,0 +1,7 @@ +#include "unity.h" +#include "Calculator.h" + +void test_add(void) +{ + TEST_ASSERT_EQUAL(3, add(1, 2)); +} diff --git a/tests/validation/workspaces/latency/project/test/test_slow.c b/tests/validation/workspaces/latency/project/test/test_slow.c new file mode 100644 index 0000000..29cc25a --- /dev/null +++ b/tests/validation/workspaces/latency/project/test/test_slow.c @@ -0,0 +1,9 @@ +#include "unity.h" +#include + +// Keeps each build running long enough for a rediscovery to start during it. +void test_slow(void) +{ + sleep(4); + TEST_PASS(); +} diff --git a/tests/validation/workspaces/layout/mixins/gcovr.yml b/tests/validation/workspaces/layout/mixins/gcovr.yml new file mode 100644 index 0000000..d8cc6ac --- /dev/null +++ b/tests/validation/workspaces/layout/mixins/gcovr.yml @@ -0,0 +1,5 @@ +:gcov: + :utilities: + - gcovr + :reports: + - Cobertura diff --git a/tests/validation/workspaces/layout/project/extra/integration/test_calc.c b/tests/validation/workspaces/layout/project/extra/integration/test_calc.c new file mode 100644 index 0000000..9503e01 --- /dev/null +++ b/tests/validation/workspaces/layout/project/extra/integration/test_calc.c @@ -0,0 +1,12 @@ +#include "unity.h" +#include "Clamp.h" + +void test_clamp(void) +{ + TEST_ASSERT_EQUAL(5, clamp(5, 0, 10)); +} + +void test_clamp_more(void) +{ + TEST_ASSERT_EQUAL(0, clamp(-5, 0, 10)); +} diff --git a/tests/validation/workspaces/layout/project/project.yml b/tests/validation/workspaces/layout/project/project.yml new file mode 100644 index 0000000..12c10ca --- /dev/null +++ b/tests/validation/workspaces/layout/project/project.yml @@ -0,0 +1,15 @@ +--- +:project: + :build_root: build +:plugins: + :enabled: + - gcov +:paths: + :test: + - "+:test/**" + - "+:extra/**" + :source: + - src/** + :include: + - src/** +... diff --git a/tests/validation/workspaces/layout/project/src/Calculator.c b/tests/validation/workspaces/layout/project/src/Calculator.c new file mode 100644 index 0000000..ac5ac6b --- /dev/null +++ b/tests/validation/workspaces/layout/project/src/Calculator.c @@ -0,0 +1,11 @@ +#include "Calculator.h" + +int add(int a, int b) +{ + return a + b; +} + +int subtract(int a, int b) +{ + return a - b; +} diff --git a/tests/validation/workspaces/layout/project/src/Calculator.h b/tests/validation/workspaces/layout/project/src/Calculator.h new file mode 100644 index 0000000..fa1989d --- /dev/null +++ b/tests/validation/workspaces/layout/project/src/Calculator.h @@ -0,0 +1,2 @@ +int add(int a, int b); +int subtract(int a, int b); diff --git a/tests/validation/workspaces/layout/project/src/Clamp.c b/tests/validation/workspaces/layout/project/src/Clamp.c new file mode 100644 index 0000000..12d3629 --- /dev/null +++ b/tests/validation/workspaces/layout/project/src/Clamp.c @@ -0,0 +1,10 @@ +#include "Clamp.h" + +int clamp(int value, int low, int high) +{ + if (value < low) + return low; + if (value > high) + return high; + return value; +} diff --git a/tests/validation/workspaces/layout/project/src/Clamp.h b/tests/validation/workspaces/layout/project/src/Clamp.h new file mode 100644 index 0000000..a3c8f76 --- /dev/null +++ b/tests/validation/workspaces/layout/project/src/Clamp.h @@ -0,0 +1 @@ +int clamp(int value, int low, int high); diff --git a/tests/validation/workspaces/layout/project/test/unit/test_calc.c b/tests/validation/workspaces/layout/project/test/unit/test_calc.c new file mode 100644 index 0000000..e5e86c8 --- /dev/null +++ b/tests/validation/workspaces/layout/project/test/unit/test_calc.c @@ -0,0 +1,12 @@ +#include "unity.h" +#include "Calculator.h" + +void test_add(void) +{ + TEST_ASSERT_EQUAL(3, add(1, 2)); +} + +void test_add_more(void) +{ + TEST_ASSERT_EQUAL(5, add(2, 3)); +}