Skip to content

✨ Native code coverage from Ceedling's gcov plugin, with gcovr and ReportGenerator (supersedes #13) - #26

Merged
mkarlesky merged 10 commits into
next_versionfrom
feature/native-coverage
Oct 23, 2026
Merged

mkarlesky merged 10 commits into
next_versionfrom
feature/native-coverage

Conversation

@mkarlesky

Copy link
Copy Markdown
Member

Supersedes #13. Adds native code coverage to the Testing view. Run with Coverage runs tests through Ceedling's gcov plugin, then shows each source file's line, branch, and function coverage in VS Code's coverage views. It reads the Cobertura report from gcovr or ReportGenerator.

Credit

The design and the first implementation are by @jake-kim-sndk (Jake Jooncheol Kim), in #13. His pull request closed so it could be retargeted, and his fork has since been deleted. His commit survived at refs/pull/13/head, and it is preserved two ways:

  • In this branch's history, unchanged as aa07df0, with his author, date, and message. A merge commit brings it in, so GitHub lists it under his name.
  • On archive/pr-13-gcov-coverage, pushed to this repo as a durable copy.

The commit that ports his engine and controller wiring onto the reworked code carries a Co-authored-by trailer for him. CHANGELOG credits him in its existing "contributed by" style. The new docs/Acknowledgments.md credits him after the README's account of the extension's origin.

Commits

Commit What
aa07df0 Jake's original commit, exactly as submitted
🔀 Merge Brings in aa07df0. next_version reworked the engine and controller it changed, so conflicts resolve to next_version's side. His new module, tests, and fixtures arrive exactly as written.
🧪 Fixtures capture-gcov.sh plus captures from Ceedling 1.2.0 with its own gcovr 7.2 and ReportGenerator 5.5.11. Each tool's Cobertura report, and dumpconfig with the gcov plugin's defaults.
✨ Port (co-authored by Jake) Coverage through the reworked engine. Coverage state comes from the resolved configuration, so no report path is hard-coded. Coverage builds run under the project directory's lock. The profile appears only while some project enables the gcov plugin.
✨ ReportGenerator The module is renamed coberturaReport.ts, so its history follows. Several <source> roots resolve to the candidate path that exists. Real-capture tests.
✅ Tests Engine edge cases. Coverage in VS Code against the fake Ceedling: the profile's lifecycle, file coverage, and its details.
📝 Docs The README "Code coverage" how-to, the CHANGELOG, BreakingChanges, Acknowledgments, Development.md, and a manual checklist with tests/manual/mixins/.
🐛 Review fixes The final review's findings (below).

Behavior

  • Which report: gcovr's, when both tools write one. ReportGenerator's is ReportGenerator/Cobertura.xml, or ReportGenerator/Cobertura/Cobertura.xml with several report types.
  • One run, one measurement: each request clears the coverage counters earlier runs left (.gcda, plus ReportGenerator's .gcov inputs), once per build directory. The new ceedlingExplorer.coverageResetsCounters setting turns that off to accumulate coverage across runs.
  • :report_task: TRUE: the run adds report:gcov to its own command line.
  • No coverage: a project without the plugin, without a Cobertura report, with a report covering no sources, or whose report tool is missing gets a notification saying why, once a session. Its tests still run.
  • gcov: task names: Ceedling 1.2.0's gcov plugin still takes only a bare file name, unlike its test: tasks. That is reported as gcov: tasks don't accept project-relative paths, unlike test: tasks in 1.2.0 Ceedling#1362. Coverage runs pass the bare name before 1.2.1, warning when same-named test files make that ambiguous, and the path from 1.2.1 (pathQualifiedCoverageTasks).

Minimum VS Code is now 1.88.0, up from 1.74.0. 1.88 finalized the test coverage API. Typechecking runs against @types/vscode 1.88.0, so no newer API is used.

Verification

  • Lint: 0 findings.
  • Tests: 437 unit and 18 integration pass in the sidecar.
  • Real-Ceedling gate, a one-off script: the compiled engine against Ceedling 1.2.0 with real gcovr and ReportGenerator.
    • Exact checks, one run at a time:
      • gcovr preferred;
      • per-run hits with the counter reset on, accumulated hits with it off;
      • function and project runs;
      • ReportGenerator resolving both a source inside the project and one above it.
    • Then 90 to 120 seconds of overlapping coverage runs, plain runs, discovery, reparses, and Clean, with random cancellation, at two seeds and up to 8 in flight. Result: 0 violations, 0 unhandled rejections, no leftover processes or temp files.

Final review

Four parallel reviews covered the engine and coverage state, the controller, the parser and merge, and the tests and docs. Each finding was checked against the code, and each behavior change was made test-first.

Fixed:

  • A function run's whole-file fallback counted its first attempt's hits twice.
  • Coverage was lost when a run threw partway.
  • A failed build gave no explanation for a configuration that can't produce coverage.
  • An empty report said nothing.
  • Line 0, negative values, and implausible branch counts could crash VS Code's positions.
  • Untrimmed sources.
  • Repeated notifications.
  • Duplicated XML helpers, and path keys taken from the host rather than the engine.
  • Doc accuracy: ceedling clean also clears counters, ReportGenerator lists earlier builds' sources at 0%, and same-named test files are ambiguous. Plus style.

Deferred, with reason: two coverage runs at once in one project share its counters. Keeping them apart would need the engine to track when each request ends. That is rare, and the README documents it.

Manual checks

docs/Development.md §5 gains coverage checks for gcovr, ReportGenerator, per-run coverage, and a project without a Cobertura report.

🤖 Generated with Claude Code

jake-kim-sndk and others added 10 commits September 29, 2026 02:38
The Testing view's Run with Coverage button now runs the suite through
Ceedling's gcov: task and reports the result through VS Code's own
coverage UI - the Test Coverage panel, per-file percentages, and inline
line and branch hit counts.

The data was already there. Ceedling's gcov plugin writes a gcovr
Cobertura report with per-line hit counts, per-branch condition coverage
and, from newer gcovr, per-method entries; nothing consumed it. This
wires that report to the coverage API VS Code finalized in 1.88.

- src/gcovrCoberturaReport.ts: pure, vscode-free normalization of the
  parsed report, unit-tested in plain Node like ceedlingXmlReport.ts.
  Every path comes out absolute. gcovr writes <filename> relative to
  <source> inside its root but absolute outside it, and before gcovr
  8.3 <source> is itself relative to where gcovr ran (gcovr#1037).
  Uri.file() puts a relative path at the filesystem root, so a path
  left unresolved shows no coverage at all.
- ceedlingEngine: a coverage variant of the run entry points, and
  reading the report against the project directory Ceedling runs in.
  A missing or unreadable report is "no coverage", not a failed run.
- testController: a non-default Coverage run profile, since a coverage
  run rebuilds instrumented and is slower. Each project's previous
  report is deleted before the run, so a failed build can't pass the
  last run's coverage off as this one's.

The fixtures are real reports from Ceedling 1.1.6 with gcovr 5.0 and
8.6, of a probe project with one source inside the Ceedling project and
one above it.

Requires raising the minimum VS Code from 1.71.0 to 1.88.0, the release
that finalized the test coverage API.
Jake Kim's pull request #13 added native test coverage. It closed before it could merge, and its
fork is gone. Its commit survives at refs/pull/13/head, and this merge brings it in unchanged, with
its author, date, and message.

The pull request targeted master. next_version has since reworked the engine and controller it
changed, so conflicts resolve to next_version's side:
- ceedlingEngine.ts, testController.ts: next_version's code. The next commits port the
  coverage wiring onto it.
- package.json: next_version's, with this commit's VS Code 1.88 engine, @types/vscode
  (~1.88.0, next_version's convention), and coverageCommandArgs setting.
- package-lock.json: regenerated from the merged package.json.
- README.md: next_version's, with this commit's coverage section, feature bullet, setting
  row, and 1.88 requirement.
- CHANGELOG.md, BreakingChanges.md: this commit's entries, folded into the existing
  [Unreleased] sections.

The new coverage module, its tests, and its gcovr fixtures arrive exactly as written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… Ceedling 1.2.0

capture-gcov.sh runs Ceedling's gcov plugin against a probe project in the madsciencelab image,
with that image's gcov, gcovr, and ReportGenerator. The probe has one source inside the Ceedling
project and one above it, as the gcovr 5.0 and 8.6 fixtures' probe did. Its mixins select
the configurations the extension must read: both utilities, several report types, a renamed
gcovr report, and reports deferred to report:gcov.

Committed captures:
- gcovr-7.2/ and reportgenerator-5.5.11/cobertura-probe.xml, the Cobertura reports.
- ceedling-1.2.0/dumpconfig-gcov-*.yml, the resolved configurations, plugin defaults
  included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runs coverage through the engine as it now works, carrying forward Jake Kim's design from #13:
- a non-default Coverage profile;
- the coverageCommandArgs setting;
- a missing report meaning "no coverage", not a failed run;
- a stale report deleted before each coverage build.

What moves where:
- Coverage state: projectState reads, from the resolved configuration, whether the gcov plugin
  is enabled, whether reports wait for report:gcov, and where each enabled utility writes its
  Cobertura report. Nothing is hard-coded to gcovr's default path.
- Engine: runFile, runFunction, and runProjectAll take a RunMode. A coverage build happens under
  the project directory's lock, as every build does:
  - it deletes stale Cobertura reports;
  - it clears earlier counters once per request, unless coverageResetsCounters is off;
  - it runs gcov: with the bare file name, which Ceedling's gcov task requires on every version;
  - it adds report:gcov when reports wait for that task;
  - it reads the report before releasing the lock.
  A project without the gcov plugin runs its tests as usual, and says why there is no coverage.
- Parser: reads the parsed XML as `unknown` rather than `any`, and gives each file its candidate
  paths. The engine picks the candidate that exists.
- Controller:
  - offers the Coverage profile only while some project enables the gcov plugin;
  - merges a request's reports with mergeCoverage;
  - keeps each FileCoverage's details for loadDetailedCoverage;
  - shows why a run has no coverage.

Jake's tests run unchanged against the ported parser, apart from their call sites and two lint
fixes.

Co-authored-by: Jake Jooncheol Kim <227114254+jake-kim-sndk@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ceedling's gcov plugin writes the same Cobertura format through ReportGenerator, which is common
on Windows. The coverage state already lists ReportGenerator's report locations after gcovr's:
- ReportGenerator/Cobertura.xml for one report type;
- ReportGenerator/Cobertura/Cobertura.xml for several.

The module is renamed coberturaReport.ts with git mv, so its history follows, since it no longer
serves gcovr alone.

Tests against the captures from Ceedling 1.2.0:
- ReportGenerator 5.5.11 lists every source directory it was given, and names files relative
  to the directory it ran in. Each file gets a candidate per source, and the engine keeps the one
  that exists. That resolves both files, the one inside the project and the one above it.
- Its <method> entries become functions, and partly taken branches carry through.
- gcovr 7.2 with the default :report_root: "." reports only sources within that root.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the fake Ceedling

Unit tests:
- a function run's whole-file fallback keeps the counters its first attempt produced;
- each build directory has its counters cleared once per run;
- a branch line without condition-coverage carries no branch outcomes;
- a line without a number or hit count is left out.

The fake Ceedling enables the gcov plugin from its scenario. Its gcov: tasks reject a
path-qualified name, as Ceedling's do, and write the CppUnit report under artifacts/gcov and a
gcovr Cobertura report of the covered sources. Integration tests, in VS Code:
- a project with the gcov plugin offers Run with Coverage;
- a coverage run reports its tests, plus line, branch, and function coverage for src/calc.c, and
  the details load through the profile's loadDetailedCoverage;
- the Coverage profile goes away when the plugin does, and comes back with it.

The controller's coverageRunProfile getter lets the tests reach the profile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rator

README:
- A "Code coverage" section replaces the gcovr-only notes. It covers:
  - setting up the gcov plugin with gcovr or ReportGenerator and a Cobertura report, in
    project.yml or a mixin;
  - how coverage runs work: counters cleared per run, gcovr preferred, report:gcov, several
    projects;
  - what each tool reports, including gcovr's :report_root: for sources above the project;
  - what to check when no coverage appears.
- The options table describes both coverage settings. The testCommandArgs row loses its stale
  utils:gcov example.
- The Gcov plugin link points at the documentation that exists.

CHANGELOG and BreakingChanges: the coverage entries in house style, crediting @jake-kim-sndk and
#13, with the minimum VS Code raised from 1.74.0, the version next_version actually required.

docs/Acknowledgments.md: the README's account of the extension's origin, verbatim, then Jake
Kim's coverage contribution. The README keeps its section and links to the new file.

Development: the module table and a Coverage section map the flow to code. The manual
checklist covers gcovr, ReportGenerator, per-run coverage, and a project without a Cobertura
report, using mixins in tests/manual/mixins/.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ified gcov from 1.2.1

Four parallel reviews covered every change on this branch. Each finding was checked against the
code, and each behavior change was made test-first.

Engine:
- The whole-file fallback after a --test-case collision clears that test's own counters first,
  so its hits aren't counted twice.
- A failed coverage build still reports a configuration that can't give coverage.
- A report covering no sources says where gcovr looks.
- Ceedling 1.2.0's gcov plugin still prepends the test file prefix to the whole name, unlike
  its test: tasks. A new pathQualifiedCoverageTasks capability passes paths from Ceedling 1.2.1.
  Before that, the log warns when same-named test files make a coverage run ambiguous.
- Stale files are deleted without a stat apiece. ENOENT is not a failure.
- keyOfPath gives the controller the engine's own path rules for merging.

Parser:
- Line numbers must be integers from 1, and hits integers from 0, since VS Code positions can't
  be negative.
- Implausible branch counts are dropped rather than rejecting the report.
- Sources are trimmed, and an empty <coverage/> covers nothing.
- The XML helpers move to parsedXml.ts, shared with testResults.ts.

Controller:
- Coverage and problems are shown from finally, so a run that throws partway keeps what its
  completed builds measured.
- Each problem notifies once a session, and every run still logs it.
- A named CoverageReport type replaces four inline copies.

Tests:
- nodeFileSystem.listFiles on a real disk;
- the Windows coverage path;
- a fallback to a file's first candidate;
- the 1.2.1 path-qualified tasks;
- the parser's edge cases;
- a coverage run in VS Code against a project without a Cobertura report.
The fake Ceedling accepts paths in gcov: tasks from 1.2.1.

Docs:
- README: the how-to's accuracy (clean also clears counters, ReportGenerator's 0% rows, two
  coverage runs at once share counters, same-named test files), its notification causes, and
  style.
- KnownIssues: same-named test files in coverage runs before Ceedling 1.2.1.
- Development.md: the coverage flow and its wording.
- The nested ReportGenerator path is backed by the capture's file listing.

Deferred, with reason: keeping two coverage runs at once in one project from clearing each
other's counters would need the engine to track when each request ends. That is rare, and the
README documents it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mkarlesky
mkarlesky merged commit 9581b16 into next_version Oct 23, 2026
3 checks passed
@mkarlesky
mkarlesky deleted the feature/native-coverage branch October 23, 2026 22:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants