Skip to content

Make check ids required, and rename SuiteSummary.verified to hasFailures - #69

Merged
ottonomy merged 2 commits into
nightshift/45-return-every-check-that-ran-and-remove-the-verbofrom
nightshift/46-make-check-ids-required-and-rename-suitesummary
Oct 20, 2026
Merged

ottonomy merged 2 commits into
nightshift/45-return-every-check-that-ran-and-remove-the-verbofrom
nightshift/46-make-check-ids-required-and-rename-suitesummary

Conversation

@ottonomy

Copy link
Copy Markdown
Contributor

Closes #46

Stacked on #68 — review and merge that first (or merge the stack together).

Departures

None. Everything in "What to build" landed: ids stamped in the suite runner and
required on the type, the after-the-fact population removed, the summary's id
fallback removed, suite un-deprecated (check still deprecated),
SuiteSummary.verified renamed to hasFailures, the stability statement about
check ids and built-in phase tags written down, and the documentation sweep
(type comments, API guide, timing guide, README, CHANGELOG).

Two small extras inside the ticket's intent, both noted below: the API guide's
example gained the status.list-issuer row it was missing, and one line of
docs/architecture.md that spells out the CheckResult shape now lists id.

Decisions not in the issue

  • computeId moved to src/compute-id.ts. The suite runner now needs it,
    and fold-results.ts already imports NOT_RUN_REASON_PREFIX from
    run-suites.ts, so importing the other way would have closed a cycle. The new
    module is where the "ids are API, fixed for 2.x" note lives.
    fold-results.ts re-exports computeId, so the barrel export and every
    existing import path are unchanged.
  • The API guide's id list is in the order it was already in (core, proof,
    status, registry, recognition, parsing, then the three Open Badges bundles),
    not suite-execution order — smaller diff, same content.
  • hasFailures is counts.failed > 0, i.e. any failure, fatal or not. The
    issue's wording ("true when any check in the suite failed") and the old
    field's inverse both read that way.
  • The §1 example in the API guide gained a cryptographic.status.list-issuer
    row
    , because adding the missing id to the reference list while the worked
    example still showed the status suite with one check would have left the two
    disagreeing. Its summary row now reads "2 of 2 checks skipped".
  • docs/architecture.md line 313 (CheckResult: { suite, check, outcome, … }) now includes id. The file is not in the ticket's documentation list,
    but that line states the shape of the type this change makes required; leaving
    it would have been a known-wrong line.
  • Test fixtures for foldCheckResults need ids now. Rather than thread the
    phase through every call site, test/fold-results.spec.ts has a small
    FIXTURE_PHASES map from fixture suite id to phase, so mkResult stamps the
    same id the runner would — including custom, which is deliberately absent
    from the map so it produces unknown.custom.x.
  • Four tests added to test/run-suites.spec.ts covering the acceptance
    item that the runner stamps every row: normal and not-run rows, an untagged
    suite (unknown. prefix), the synthetic .applies skip, and the .applies
    failure from a throwing predicate. The parse-failure id was already covered by
    an existing test in test/verify-credential.spec.ts.

Agent-Run: d76a817e-79b3-4e1c-bc5f-101e5d4c866e

`runSuites` now computes `<phase>.<suite>.<check>` for every row it
writes — normal, not-run, and both kinds of synthetic `.applies` row —
so `CheckResult.id` is required and the verifier's after-the-fact
`populateCheckIds` pass is gone. The parse-failure row gets its
`cryptographic.parsing.envelope` id where it is built, and the summary
code's fallback for a missing id goes with it. `computeId` moves to its
own module so the suite runner can use it without importing the
summarizer.

`SuiteSummary.verified` becomes `hasFailures`, true when any check in
the suite failed. "Verified" now exists only at the top level, where it
means "no fatal check failed" — a non-fatal failure no longer reads as
`verified: false` inside a `verified: true` result.

`CheckResult.suite` is no longer deprecated: it is how rows are grouped,
it matches `SuiteSummary.suite`, and it cannot be recovered from `id`,
because suite ids contain dots. `check` stays deprecated.

Docs: record that check ids and the built-in suites' phase tags are
fixed for 2.x, and complete the API guide's list of ids with the status
list-issuer check, the parsing suite and the Open Badges suites'
synthetic `.applies` rows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`hasFailures` was always `status === 'failure' || status === 'mixed'`
and nothing read it, so it goes before the 2.0 shape freeze. Adding a
field later is not a breaking change; removing one is.

`fatalFailureAt` is now documented as the per-suite fatal signal. A
fatal failure halts the subject, so at most one summary carries it, and
a result is unverified exactly when one of its summaries does. `status`
stays a plain count-based fact; the docs now tell UIs to colour the
suite with `fatalFailureAt` as an error and other failing suites as
warnings, so a non-fatal failure no longer reads as red inside a
verified credential.

Docs fixes in the same pass:
- `verified` on both result types said "no check failed"; it means no
  fatal check failed.
- The API guide's rendering example and prompt appendix showed a
  credential as unverified because of a registry failure, which is
  non-fatal. The examples now show a verified credential with warnings
  and an unverified one with a fatal signature failure.
- `'mixed'` was documented as covering failure plus skipped; the code
  returns `'failure'` for that case.

Tests cover a non-fatal custom failure (suite fails, no
`fatalFailureAt`, still verified) and the one-fatal-summary invariant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ottonomy
ottonomy merged commit 546f3f7 into main Oct 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-authored Opened by an unattended agent — review before trusting

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make check ids required, and rename SuiteSummary.verified to hasFailures

1 participant