Skip to content

Replace the prober with ValueRanges' oracle kit: claims, surface coverage, oracles and per-oracle seeds #97

Closed

Description

@CaffeinatedCoder

The prober costs more than it pays here. prober seeds builds once per seed: 170 seeds at about four minutes each, so a release's seeds --all takes roughly eleven hours, and a change to one fault-response file stales 23 seeds (about 90 minutes) of five probes. The replay, the footprints, the ledger, the covering arrays and the release gate carry the same weight. CodoMetis.ValueRanges kept the parts of this approach that pay for themselves (its docs/oracles.md): no covering arrays, no ledger, no replay footprints, no agent.

Proposal

Replace the prober with ValueRanges' oracle kit, ported to xunit v3:

  • test/oracles/claims.json: one entry per consumer-facing promise, with its source heading, a verbatim quote that a conventions test finds in the document, its surfaces, and heldBy, the test classes that hold it. An empty heldBy is the worklist.
  • API-surface coverage: every public member of the shipping assemblies is named by a claim's surfaces, or waived with a reason in surface-waivers.json.
  • oracles.json: the registry of test classes that are oracles (*OracleTests, *ProbeTests), ordinary tests in the projects that have their harness.
  • seeds/*.patch: one or more per oracle, each naming the oracles it fails. CI checks without building that every oracle has a seed and every seed still applies. dotnet run tools/oracles.cs seeds proves locally that each is caught by building one project and running one class, before a release.
  • candidates.json: known disagreements tied to an issue, so a sweep stays green without hiding them.
  • axes.md: hand-written, the settings a consumer varies and the pairs no test varies yet.

Retired:

  • tools/Prober and its tests, the corpus under tools/probe/, and the 170 seeds;
  • the probes and probe-status jobs of dotnet.yml, and the replay and gate steps of release.yml;
  • the probe conventions tests, .claude/probe-axes.md, the probe-triage skill and the post-commit replay hook.

One decision record supersedes decisions 96, 100, 101, 103, 104, 107 and 108, and the gate part of 97. DocsQuery keeps its facts. The probes that found a bug are ported as oracle classes; the rest go.

Done when

  • No file in the repository runs, builds or documents the prober, and CI and the release run without it.
  • The oracle kit's conventions tests pass in CI, and every registered oracle has a seed that the local tool shows caught.
  • AGENTS.md, CONTRIBUTING.md and design/plan.md describe the new approach.

Activity

  1. added this to the 2.3.0 milestone on Oct 28, 2026
  2. added
    choreBuild, CI, tooling, packaging, refactoring with no behaviour change
    area/releaseVersioning, changelogs, packaging and release wiring
    on Oct 28, 2026
  3. added
    area/toolingThe repository's tools: prober, verdict, guard, docs-query, smoke test
    on Oct 28, 2026
  4. CaffeinatedCoder commented on Oct 28, 2026

    @CaffeinatedCoder
    OwnerAuthor

    Done in PR #108, merged into release/2.3.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/releaseVersioning, changelogs, packaging and release wiringarea/toolingThe repository's tools: prober, verdict, guard, docs-query, smoke testchoreBuild, CI, tooling, packaging, refactoring with no behaviour change

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions