fix: oracles seeds credits a seed only against a passing baseline

AGENTS.md

@@ -266,7 +266,10 @@ probe.
 
 ## Releasing
 
-Before `git tag`, show every seed caught, locally, a build of one project per seed:
+Before `git tag`, show every seed caught, locally: a baseline with no seed, then a build of one project per seed.
+A seed counts only where its oracle passes without it and fails with it in a test of its own; a fixture that never
+started, a crashed host or a failing baseline is `ERROR`, exit 4 (decision 117). On an arm64 Mac set
+`TYPEKIT_MSSQL_IMAGE=mcr.microsoft.com/azure-sql-edge:latest`, or the SQL Server oracles are `ERROR`:
 
 ```bash
 dotnet tools/Oracles/bin/Release/net10.0/oracles.dll seeds
@@ -283,4 +286,5 @@ It reads api.nuget.org, needs no model, and is never a CI check: time passing fa
 
 The release workflow runs the suite, which holds the catalogue, before it packs; there is no gate beside it
 (decision 112 withdrew decision 97's). A seed that is not caught is answered by a stronger oracle or a better
-seed, never by deleting the seed.
+seed, never by deleting the seed, and an `ERROR` by fixing the environment or the baseline, never by reading it as
+caught.

CONTRIBUTING.md

@@ -173,8 +173,10 @@ The packages share one version and are released together, only through
 `.github/workflows/release.yml` (see [SECURITY.md](SECURITY.md)).
 
 1. Check what the documents rest on, locally:
-   - Show every oracle seed caught: `dotnet tools/Oracles/bin/Release/net10.0/oracles.dll seeds` (a build of one
-     project per seed; [test/oracles/README.md](test/oracles/README.md)).
+   - Show every oracle seed caught: `dotnet tools/Oracles/bin/Release/net10.0/oracles.dll seeds` (a baseline with
+     no seed, then a build of one project per seed; [test/oracles/README.md](test/oracles/README.md)). It exits 0
+     only when every oracle passes without its seed and fails with it. An `ERROR` (exit 4) is an environment or a
+     baseline to fix, such as a SQL Server container that never started, never a seed to accept.
    - Re-read the competitor facts that are due: `dotnet tools/DocsQuery/bin/Release/net10.0/docs-query.dll due --nuget`
      lists each `design/knowledge/` entry pinned `external` whose `verified:` date is more than six months old, or
      whose pinned package NuGet has a later release of. Read each again at the latest release, from its source and

design/adr/0117-a-seed-is-caught-against-a-passing-baseline.md

@@ -0,0 +1,54 @@
+# 0117. A seed is caught against a passing baseline, by a test of its own
+
+Date: 2026-10-29
+Status: accepted
+
+Context: [#151](https://github.com/CaffeinatedCoder/CodoMetis.TypeKit/issues/151), found by the 2.3.0 pre-release
+review. [Decision 112](0112-the-prober-gives-way-to-valueranges-oracle-kit.md) made `oracles seeds` the release's
+proof that every oracle can fail; [decision 108](0108-a-probe-check-earns-credit-against-a-baseline-and-ws2-closes.md)
+had held the prober it replaced to a baseline, and the port did not carry that over.
+
+`oracles seeds` credited a seed whenever one test of its oracle failed with a row that was not a known candidate,
+whatever the failure was. On an arm64 Mac, SQL Server 2022 under QEMU stops before it is ready, so the SQL Server
+collection fixture throws `ContainerNotRunningException` in `InitializeAsync` and every test of both SQL Server
+oracles fails, with and without a seed. The tool reported both seeds `CAUGHT` (10,071 of 10,071 and 170 of 170 tests
+failing) and exited 0, ending `every seed is caught`, while no query or insert under review had run.
+
+**A seed is caught only when its oracle passes without it and fails with it, in a test of its own.** `oracles seeds`
+first runs every oracle the selected seeds name on a copy of the tree with no seed, built and filtered as the seeded
+runs are, in the same process and environment. An oracle that does not pass there (a run that never reached its
+tests, or a test that fails on a row that is not a known candidate) makes each of its pairs `ERROR`, and is not run
+with the seed. A baseline whose only failures are known candidates passes, since an unfiled candidate fails its sweep
+until it is filed.
+
+**A run that did not reach the tests' own code is `ERROR`, never `CAUGHT`, with or without the seed.** What the TRX
+holds decides it, measured with xunit.v3 4.0.2 on Microsoft Testing Platform on 2026-10-29, one class per kind of
+failure:
+
+- A class, collection or assembly fixture that throws in its constructor or `InitializeAsync` fails every test that
+  uses it, each with xunit's own message (`Collection fixture type 'T' threw in InitializeAsync`, the fixture's
+  exception on the next line, the stack only an inner one). xunit.v3.core 4.0.2 holds four more setup messages of
+  the same kind: unresolved constructor arguments, more than one public constructor, constructor parameters without
+  fixture data, and a type it cannot create. A failed test whose message begins with one of them is setup.
+- A fixture that throws in `Dispose` or `DisposeAsync` fails no test: the TRX counts it as an `error` in its
+  `Counters` and holds no result for it. A nonzero `error`, `aborted`, `timeout` or `disconnected` counter is an error.
+- A test host that crashes (`Environment.FailFast`) writes no TRX and exits 134. Microsoft Testing Platform exits 0
+  when every test passes and 2 when one fails; any other exit code, or no TRX, is an error.
+- A run whose every test is skipped ran nothing, and is an error.
+- A failure in the test class's own constructor or `InitializeAsync` is written like one in the test's body: the
+  exception's message, and a stack trace under the test's code. It counts as the test's: the oracles build there
+  what they assert on (`KeyGenerationVariantTests` opens its connection in its constructor), and the baseline has
+  shown that it passes without the seed.
+
+The rule is one file, `test/oracles/SeedVerdict.cs`, compiled into the tool and into `SeedVerdictTests`, which hold
+it on TRX documents of the measured shape. With the setup check and the baseline taken out, the dead fixture of #151
+reads as caught again and three of those tests fail.
+
+**`ERROR` exits 4**, beside 1 for a seed missed, so a release script tells a seed that needs a stronger oracle from an
+environment that needs fixing. A release on an arm64 Mac runs `oracles seeds` with `TYPEKIT_MSSQL_IMAGE` naming
+Azure SQL Edge, as the suite does (CONTRIBUTING's Docker bullet).
+
+Rejected: telling a setup failure apart by its stack trace (a frame of `InitializeAsync`, a constructor), since a
+test's own exception passes through product code whose frames carry the same names, and xunit writes the fixture
+cases in fixed words of its own; and requiring the seeded run to fail on the rows the baseline passed, row by row,
+since a test's rows are not in the TRX and the baseline already rules out a failure that does not depend on the seed.

design/adr/README.md

@@ -123,3 +123,4 @@ per area ([design](../plan.md)) link to the records they rest on.
 | [0114](0114-a-value-object-s-temporary-number-is-its-wrapped-type-s.md) | A value object over a number gets the temporary values a property of its wrapped type gets, from a factory in its converter's mapping hints, which EF consults after a generator on the property and after HiLo: a default `ulong` key no longer throws at `Add`, and a `char` key configured on add gets a temporary `char` on SQL Server; under `--nativeaot` the factory is lost and a default `ulong` key still throws | 2026-10-29 | accepted |
 | [0115](0115-value-objects-can-be-inlined-everywhere.md) | `AddTypeKit(typeKit => typeKit.InlineValueObjects = true)` writes every value object in place, by its wrapped type's schema, in every position and version, with no component of its own; off by default, the document unchanged; closed sets and wrapped enums keep their components; an inlined value object claims no name (decision 55) and shares no schema between its JSON positions and its parameters (decision 106) | 2026-10-29 | accepted |
 | [0116](0116-the-docs-site-is-docfx-built-from-main.md) | The docs site is DocFX, published to GitHub Pages from `main` alone: the guides, the READMEs, the changelog and the API pages of the run-time packages, from the files as they are; a link to a file that is not a page becomes a GitHub address at the built commit; a pull request touching a page builds it with warnings as errors; `site/**` is prose; DocFX 2.81.0 lists the members of C# 14 extension blocks as extension methods | 2026-10-29 | accepted |
+| [0117](0117-a-seed-is-caught-against-a-passing-baseline.md) | `oracles seeds` credits a seed only when its oracle passes without it, in a baseline run with the same build and filter, and fails with it in a test of its own; a run that never reached the tests (a fixture xunit could not set up, an error counted outside any test, a crashed host, no TRX, every test skipped) or a failing baseline is `ERROR`, exit 4, never `CAUGHT` | 2026-10-29 | accepted |

test/CodoMetis.TypeKit.Oracles.Tests/SeedVerdictTests.cs

@@ -1,10 +1,18 @@
+using System.Xml.Linq;
+
 namespace CodoMetis.TypeKit.Oracles.Tests;
 
 /// <summary>
-/// The rule <c>oracles seeds</c> judges a failed test by (<see cref="SeedVerdict"/>), fed the messages
-/// <see cref="Candidates.Verdict"/> writes: a seed is caught when one row of a failed test is not a known candidate,
-/// whatever else shares its message.
+/// The rule <c>oracles seeds</c> judges a seed by (<see cref="SeedVerdict"/>): a seed is caught when its oracle passes
+/// without it and one of its tests fails with it, with a row that is not a known candidate, whatever else shares its
+/// message, and never by a failure xunit wrote for a fixture it could not set up or by a run that did not reach the
+/// tests (design/adr/0117).
 /// </summary>
+/// <remarks>
+/// The TRX documents here have the shape xunit.v3 4.0.2 writes through <c>--report-xunit-trx</c>, measured on
+/// 2026-10-29 with one class per kind of failure: the messages are xunit's own, a fixture's <c>Dispose</c> that throws
+/// is a counter and no result, a crashed host writes no file.
+/// </remarks>
 public sealed class SeedVerdictTests
 {
     private static readonly Candidates.Candidate Unfiled = new("AnyOracle", "row 1:", "a known disagreement", Issue: "");
@@ -60,6 +68,127 @@ public void A_known_row_spanning_lines_does_not_catch()
     public void A_sweep_with_no_failure_and_no_candidate_passes() =>
         Should.NotThrow(() => Candidates.Judge([], [], "a sweep"));
 
+    /// <summary>What SQL Server 2022 under QEMU wrote for every test of a SQL Server oracle (#151).</summary>
+    private const string DeadSqlServer =
+        "Collection fixture type 'CodoMetis.TypeKit.EntityFrameworkCore.Tests.SqlServer' threw in InitializeAsync\n" +
+        "---- DotNet.Testcontainers.Containers.ContainerNotRunningException : Container is not running.";
+
+    // Before Passing: static fields initialize in the order they are written.
+    private static readonly XNamespace Ns = "http://microsoft.com/schemas/VisualStudio/TeamTest/2010";
+
+    private static readonly SeedVerdict.Run Passing = Read(Result("Passed"), Result("Passed"));
+
+    /// <summary>The defect of #151: a dead fixture failed every test with and without the seed, and the seed read as caught.</summary>
+    [Fact]
+    public void A_fixture_that_fails_to_initialize_is_an_error_with_or_without_the_seed()
+    {
+        var dead = Read(Result("Failed", DeadSqlServer), Result("Failed", DeadSqlServer));
+
+        dead.Error.ShouldNotBeNull().ShouldContain("2 of 2 tests failed in setup");
+        SeedVerdict.Judge(dead, dead).ShouldStartWith("ERROR   without the seed");
+        SeedVerdict.Judge(Passing, dead).ShouldStartWith("ERROR   with the seed");
+    }
+
+    /// <summary>A seed that breaks a fixture is not credited either: only a test of the oracle's own catches.</summary>
+    [Fact]
+    public void A_fixture_failure_beside_a_caught_test_is_an_error()
+    {
+        var seeded = Read(Result("Failed", "Shouldly.ShouldAssertException : value should be 1 but was 2"), Result("Failed", DeadSqlServer));
+
+        SeedVerdict.Judge(Passing, seeded).ShouldStartWith("ERROR   with the seed");
+    }
+
+    /// <summary>Every message xunit.v3.core 4.0.2 holds for a test it could not set up.</summary>
+    [Theory]
+    [InlineData(DeadSqlServer)]
+    [InlineData("Class fixture type 'Probe.ThrowingCtorFixture' threw in its constructor\n---- System.InvalidOperationException : fixture ctor threw")]
+    [InlineData("Class fixture type 'Probe.ThrowingInitFixture' threw in InitializeAsync\n---- System.InvalidOperationException : threw")]
+    [InlineData("Assembly fixture type 'Probe.Fixture' threw in InitializeAsync\n---- System.Exception : threw")]
+    [InlineData("Collection fixture type 'Probe.Fixture' had one or more unresolved constructor arguments: Probe.Other other")]
+    [InlineData("Class fixture type 'Probe.Fixture' may only define a single public constructor.")]
+    [InlineData("The following constructor parameters did not have matching fixture data: SqlServer server")]
+    [InlineData("Cannot create type 'Probe.Fixture' due to missing constructor arguments: Probe.Other")]
+    public void A_message_xunit_writes_for_a_fixture_it_could_not_set_up_is_setup(string message) =>
+        SeedVerdict.IsSetupFailure(message).ShouldBeTrue();
+
+    /// <summary>An assertion, an exception from the code a test calls, and a sweep's verdict are the test's own failures.</summary>
+    [Theory]
+    [InlineData("Shouldly.ShouldAssertException : [Fact] public void Shouldly_fails1\n    should be\n2\n    but was\n1")]
+    [InlineData("Assert.Equal() Failure: Values differ\nExpected: 1\nActual:   2")]
+    [InlineData("System.InvalidOperationException : product threw")]
+    [InlineData("a sweep: 3 of 170 rows disagree\n(shape: TptRoot, kind: \"ushort\"): the key reads back as 0")]
+    [InlineData("row 1: the Collection fixture type 'X' threw in InitializeAsync, quoted by a test")]
+    public void A_failure_in_the_test_itself_is_not_setup(string message) => SeedVerdict.IsSetupFailure(message).ShouldBeFalse();
+
+    [Fact]
+    public void An_oracle_that_fails_without_the_seed_is_an_error()
+    {
+        var failing = Read(Result("Passed"), Result("Failed", "Assert.Equal() Failure: Values differ"));
+
+        SeedVerdict.Judge(failing, failing).ShouldBe("ERROR   1 of 2 tests fail without the seed, e.g. Probe.Oracle.T1");
+    }
+
+    [Fact]
+    public void A_test_failing_only_with_the_seed_catches() =>
+        SeedVerdict.Judge(Passing, Read(Result("Passed"), Result("Failed", "Assert.Equal() Failure: Values differ")))
+                   .ShouldBe("CAUGHT  1 of 2 tests fail, e.g. Probe.Oracle.T1");
+
+    [Fact]
+    public void An_oracle_that_passes_with_the_seed_misses_it() =>
+        SeedVerdict.Judge(Passing, Passing).ShouldBe("MISSED  all 2 tests pass with the defect planted");
+
+    /// <summary>An unfiled candidate fails its sweep without a seed; the baseline passes, and the seed is judged by its other rows.</summary>
+    [Fact]
+    public void A_baseline_failing_only_on_known_candidates_passes()
+    {
+        var known = Read(Result("Passed"), Result("Failed", MessageOf([Unfiled], ["row 1: expected 1, got 2"])));
+        var seeded = Read(Result("Passed"), Result("Failed", MessageOf([Unfiled], ["row 1: expected 1, got 2", "row 2: expected 1, got 3"])));
+
+        known.Error.ShouldBeNull();
+        SeedVerdict.Judge(known, known).ShouldBe("MISSED  1 of 2 tests fail, every row a known candidate");
+        SeedVerdict.Judge(known, seeded).ShouldStartWith("CAUGHT");
+    }
+
+    /// <summary>A fixture whose Dispose throws fails no test; the TRX counts it as an error outside any result.</summary>
+    [Fact]
+    public void An_error_counted_outside_any_test_is_an_error() =>
+        SeedVerdict.Read(2, Trx([Result("Passed")], error: 1), "Probe.Oracle").Error.ShouldNotBeNull().ShouldContain("1 error outside any test");
+
+    /// <summary>A host that crashes writes no TRX; one that exits otherwise than 0 or 2 did not finish its tests.</summary>
+    [Theory]
+    [InlineData(134, false)]
+    [InlineData(7, true)]
+    [InlineData(3, true)]
+    public void A_host_that_did_not_finish_is_an_error(int exitCode, bool wroteResults) =>
+        SeedVerdict.Read(exitCode, wroteResults ? Trx([Result("Failed", "Assert.Equal() Failure: Values differ")]) : null, "Probe.Oracle")
+                   .Error.ShouldNotBeNull().ShouldContain($"exited {exitCode}");
+
+    [Fact]
+    public void A_run_whose_every_test_is_skipped_is_an_error() =>
+        SeedVerdict.Read(0, Trx([Result("NotExecuted"), Result("NotExecuted")]), "Probe.Oracle").Error.ShouldBe("no test of Probe.Oracle ran");
+
+    private static SeedVerdict.Run Read(params XElement[] results) =>
+        SeedVerdict.Read(results.Any(result => (string?)result.Attribute("outcome") == "Failed") ? 2 : 0, Trx(results), "Probe.Oracle");
+
+    private static XElement Result(string outcome, string? message = null) =>
+        new(Ns + "UnitTestResult", new XAttribute("testName", "?"), new XAttribute("outcome", outcome),
+            message is null ? null : new XElement(Ns + "Output", new XElement(Ns + "ErrorInfo", new XElement(Ns + "Message", message), new XElement(Ns + "StackTrace", "   at Probe.Oracle.T()"))));
+
+    private static XDocument Trx(IReadOnlyList<XElement> results, int error = 0)
+    {
+        var named = results.Select(result => new XElement(result)).ToList();
+        for (var i = 0; i < named.Count; i++) named[i].SetAttributeValue("testName", $"Probe.Oracle.T{i}");
+
+        int Count(string outcome) => named.Count(result => (string?)result.Attribute("outcome") == outcome);
+        return new XDocument(new XElement(Ns + "TestRun",
+            new XElement(Ns + "Results", named),
+            new XElement(Ns + "ResultSummary", new XAttribute("outcome", Count("Failed") > 0 || error > 0 ? "Failed" : "Completed"),
+                new XElement(Ns + "Counters",
+                    new XAttribute("total", named.Count), new XAttribute("executed", Count("Passed") + Count("Failed")),
+                    new XAttribute("passed", Count("Passed")), new XAttribute("failed", Count("Failed")), new XAttribute("error", error),
+                    new XAttribute("timeout", 0), new XAttribute("aborted", 0), new XAttribute("notExecuted", Count("NotExecuted")), new XAttribute("disconnected", 0)))));
+    }
+
     private static string MessageOf(IReadOnlyList<Candidates.Candidate> known, IReadOnlyList<string> failures, int shown = 25) =>
         Should.Throw<Exception>(() => Candidates.Judge(known, failures, "a sweep", shown)).Message;
 }