[CaffeinatedCoder]

Closes #151

Problem

oracles seeds counted any failed test as a catch and never checked that the oracle passes without the seed. On an arm64 Mac, SQL Server 2022 under QEMU never starts, so SqlServer.InitializeAsync throws ContainerNotRunningException and every test of both SQL Server oracles fails. The tool still reported both seeds CAUGHT (10,071 of 10,071 and 170 of 170) and exited 0 with every seed is caught.

Change

• test/oracles/SeedVerdict.cs, linked into the tool and its tests:

  • Read(exitCode, trx, oracle) turns one run into a Run.
  • IsSetupFailure recognises xunit's own setup messages.
  • Judge(baseline, seeded) gives the verdict.
  • The remarks record what the TRX holds for each kind of failure.

• tools/Oracles/Program.cs:

  • It first runs a baseline: every oracle the selected seeds name, on a copy with no seed, built and filtered the same way. It prints one BASELINE line per oracle.
  • An oracle whose baseline fails is not run with the seed, and its pairs are ERROR.
  • A project that does not build is an error for its oracles only, not for the whole seed.
  • ERROR exits 4, so it can be told apart from a missed seed (exit 1).

• How a setup failure is told apart, measured on xunit.v3 4.0.2 with Microsoft Testing Platform:

  • Fixture fails to set up: a class, collection or assembly fixture that fails to construct or initialize fails each test with xunit's fixed message (Collection fixture type 'T' threw in InitializeAsync). xunit.v3.core 4.0.2 holds four more setup messages, and all are matched.
  • Fixture fails in Dispose: only an error counter in the TRX, with no failed result.
  • Host crash: no TRX, and an exit code other than 0 or 2.
  • Every test skipped: nothing ran.

  A failure in a test class's own constructor or InitializeAsync reads like one in the test body, so it counts as the test's own.

• design/adr/0117: the rule, its measurement, and what was rejected. It restores for the oracle kit the baseline that decision 108 had given the prober. Decision 112's port had dropped it.

• test/oracles/README.md has a verdict table. CONTRIBUTING's and AGENTS' release steps now say what ERROR means and that an arm64 Mac needs TYPEKIT_MSSQL_IMAGE.

Tests

• SeedVerdictTests gains tests on TRX documents shaped like the measured ones:
  • the Oracle seeds report CAUGHT and pass when SQL Server fixture initialization fails #151 dead fixture, with and without the seed;
  • a fixture failure next to a caught test;
  • every xunit setup message, and five failures that are the test's own;
  • a failing baseline, a baseline that fails only on known candidates, CAUGHT and MISSED;
  • an error counter, a host that exits 134, 7 or 3, and a run where every test was skipped.
• Seeded the old acceptance back (setup check and baseline turned off): 3 tests fail, the Oracle seeds report CAUGHT and pass when SQL Server fixture initialization fails #151 case among them. Restored, the Oracles test project passes, 51 of 51.
• Conventions: DesignRecordTests pass, 120 of 120.

oracles seeds on this arm64 Mac, at this branch:

SQL Server 2022 (no TYPEKIT_MSSQL_IMAGE): exit 4, 2 min 22 s

BASELINE SqlServerKeyVariantTests: DOES NOT PASS without a seed, 170 of 170 tests failed in setup, not in the test: Collection fixture type 'CodoMetis.TypeKit.EntityFrameworkCore.Tests.SqlServer' threw in InitializeAsync
BASELINE SqlServerQueryVariantTests: DOES NOT PASS without a seed, 10071 of 10071 tests failed in setup, not in the test: Collection fixture type 'CodoMetis.TypeKit.EntityFrameworkCore.Tests.SqlServer' threw in InitializeAsync
ERROR   column-cast-not-re-typed.patch -> SqlServerQueryVariantTests: without the seed, 10071 of 10071 tests failed in setup, ...
ERROR   ushort-key-not-generated.patch -> SqlServerKeyVariantTests: without the seed, 170 of 170 tests failed in setup, ...
(the six other pairs CAUGHT, their baselines passing)
AN ORACLE COULD NOT BE JUDGED: its baseline or its run did not reach the tests' own code; fix that and run again

TYPEKIT_MSSQL_IMAGE=mcr.microsoft.com/azure-sql-edge:latest: exit 0, 5 min 36 s. All 8 baselines pass.

BASELINE SqlServerKeyVariantTests: passes without a seed, 167 tests
BASELINE SqlServerQueryVariantTests: passes without a seed, 9621 tests
CAUGHT  column-cast-not-re-typed.patch -> SqlServerQueryVariantTests: 2978 of 9621 tests fail, e.g. ...(kind: "DateOnly", conversion: TypeKit, site: NamedQueryFilter, accessor: ValueThroughTypeParameter, op: GreaterOrEqual)
CAUGHT  inline-value-objects-ignored.patch -> SchemaTests: 92 of 164 tests fail
CAUGHT  json-key-written-by-to-string.patch -> JsonParityTests: 71 of 157 tests fail
CAUGHT  patch-traverse-takes-a-null-selector.patch -> NullDelegateTests: 1 of 142 tests fail
CAUGHT  result-reads-as-json.patch -> NotWireTypeTests: 15 of 99 tests fail
CAUGHT  traverse-async-resumes-on-the-caller-s-context.patch -> AsynchronousContextTests: 1 of 73 tests fail
CAUGHT  ushort-key-not-generated.patch -> KeyGenerationVariantTests: 19 of 547 tests fail
CAUGHT  ushort-key-not-generated.patch -> SqlServerKeyVariantTests: 10 of 167 tests fail, e.g. ...(shape: TpcRoot, kind: "ushort")
every seed is caught

The executed counts under Edge (9621, 167) are lower than 2022's because the tests that need 2022-only functions skip on Edge.

🤖 Generated with Claude Code