Repository navigation
Oracle seeds report CAUGHT and pass when SQL Server fixture initialization fails #151
Description
Problem
The release requirement oracles seeds can pass when the oracle never reaches product code. tools/Oracles/Program.cs treats any failed TRX result as a catch when SeedVerdict.Catches(message) is true; that method accepts every message without a known-candidate marker. Fixture initialization and infrastructure failures satisfy this rule. The runner never first establishes a passing unseeded baseline or correlates failures with a seeded behavior.
Reproduced on the v2.3.0 release tip
Commit: 99c6032d9364a5f94b71ed8f7dce1cdada200870.
On this arm64 macOS host Docker is available, but SQL Server 2022 exits before readiness with Unable to create a new asynchronous I/O context (the existing fixture comments document the emulation limitation).
dotnet tools/Oracles/bin/Release/net10.0/oracles.dll seedsexits 0, endingevery seed is caught.- Its column-cast seed reports 10,071 of 10,071 tests fail as
CAUGHT -> SqlServerQueryVariantTests. - Its ushort seed reports 170 of 170 tests fail as
CAUGHT -> SqlServerKeyVariantTests(even the int cases fail). - An unmodified baseline EF run, filtered to
Needs=Docker, fails 10,248 of 10,257, with 9 passing PostgreSQL tests. All SQL Server failures come fromSqlServer.InitializeAsync, withContainerNotRunningException; no query or insert under review executes.
The independent non-Docker solution run passed 19,157, with 471 skips. This is a seed-verification defect, not evidence of a product defect or SQL Server behavior.
Expected
An unavailable fixture or a failing unseeded oracle must yield ERROR/nonzero, never CAUGHT. Establish a passing baseline for each participating oracle and distinguish setup/test-host failures from product assertions before crediting mutations. Add a regression check that a fixture initialization failure cannot satisfy a seed. Re-run both affected SQL Server oracle/seed pairs on a working server before signing off v2.3.0.
Found by the pre-release code review requested by the maintainer.
Activity
CaffeinatedCoder commentedon Oct 29, 2026
Re-run on a working server, as asked under Expected. Same commit (
99c6032), arm64 macOS, withTYPEKIT_MSSQL_IMAGE=mcr.microsoft.com/azure-sql-edge:latest(CONTRIBUTING's Docker bullet for an arm64 Mac).build-verdict.dll testover the solution givesTESTS OK: 28968 passed, 924 skipped. The SQL Server collection fixture starts.oracles.dll seedsexits 0. The two SQL Server pairs now fail on only part of their rows, not on every row as with a dead fixture:column-cast-not-re-typed.patch -> SqlServerQueryVariantTests: 2978 of 10071 fail, e.g.(kind: "DateOnly", conversion: TypeKit, site: Column, accessor: Value, op: Least).ushort-key-not-generated.patch -> SqlServerKeyVariantTests: 10 of 170 fail, e.g.(shape: TptRoot, kind: "ushort").So both seeds are really caught on a server that runs. The defect in the tool remains: with SQL Server 2022 under QEMU, the same command credited both pairs while 10,255 tests failed in
SqlServer.InitializeAsync. Measured in a plain solution run earlier in the same session.docs: the new-API rule for Errors is decision 118
CaffeinatedCoder commentedon Oct 29, 2026
Fixed in #158 (into
release/2.3.0), recorded as decision 117.Baseline first.
oracles seedsnow runs every oracle the selected seeds name on a copy with no seed, built and filtered the same way. A seed isCAUGHTonly when its oracle passes without it and fails with it, in a test of its own.Setup failures are
ERROR, exit 4, neverCAUGHT. Each of these is read from the TRX and the exit code, as measured with xunit.v3 4.0.2:Collection fixture type 'T' threw in InitializeAsyncand four more;errorcounter with no failed test: a fixture'sDispose;A failing baseline is
ERRORtoo.Re-run at the branch on this arm64 Mac:
ERROR(baseline 170 of 170 and 10071 of 10071 tests failed in setup), exit 4.TYPEKIT_MSSQL_IMAGE=mcr.microsoft.com/azure-sql-edge:latest: all 8 pairsCAUGHT, exit 0. Column cast: 2978 of 9621 tests fail. Ushort: 10 of 167 fail.Since this merges into the release branch,
Closeswon't close the issue. Close it by hand when #158 merges.CaffeinatedCoder commentedon Oct 29, 2026
Fixed in #158, merged into
release/2.3.0(1d2c547): every oracle runs a seed-free baseline first; a setup or host failure is ERROR (exit 4), never CAUGHT (decision 117).