Repository navigation
CI: shorten the Linux test leg: CMTK0003's sweep repeats its compilation per row, and the EF Core query sweeps run their rows one after another #251
Description
Why
After #250 (#245), CI's Linux test leg is the end of every run: a 12m 36s job and a 9m 28s test step in #250's run (41187518150). Windows took 8m 15s. The leg is no longer stuck behind one slow theory. Two projects finish within 40 s of each other, sharing the runner's four cores:
| Project (Linux, run 41187518150) | Wall | Biggest classes (rows, summed row time) |
|---|---|---|
CodoMetis.TypeKit.Analyzers.Tests |
531 s | DroppedTaskVariantTests (1,837 rows, 1,792 s) |
CodoMetis.TypeKit.EntityFrameworkCore.Tests |
494 s | SqlServerQueryVariantTests (12,275 rows, 396 s), QueryTranslationVariantTests (12,275 rows, 357 s) |
Two things keep them there:
- CMTK0003's sweep does the same work in every row. Each row builds a new compilation of the same
DroppedTaskShapes.Declarationsand references. Under row parallelism the rows now wait on each other for CPU: summed row time went from 763 s to 1,792 s. - The EF Core sweeps run their rows one after another, as
DroppedTaskVariantTestsdid before test: run the analyzer sweeps' rows in parallel, and leave the pragma pass to each rule's focused tests #250. Each is one class.SqlServerQueryVariantTestsis also inSqlServerCollection, around one shared container. Whether a row only reads from a seeded database, and so could run beside another, is not yet known.
Outcome of the profile (31 Oct)
CMTK0003: no change. A row's cost is its own file, and reusing the compilation saves nothing (the profile comment has the numbers). EF Core: no change to how rows run either. Parallel rows were 2.2x faster locally, where the database containers had cores of their own, and bought almost nothing on a 4-core CI runner (the CI comment has the numbers), so #252 keeps only the fix for the race they exposed in the SQL Server fixture. The Linux leg is bound by its four cores; what is left is more cores, by splitting the Linux test step across two jobs. The remaining items below are kept as the record of what was asked.
Proposal
Measure first, then change only what the measurement supports:
- Profile one row of each sweep. For CMTK0003, how much of a row is the compilation, the analyzer and the verifier, and how much is shared across rows. For EF Core, how much of a row is building the context, translating the query, running it on the database and running it in memory.
- CMTK0003: if most of a row is work it shares with other rows, compile the declarations once (for example a compilation reused for each call shape) and keep one row per case, so a failure still names its call and form. If the harness rules that out, say so here.
- EF Core: if a row only reads, run the rows of each sweep in parallel (
parallelMode: allfor the project, withDisableParallelizationon anything that writes or shares a mutable fixture). If rows write, report what they write and whether a seeded, read-only database per collection would serve. - Each change goes through verify-the-guard: a seeded defect each sweep catches before still fails a row after.
Done when
The profile is in this issue. Every change it supports is merged, with the Linux test step's before and after from CI in this issue. Each sweep still catches its seed. Rows are neither dropped nor merged into one test, and each still reports on its own.
Activity
CaffeinatedCoder commentedon Oct 31, 2026
Profile, measured locally on 31 Oct (Apple silicon, 16 cores), on
release/2.3.1at c8fb57c.CMTK0003 (
DroppedTaskVariantTests), one row in ten (184 rows), run one at a time:Declarations.csis 14 lines. A row's cost is its own file, which holds the call in its form through every pass-through and in every use, so rows share nothing worth compiling once. Item 2 is answered with no: reuse saves nothing. The only reducible part is the verifier's own work, about 28 of 92 ms. Removing it means replacing the verifier with a matcher of our own, which could pass silently, so I am not proposing it.EF Core, the whole
CodoMetis.TypeKit.EntityFrameworkCore.Testsproject at 4 threads, as on a CI runner:-parallelMode allThe project waits on SQLite, SQL Server and PostgreSQL more than it computes: the same CPU in less than half the time. The three failures are
SqlServerRoundTripTestsracing itself ("There is already an object named 'Customers'"). Its tests share the databasetypekit_round_tripsand each creates the same tables.Who writes what, read from the source:
Lazyand read under a lock per connection, whichA_seeded_database_read_by_many_cases_at_once_answers_each_alikealready holds (EF variant tests: parallel cases share one SqliteConnection and now and then fail with 'database is locked' #148). SQL Server gives each model a schema of its own. PostgreSQL is translation only, with no server.SqlServerKeyVariantTestsgives each row a schema of its own (Interlocked).SqlServerRoundTripTests(7 tests, one database) andPostgreSqlRoundTripTests(9 tests, several writing to the fixture'stypekitdatabase) share state between their own tests.Next, on
test/linux-leg-sweeps-2-3-1:parallelMode: allfor the EF Core tests, with the two round-trip classes opted out ([TestClass(DisableParallelization = true)]). Then several runs in parallel mode, a seed showing that an opted-out class racing is caught, and the CI before and after.CaffeinatedCoder commentedon Oct 31, 2026
Implemented on
test/linux-leg-sweeps-2-3-1(not pushed yet):xunit.runner.jsonwith"parallelMode": "all"for the EF Core tests.SqlServerRoundTripTestsopts out ([TestClass(DisableParallelization = true)]). Seeded by removing the opt-out: 5 of its 7 tests fail ("There is already an object named 'Customers'").PostgreSqlRoundTripTestsneeds no opt-out, and my earlier reason for one was wrong. Every read is scoped to the id of the row its test saved, and the other tests each use a database of their own. Without an opt-out it passed 3 runs of 3.SqlServer.DatabaseAsyncusedConcurrentDictionary.GetOrAdd(name, CreateAsync), which can run the factory more than once for callers that arrive together. The first rows of a sweep then dropped and createdtypekit_queriesunder each other (11,832 failures in that run). It is now aLazy<Task<string>>, the pattern the query sweeps already use. The new guardSqlServerFixtureTests.A_database_asked_for_by_many_callers_at_once_is_created_onceasks for one database from 32 callers at once: it fails 3 of 3 runs with the plainGetOrAddand passes with theLazy. The otherGetOrAddcalls in the tests (the AspNetCore document caches) have no side effects.The EF Core project locally, through
dotnet testat 16 threads: 44-46 s in each of 3 runs, all 24,258 passing. Before: 123 s in the default mode at 4 threads. The whole solution passes: 35,337 passed (one more, the guard), 1,248 skipped as before. The CI before and after comes from the PR.CaffeinatedCoder commentedon Oct 31, 2026
CI on #252 (run 41191607858) against #250's run (41187518150). One run each, so one sample of runner hardware each:
On CI, parallel EF rows bought almost nothing. The local measurement misled: the Mac's Docker had 16 cores, so the database containers ran beside the tests. On a 4-core runner, SQL Server, PostgreSQL and the tests share the same four cores. What looked like waiting became CPU on the same machine. The rows' summed time rose (
QueryTranslationVariantTests357 s to 893 s,SqlServerQueryVariantTests396 s to 553 s) while the project's wall time moved by 51 s. On Windows (no containers) EF Core itself got faster (261 s to 168 s), but the projects beside it got slower by about as much.So the Linux leg is bound by four cores. Only less CPU work or more cores shorten it. The lever left that keeps every row is to split the Linux test step across two jobs: each repeats the ~2.5 minute checkout and build, and gets four cores of its own.
test: run the EF Core rows one collection at a time again, keeping th…
CaffeinatedCoder commentedon Oct 31, 2026
Done in #252, merged into
release/2.3.1as abe2763.Closesdoes not close an issue from a release branch, so this is closed by hand. Neither sweep changes how its rows run (the outcome above). #252 keeps the fix for the SQL Server fixture race that parallel rows exposed. What is left for the Linux leg, more cores, is #253.