Skip to content

test: run the analyzer sweeps' rows in parallel, and leave the pragma pass to each rule's focused tests - #250

Merged
CaffeinatedCoder merged 1 commit into
release/2.3.1from
test/analyzer-sweeps-parallel-2-3-1
Oct 31, 2026
Merged

CaffeinatedCoder merged 1 commit into
release/2.3.1from
test/analyzer-sweeps-parallel-2-3-1

Conversation

@CaffeinatedCoder

Copy link
Copy Markdown
Owner

Closes #245

Problem

Since #172, a full CI run takes 17-19 minutes instead of about 10. DroppedTaskVariantTests is one class of 1,838 rows, which xunit's default mode (one collection per class) ran one after another. In the test results of runs 41180296321 and 41178154299, every other test of every project has finished by 334-453 s. The theory then runs alone on one core for another two to seven minutes. Each of its reporting rows also ran a second time under #pragma warning disable: the same check for every shape, 28% of the theory's time.

Change

  • xunit.runner.json with "parallelMode": "all" for CodoMetis.TypeKit.Analyzers.Tests, copied to the output by the project file, which says why. Every row of every theory is now scheduled on its own. The [assembly: Parallelization] attribute the xunit docs show is not in the pinned 4.0.2. No test in the project shares state (each builds its own compilation), so none opts out.
  • VariantSweep.Behaviors (AnalyzerTest.cs): TestBehaviors.SkipSuppressionCheck, set by the four sweeps: DroppedTaskVariantTests, TypeArgumentConstraintKindTests, QueryDelegateConditionTests and TypeArgumentSiteRun (TypeArgumentConstraintVariantTests). The focused tests of each rule (IgnoredOutcomeAnalyzerTests, TypeArgumentConstraintAnalyzerTests, DefaultProducingCallAnalyzerTests) keep the pragma pass. The verifier's generated-code pass already never ran here: it is skipped for an analyzer that configures generated-code analysis, and every CMTK analyzer does.

Test code only: no package, no generated member, no changelog entry.

Tests

  • The whole analyzer test project, locally (16 cores, two runs each): 3m 14s and 3m 12s on release/2.3.1, then 1m 29s and 1m 27s on this branch, with 3,947 passing on both. The theory alone at 4 threads, as on a CI runner: 186 s down to 70 s. This PR's own run gives the CI before and after on both legs, which go into CI: run the analyzer variant sweeps' rows in parallel, and skip the harness's rule-wide passes there #245.
  • The whole solution passes in the worktree: 35,336 passed, 1,248 skipped (SQL Server query-variant rows, unchanged).
  • Seeded twice, each reverted by its inverse edit:
    • CMTK0003 skipping the first element of an array of tasks: DroppedTaskVariantTests fails 2 rows under the new mode, so the sweep still catches its own defects.
    • CMTK0003 reporting a probe rule only below a leading #pragma warning disable: IgnoredOutcomeAnalyzerTests fails 16 tests in the pragma pass, and DroppedTaskVariantTests passes. The pass is now the focused tests', as intended.

🤖 Generated with Claude Code

… pass to each rule's focused tests

DroppedTaskVariantTests is one class of 1,838 rows, which xunit's default mode ran one after another. On CI
it kept the test step going for two to seven minutes after every other project had finished (#245). The
analyzer test project now runs with parallelMode "all" (xunit.runner.json), so every row of every theory
is scheduled on its own. No test there shares state: each builds its own compilation.

The four sweeps (DroppedTaskVariantTests, TypeArgumentConstraintKindTests, QueryDelegateConditionTests and
TypeArgumentSiteRun, which TypeArgumentConstraintVariantTests runs) set VariantSweep.Behaviors, which skips
the verifier's second run of each reporting case under '#pragma warning disable'. Whether a pragma silences
a rule is the same for every shape, and the focused tests of each rule (IgnoredOutcomeAnalyzerTests,
TypeArgumentConstraintAnalyzerTests, DefaultProducingCallAnalyzerTests) keep it. The verifier's
generated-code pass already never ran: every analyzer configures generated-code analysis.

Seeded twice, each reverted by its inverse edit:
- CMTK0003 dropping the first element of an array's tasks: DroppedTaskVariantTests fails 2 rows under the
  new mode, so the sweep still catches its own defects.
- CMTK0003 reporting a probe rule only below a leading '#pragma warning disable': IgnoredOutcomeAnalyzerTests
  fails 16 tests in the pragma pass, DroppedTaskVariantTests passes. The pass is now the focused tests'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CaffeinatedCoder CaffeinatedCoder added this to the 2.3.1 milestone Oct 31, 2026
@CaffeinatedCoder CaffeinatedCoder added the tests Adds or changes tests only (tests, probes, harnesses); no library code label Oct 31, 2026
@CaffeinatedCoder
CaffeinatedCoder merged commit c8fb57c into release/2.3.1 Oct 31, 2026
10 checks passed
@CaffeinatedCoder
CaffeinatedCoder deleted the test/analyzer-sweeps-parallel-2-3-1 branch October 31, 2026 09:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tests Adds or changes tests only (tests, probes, harnesses); no library code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant