From 40db1d4c3dff6f4a792aeecec1cbb37650f71273 Mon Sep 17 00:00:00 2001 From: lmoresi Date: Sat, 10 Oct 2026 11:02:07 -1000 Subject: [PATCH] A proven new test starts at tier B; tier C is for unproven tests Louis, 2026-10-11: tier C is virtually obsolete, there to keep an unproven test from derailing the code. The testing document's guideline "all new tests begin at tier C" contradicted that and its own C1/C2 sections: a regression test written first and shown to fail, or a test asserting a checked hard baseline, now starts at tier B and gates. CLAUDE.md's tests section says the same, and test_0025 (the -fno-math-errno pin, shown failing before its fix) moves to tier B. Underworld development team with AI support from Claude Code --- CLAUDE.md | 9 +++++---- docs/developer/TESTING-RELIABILITY-SYSTEM.md | 8 ++++++-- tests/test_0025_jit_compile_flags.py | 2 +- 3 files changed, 12 insertions(+), 7 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 8caffb4d8..0c7978aa5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -204,10 +204,11 @@ pytest -m "level_1 and tier_a" # quick pytest -m "tier_a or tier_b" # full validation ``` -Tier A is hardened and reviewed — safe to build code around. Tier C is not mature -enough to drive coding. Every bug fix ships the regression test that would have -caught it, written first and shown to fail. Validate a new test's own correctness -before changing library code to satisfy it. +Tier A is hardened and reviewed — safe to build code around. Every bug fix ships +the regression test that would have caught it, written first and shown to fail; +such a proven test starts at tier B, so it gates. Tier C is only for a test not yet +proven, so it cannot derail the code. Validate a new test's own correctness before +changing library code to satisfy it. Every test file must be reachable from a glob in `scripts/test.sh`; `scripts/check_test_coverage.py` enforces it. diff --git a/docs/developer/TESTING-RELIABILITY-SYSTEM.md b/docs/developer/TESTING-RELIABILITY-SYSTEM.md index 4e8d072c8..a6d6f5ee8 100644 --- a/docs/developer/TESTING-RELIABILITY-SYSTEM.md +++ b/docs/developer/TESTING-RELIABILITY-SYSTEM.md @@ -291,11 +291,15 @@ pytest -m "not tier_c" ### When Writing a New Test -1. **Start at Tier C**: All new tests begin as experimental +1. **A proven test starts at Tier B**: a regression test written first and shown to + fail without the fix it guards, or a test asserting a hard baseline whose own + correctness has been checked, gates from the start. Tier C is for a test that is + not yet proven, so that it cannot derail the code it was written against (the C2 + population above); it is otherwise close to obsolete (Louis, 2026-10-11). 2. **Document Intent**: Clear docstring explaining what behavior is tested 3. **Use xfail Appropriately**: If testing unimplemented feature, mark with xfail 4. **Don't Break CI**: Tier C tests with xfail won't break automated testing -5. **Promote Deliberately**: Don't rush promotion - let tests prove reliability +5. **Promote Deliberately**: Tier B → A is the review step described above ### When a Test Fails diff --git a/tests/test_0025_jit_compile_flags.py b/tests/test_0025_jit_compile_flags.py index 5fe304997..006144840 100644 --- a/tests/test_0025_jit_compile_flags.py +++ b/tests/test_0025_jit_compile_flags.py @@ -14,7 +14,7 @@ import underworld3 as uw -pytestmark = [pytest.mark.level_1, pytest.mark.tier_c] +pytestmark = [pytest.mark.level_1, pytest.mark.tier_b] def _compile_args(bundles):