Skip to content

Tests: a README, and statistical consistency checks against a known posterior - #191

Merged
patnr merged 1 commit into
Python-Ensemble-Toolbox:mainfrom
patnr:tests-statistical-consistency
Oct 29, 2026
Merged

patnr merged 1 commit into
Python-Ensemble-Toolbox:mainfrom
patnr:tests-statistical-consistency

Conversation

@patnr

@patnr patnr commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator

Adds a README to tests/, and a new test module, test_statistical_consistency.py.

The README describes the cases the methods are run on end to end (Van der Pol, the linear 1D field, MiniRes, Rosenbrock and the quadratic), and which scheme and analysis is checked on which case, and how: pinned numbers, property tests, the benchmark, or the new checks. The unit and regression tests are left to their docstrings.

The new checks are a linear sanity check. With a linear model and a Gaussian prior and noise, the posterior is known exactly, so every registered scheme and analysis is compared with the correct answer, on its mean and its spread, rather than with previously pinned numbers. The cases are multivariate but not spatial: 1, 2 and 10 unknowns, with a random prior correlation, a dense random G, and noise variances spanning three orders of magnitude. The reference is the exact update of the prior ensemble's own mean and covariance, so prior sampling error does not count against a method; the rest is averaged over 5 seeds of 100 members.

Combinations left out are listed in EXCLUDED, each with its reason, and expected deviations in DEVIATIONS, as strict xfails, so that a fix shows up as a failure until its entry is removed. The tolerances were set from repeated runs with independent seeds.

The module runs in about 20 s.

Checked locally: the new module, and ruff.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@patnr
patnr merged commit 3b707ab into Python-Ensemble-Toolbox:main Oct 29, 2026
1 of 4 checks passed
@patnr
patnr deleted the tests-statistical-consistency branch October 29, 2026 12:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant