Skip to content

Benchmark: a row for ES, and the CRPS - #187

Merged
patnr merged 1 commit into
Python-Ensemble-Toolbox:mainfrom
patnr:benchmark-es-crps
Oct 28, 2026
Merged

patnr merged 1 commit into
Python-Ensemble-Toolbox:mainfrom
patnr:benchmark-es-crps

Conversation

@patnr

@patnr patnr commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

Adds to the MiniRes benchmark:

  • A row for ES: a single update, without localization.
  • The CRPS (continuous ranked probability score) of the ensemble, for the forecast water cut, the log-permeability field and sor. Unlike the RMSE of the ensemble mean, it also scores the spread: an ensemble that is confidently wrong scores worse than one that is wrong but knows it.

RESULTS.md is not rerun here, so it has neither yet. A score missing from the official results shows as blank.

Tentative numbers, from run.py --quick (coarse case, 5 truths; means):

misfit CRPS forecast WWCT logK RMSE logK CRPS sor CRPS simulations
prior 97.4 0.103 0.772 0.437 0.0178
ES (one update) 14.4 0.060 0.856 0.494 0.0203 100
ES-MDA (4 steps, α = 4) 1.56 0.039 0.795 0.464 0.0269 250
adaptive localization (universal threshold) 16.1 0.039 0.748 0.421 0.0124 250
adaptive localization (3σ threshold) 11.8 0.031 0.741 0.416 0.0117 250
distance localization (Gaspari-Cohn, 300 m) 8.67 0.028 0.730 0.407 0.0165 250

The ES row's 100 simulations include the simulation of the posterior.

Checked locally: the benchmark's smoke test, and ruff.

🤖 Generated with Claude Code

RESULTS.md is not rerun. A score missing from the official results is
left blank, and --quick prints no official table when it shares no row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@patnr
patnr merged commit b2add92 into Python-Ensemble-Toolbox:main Oct 28, 2026
4 checks passed
@patnr
patnr deleted the benchmark-es-crps branch October 29, 2026 12:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant