From fe06f94cb03a94ca933584d16a485908dc8ce24c Mon Sep 17 00:00:00 2001 From: patnr Date: Wed, 28 Oct 2026 14:56:44 +0100 Subject: [PATCH] Benchmark: a README, with its purpose, a disclaimer, and its layout Co-Authored-By: Claude Opus 5.5 --- benchmarks/minires/README.md | 55 ++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100644 benchmarks/minires/README.md diff --git a/benchmarks/minires/README.md b/benchmarks/minires/README.md new file mode 100644 index 0000000..3d98e00 --- /dev/null +++ b/benchmarks/minires/README.md @@ -0,0 +1,55 @@ +# MiniRes benchmark + +Twin experiments on a small waterflood, simulated with [MiniRes](https://github.com/patnr/MiniRes), +to compare PET's update schemes, analyses and localizations on equal terms: the same cases, the +same truths, the same scores. + +> [!WARNING] +> This is not (yet) a representative or authoritative benchmark. It has one reservoir, at two +> resolutions; few truths; and methods run at default or textbook settings, mostly untuned. +> Its results say how the methods fare on *this* case and *these* settings, and can change with +> either. Use it to catch regressions and to guide development, not to rank methods. + +## Purpose + +- **A score kept up to date.** `RESULTS.md` records how each method does, so that a change to PET's + code can be checked against it. +- **Scores a misfit cannot give.** The truth is known, so an update is scored on the unknowns + (the log-permeability field, and the residual oil saturation `sor`) and on a forecast that was + not assimilated, as well as on how well it fits the data. + +## Layout + +| | | +|---|---| +| `cases.py` | The cases: the reservoir, its wells and schedule, the prior, the observations, and the twin experiments drawn from them. | +| `methods.py` | The methods compared, each a label and its PET settings (`[dataassim]`). | +| `scores.py` | What an ensemble is scored on. | +| `run.py` | Runs the methods on the cases, and generates `RESULTS.md`. | +| `results/` | The scores of each case, per method and truth (JSON), from which `RESULTS.md` is generated. | +| `RESULTS.md` | The tables. Generated; do not edit by hand. | + +`tests/test_benchmark_minires.py` checks that each method still runs, on one tiny experiment. + +## Running + +Needs the `minires` extra (`pip install -e .[minires]`). From this directory: + +```sh +python run.py --quick # while developing: the coarse case, 5 truths, about a minute +python run.py # the official run: every method and case (over 10 minutes) +python run.py --method "ES-MDA" # only the methods whose label contains this +``` + +A quick run (fewer truths than the case's own) prints its table and writes nothing. It is +printed next to the official scores on the same truths; the experiments are seeded, so any +difference is due to the code. Its numbers are tentative: with so few truths, the differences +between methods are often within the noise. + +The official run writes `results/` and `RESULTS.md`. It is redone from time to time, not with +every change, so `RESULTS.md` may lag behind the methods and scores defined here. + +## Adding a method or a score + +- A method: an entry in `METHODS` of `methods.py`. +- A score: a key in the output of `scores.scores`, and a column in `COLUMNS` of `run.py`.