[professor3333]

Summary

reports/calibration/: calibration and review thresholds for crf-batches-01-02 and the best run-3 DeBERTa.

• Method: everything is fitted on the selection half of the model-drafted dev records and measured on the held-out half (61 docs). calibrate's internal halves were checked to equal splits/selection-v1.
• Scope: agreement with model drafts, not accuracy. The live review policy (review-policy-0) and the test split are untouched.

Results:

• Calibration error (held-out ECE): CRF 0.305 → 0.101; DeBERTa 0.548 → 0.059.
• Matched precision targets (thresholds from the selection half, scripts/calibration/compare_thresholds.sh):
  • DeBERTa keeps two to three times as many agreeing conditions.
  • Typed F1 is clearly higher at targets 0.30 and 0.35 (+0.100 [+0.034, +0.172], +0.100 [+0.024, +0.171]) and not clear at 0.40.
  • Exact span precision is not clearly different.
  • DeBERTa's false-prerequisite rate is higher at every target (+0.10 to +0.17), but not clearly.
• Decision: by the rule set beforehand (switch only if clearly better on precision or invented prerequisites), the CRF stays the default. DeBERTa is the leading candidate.

Code:

• Fix: fit_platt diverged on DeBERTa confidences (a ≈ 1.75e5, b ≈ −2.9e11) and Calibrator overflowed. It now uses damped Newton steps (step halving on the penalised log-loss) and an overflow-free sigmoid. The CRF's fitted parameters are unchanged. A regression test fails without the fix.
• New triggermap threshold PRED --min-confidence T --out OUT: keeps the predictions a reviewer would see unflagged, trims groups and fixes the document status. Tested.
• README results, model card (new "Calibration and default model" section) and evaluation guide updated.

Checks

• ruff check, ruff format --check, mypy src/, pytest -m "not slow" (243 passed); shellcheck clean; the comparison script ran end to end.