fix(pricing): price Delegate's underscored model ids and flag unpriced judge/simulator cost

[abhiram-vad]

Motivation

The Delegate backend names its models with underscores (gpt_5_6_luna, claude_opus_5_5), and task configs use these names. pricing.py keys Delegate models with hyphens (gpt-5-6-luna). The delegate-sdk adapter in coder_eval_uipath changed _ to - before it priced a turn. The port to the built-in agent dropped that step, so every agent turn on an underscored id priced at None, and the pre-flight warned "No pricing rate".

Example from the Scribe suite: in run adhoc-2026-10-08_21-33-28 (ADO 20261008.13, gpt_5_6_luna), the task pdd-interview-client-portfolio-reporting used 34.6M agent tokens, about $1.11 at the luna rate. It booked $0.79, which is exactly the sum of its five judges.

The same run showed two more gaps:

• Its simulator ran on claude-sonnet-5-5, which had no rate in 0.12.12 (fix(pricing): add Haiku 5.5, Sonnet 5.5 and GPT-6.1 Sol, re-verify every row #220 added it). The pre-flight did not warn, because it did not check the simulator model.
• cost_complete stayed true when a judge or the simulator was unpriced, although REPORT_SCHEMA.md and the pre-flight warning say that an unpriced model makes it false.

Changes

• pricing.delegate_rate_key changes _ to -. DelegateAgent prices each turn under it, and check_pricing_coverage looks up a Delegate agent's model under it. model_used keeps the backend's own id. _normalize_model and the rates do not change.
• Evalboard: rateKey in lib/pricing.ts mirrors the rule. A row does not record its harness, so resolvePricing tries it only after the exact id, and tallyModels groups on it.
• The pre-flight also checks the model of each enabled simulation.
• _cost_complete also returns false when a judge or the simulator used tokens with no cost. The pre-flight warning now names the simulator turn.
• Docs: .claude/notes/agents.md, docs/REPORT_SCHEMA.md.

Testing

• CI passes (17 of 17 checks).
• These tests fail on main and pass here: a Delegate turn on gpt_5_6_luna books $1.40 for 1M input and 1M output tokens; the pre-flight accepts a Delegate task on gpt_5_6_luna and flags an unpriced simulator model; the board prices gpt_5_6_luna and counts it as one model with gpt-5-6-luna; cost_complete is false for an unpriced judge or simulator.
• coder-eval run against a fake host that echoes the configured id:

Model in the task	main	This PR
gpt_5_6_luna	None, pre-flight warns	$1.40, no warning
claude_opus_5_5	None, pre-flight warns	$24.00, no warning
gpt_6_luna	None, pre-flight warns	$0.60, no warning
gpt-5-6-luna	$1.40	$1.40
gpt_9_imaginary (no rate)	None, pre-flight warns	None, pre-flight warns
gpt_5_6_luna on the pi harness	pre-flight warns	pre-flight warns

🤖 Generated with Claude Code

[github-actions]

Claude finished @abhiram-vad's task in 1m 59s —— View job

---

PR Review in Progress

Todo List

• Read review guidelines in .github/code_review.md
• Read CLAUDE.md for project conventions
• Get full diff with git diff origin/main...HEAD
• Read each changed file in full context
• Perform cross-file consistency checks
• Analyze what's missing
• Provide design-level feedback
• Submit review feedback

Starting review of PR #223: fix(delegate): price the underscored model ids the Delegate backend echoes

[uipreliga]

Review: coder_eval — PR #223 "fix(delegate): price the underscored model ids the Delegate backend echoes" by @abhiram-vad · abhiram/fix/delegate-underscore-model-pricing → main · OPEN

Scope: PR #223 "fix(delegate): price the underscored model ids the Delegate backend echoes" by @abhiram-vad · abhiram/fix/delegate-underscore-model-pricing → main · OPEN · branch pr-223 (abhiram/fix/delegate-underscore-model-pricing) · 96edd48 · 2026-10-12T19:06Z · workflow variant

Change class: simple — maps the Delegate backend's underscored model ids to the rate-card key at the two pricing lookups (turn cost + pre-flight), with tests; no schema, control-flow or rate change

The codebase is healthy: no finding can change a task's score or final_status for identical agent output, type safety and security have no findings, and the only real risk is that the Delegate underscore-to-hyphen rate-key mapping sits outside the shared pricing seam, so cost accounting can drift again between the pre-flight, the runtime agent and the evalboard; the bottom line is to merge after you move that mapping into one shared place and add the missing inverse and fallback tests.

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	9.9 / 10	0	0	0	1	_run_models branch re-tests agent None-ness and splits one yield across two arms
2. Type Safety	10 / 10	0	0	0	0	—
3. Test Health	9.8 / 10	0	0	0	2	The new Delegate-only branch in _run_models is tested only on its happy path; nothing pins that other agents' ids are NOT rewritten
4. Security	10 / 10	0	0	0	0	—
5. Architecture & Design	9.5 / 10	0	0	1	0	Delegate rate-key mapping applied at two call sites, with an AgentKind.DELEGATE branch in core batch.py pre-flight that goes around the agent SPI and the shared _normalize_model seam
6. Error Handling & Resilience	9.9 / 10	0	0	0	1	Delegate _finalize_turn still drops an unpriced turn's cost to None without a per-turn diagnostic
7. API Surface & Maintainability	9.8 / 10	0	0	0	2	The pre-flight warning and the return value now name the rewritten rate key, not the model id the user wrote
8. Evaluation Harness Quality	9.9 / 10	0	0	0	1	delegate_rate_key is a second normalization seam outside the mirrored _normalize_model, so the evalboard and Python/run.json resolve underscored Delegate ids differently

Overall Score: 9.9 / 10 · Weakest Axis: Architecture & Design at 9.5 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 1 · 🔵 7 across 8 axes.

Blockers

None.

Non-blocking, but please consider before merge

1. [Axis 5] Delegate rate-key mapping applied at two call sites, with an AgentKind.DELEGATE branch in core batch.py pre-flight that goes around the agent SPI and the shared _normalize_model seam (src/coder_eval/orchestration/batch.py:57) — _run_models used to treat every agent the same way (yield rt.task.agent.model if rt.task.agent else None). The PR replaces that with a hardcoded dispatch on one agent kind:

agent = rt.task.agent
if agent is not None and agent.model and agent.type == AgentKind.DELEGATE:
    yield delegate_rate_key(agent.model)
else:
    yield agent.model if agent else None

Before this PR, batch.py named only AgentKind.UNKNOWN. CLAUDE.md says agents register through the plugin SPI, that "there is no closed enum or dispatch to edit, and in-tree and third-party agents take the same path". Review criterion 16 rejects if x.type == ... ladders in core. This PR creates exactly such a dispatch: the in-tree Delegate agent now gets a core-side id-to-rate-key hook that a third-party agent cannot get without editing orchestration/batch.py. The id knowledge is also split across three modules: the rule in pricing.py:62, the call in delegate_agent.py:1069 and the branch here. Each call site that turns a Delegate model into a rate key must remember to apply it.

One fix removes the branch and both call-site wrappers. Put the mapping in the shared lookup as an exact-first fallback, for example in _lookup_rate / is_priced / calculate_cost: try key, then key.replace("_", "-"). No built-in rate key contains _, so built-in prices do not change. An underscore-keyed plugin rate is matched exactly first, so it still resolves to its own rate. After that, _run_models returns to agent-agnostic code and DelegateAgent calls calculate_cost(state.model_used or ...) with no wrapper. If the mapping must stay agent-scoped, use a polymorphic hook instead (for example an AgentConfig.rate_key() that defaults to self.model and that DelegateAgentConfig overrides). Do not branch on the enum in batch.py.

Nits

1. [Axis 1] _run_models branch re-tests agent None-ness and splits one yield across two arms (src/coder_eval/orchestration/batch.py:57) — if agent is not None and agent.model and agent.type == AgentKind.DELEGATE: / yield delegate_rate_key(agent.model) / else: / yield agent.model if agent else None checks agent twice and yields the same model through two paths. radon now scores it B(7), up from about B(5). If you keep the branch, compute the model once: model = agent.model if agent else None and then one yield. If you take the generic fix in the previous finding, restore the original single line yield rt.task.agent.model if rt.task.agent else None.
2. [Axis 3] The new Delegate-only branch in _run_models is tested only on its happy path; nothing pins that other agents' ids are NOT rewritten (tests/test_cost_accounting_paths.py:93) — The new test test_delegate_backend_id_is_priced_under_its_rate_card_key (line 93: resolved.task.agent = DelegateAgentConfig(type="delegate", model="gpt_5_6_luna")) covers only the Delegate side of the new guard in batch.py:57 (if agent is not None and agent.model and agent.type == AgentKind.DELEGATE:). No test covers the inverse. If someone deleted the agent.type == AgentKind.DELEGATE condition, or moved the replace("_", "-") into is_priced, every test would still pass. A claude-code task with model: claude_sonnet_5 would then pass the pre-flight silently, even though the runtime agent (which does not call delegate_rate_key) books that model's turns at cost None. The pre-flight and the runtime would disagree, and that disagreement is the defect this PR fixes. Add a mirror test: _resolved("claude_sonnet_5", tmp_path) (claude-code agent) gives check_pricing_coverage(...) == ["claude_sonnet_5"] and logs the warning. Also give _resolved an agent= parameter so the test does not have to replace task.agent after construction, because that replacement skips TaskDefinition's own validation.
3. [Axis 3] The underscored-id agent test cannot tell whether the configured-model fallback is normalized, because the echoed id and the configured id are the same (tests/test_delegate_agent.py:1017) — test_cost_prices_the_underscored_id_the_backend_echoes sets the same underscored id on both sides: model="gpt_5_6_luna" in _result(...) (line 1013) and _started_agent(..., model="gpt_5_6_luna") (line 1017). The production expression is delegate_rate_key(state.model_used or self.config.model or "") (delegate_agent.py:1069). This test only exercises the state.model_used operand. Suppose someone narrowed the fix to delegate_rate_key(state.model_used) if state.model_used else self.config.model. This test would still pass, but a backend frame that echoes no model would leave a config written as model: gpt_5_6_luna unpriced. That config is the exact scenario the new notes paragraph describes. The existing no-echo test, test_cost_wired_into_record_for_configured_model, uses only a dashed id (virtuoso-1-5). Parametrize the new test over (echoed=None, config="gpt_5_6_luna") and (echoed="gpt_5_6_luna", config=None). Both cases must assert total_cost_usd == pytest.approx(0.20 + 1.20).
4. [Axis 6] Delegate _finalize_turn still drops an unpriced turn's cost to None without a per-turn diagnostic (src/coder_eval/agents/delegate_agent.py:1075) — The Delegate backend reports no cost of its own, so this rate-card lookup is the only thing that prices any Delegate turn. A miss (if cost is not None: usage = usage.model_copy(...) at lines 1075-1076, with no else branch) leaves total_cost_usd as None and logs nothing. This silent miss is what let the underscored-id defect ship without anyone noticing. After this PR, two cases still miss without a log line: a Delegate model left to the route (config.model unset, so the pre-flight in batch._run_models cannot see it), and a backend-echoed state.model_used that is not in _DELEGATE_PRICING. The run is not totally blind: the run-level RunSummary.cost_complete flag and the orchestrator's warning 'max_usd budget configured but no turn reported cost' (orchestrator.py:1174) both still fire. That is why this is Low, not Medium. ClaudeCodeAgent._backfill_cost and _reprice_for_litellm both log a warning on the same miss. Fix: add an else: logger.warning("No pricing for delegate model %r (rate key %r); turn cost left unset", state.model_used or self.config.model, key) that names both the raw id and the normalized key. With that line, the next id-shape mismatch shows up on the first turn and not in a downstream suite's cost totals.
5. [Axis 7] The pre-flight warning and the return value now name the rewritten rate key, not the model id the user wrote (src/coder_eval/orchestration/batch.py:58) — _run_models now yields delegate_rate_key(agent.model), so unpriced_models returns, and the warning at line 84 ("No pricing rate for %s. ...") prints, the hyphenated key. Example: a task with model: gpt_9_imaginary warns about 'gpt-9-imaginary', which is a string that does not occur in the user's YAML. This also changes the documented return contract at line 78 (The sorted, de-duplicated unpriced model ids): it now returns rate-card keys, not configured model ids. Recommendation: check the rate key, but report the authored id (for example, collect (authored, key) pairs and return or log the authored one, or the form gpt_9_imaginary (rate key gpt-9-imaginary)). Also update the docstring so the return value is stated without ambiguity.
6. [Axis 7] The comment on the Delegate rate block still says the ids are hyphenated, but the new helper is about underscored ids and is also used for non-Delegate keys (src/coder_eval/pricing.py:62) — delegate_rate_key sits directly under _DELEGATE_PRICING / DELEGATE_MODEL_IDS, whose comment at line 37 still reads # Models the Delegate agent serves, under the SDK's own hyphenated ids. But the helper also maps backend ids that land in the general _PRICING table. The PR body shows claude_opus_5_5 -> claude-opus-5-5 priced at the Anthropic rate. A reader can therefore think the helper only applies to _DELEGATE_PRICING keys. Recommendation: change the block comment to say that the backend echoes underscored ids and that delegate_rate_key is the mapping for every id a Delegate agent reports, not only the ids in this block. Or move the helper to the single seam suggested in finding 1, so this comment does not need to cover it.
7. [Axis 8] delegate_rate_key is a second normalization seam outside the mirrored _normalize_model, so the evalboard and Python/run.json resolve underscored Delegate ids differently (src/coder_eval/pricing.py:62) — def delegate_rate_key(model: str) -> str: ... return model.replace("_", "-") sits beside _normalize_model. evalboard/lib/pricing.ts:37 says its normalizeModel is the "mirror of src/coder_eval/pricing.py::_normalize_model". Before this PR, Python and the board both priced gpt_5_6_luna as unpriced, which was at least consistent. Now the Python task total shows the full cost, while the board's Tokens-to-USD view, the thinking-cost simulator and tallyModels show —, or split gpt_5_6_luna and gpt-5-6-luna into two models. The PR body admits this gap and defers the fix. CE065 compares only the rate table, not normalization, so nothing catches the drift. Low because the board's task total and per-message Cost column stay correct. Track the board fix (port delegateRateKey to lib/pricing.ts at the same call sites). Also consider extending the CE065 parity check to cover the normalization helpers, so the next normalization rule cannot drift silently.

What's Missing

Daily/nightly:

• 🟡 The PR names the Scribe suite run, but it does not say what happens to the nightly pipeline. After merge, Delegate rows on underscored ids go from None to a real agent cost. In the example run that is about +$1.11 on a $0.79 judge-only total. RunSummary.cost_complete flips to true, and the downstream cost series show a step change. Runs already in blob storage (for example adhoc-2026-10-08_21-33-28) stay understated, because evaluate <run_dir> re-grades but does not re-price agent turns, and the PR gives no backfill path. State this in the PR, or give a one-off re-price recipe for past runs. (trigger: src/coder_eval/agents/delegate_agent.py)

Downstream consumers:

• 🔵 The orchestrator's max_usd gate (orchestrator.py:1169-1180) skipped Delegate turns on underscored ids before, because every turn reported cost None ('no turn reported cost; skipping cost check'). It now enforces the budget. A nightly Delegate task with a tight max_usd that always passed can now stop with a budget error. Nothing in the PR or in docs/agents/DELEGATE.md says that this gate is newly live. (trigger: src/coder_eval/agents/delegate_agent.py)

Display & mapping dicts:

• 🔵 evalboard/lib/pricing.ts normalizeModel/resolvePricing (the documented mirror of _normalize_model) does not get the _->- mapping. Underscored Delegate ids therefore show '—' in the Tokens-to-USD view and in the thinking-cost simulator, and tallyModels counts the same model twice. The PR body acknowledges this and defers it. CE065 diffs only the rate table, so nothing tracks it. (trigger: src/coder_eval/pricing.py) _(restates: Axis 8: delegate_rate_key is a second normalization seam outside the mirrored normalize_model)

Tests:

• 🔵 No test covers the unpriced side of the new Delegate pre-flight branch. A Delegate task with an underscored id that has no rate (for example gpt_9_imaginary) should still return a non-empty list and log 'No pricing rate'. No test pins which spelling the warning and the return value use (the authored id or the rewritten key). (trigger: tests/test_cost_accounting_paths.py) (restates: Axis 7: The pre-flight warning and the return value now name the rewritten rate key, not the model id the user wrote)
• 🔵 No test covers the inverse of the Delegate-only branch in batch._run_models: an underscored id on a non-Delegate agent (claude-code or pi) must still be reported as unpriced. The PR body's manual table shows this for pi, but no test pins it. (trigger: src/coder_eval/orchestration/batch.py) _(restates: Axis 3: The new Delegate-only branch in run_models is tested only on its happy path)
• 🔵 The configured-model fallback (state.model_used is None and config.model is gpt_5_6_luna) is not tested through delegate_rate_key. The new agent test sets the same underscored id on both the echoed operand and the configured operand. (trigger: tests/test_delegate_agent.py) (restates: Axis 3: The underscored-id agent test cannot tell whether the configured-model fallback is normalized)

Parallel paths:

• 🔵 The user-facing Delegate guide is not updated. docs/agents/DELEGATE.md line 126 documents model_used as the result frame's model, and every example uses dashed ids (virtuoso-1-5). The guide does not say that underscored backend ids (gpt_5_6_luna, claude_opus_5_5) are accepted in agent.model, or that cost is priced under the hyphenated rate key while model_used keeps the backend spelling. The rationale went only into .claude/notes/agents.md, which users do not read. (trigger: .claude/notes/agents.md)

Harness & Lint Improvements

Static checks (lint / type):

• [ce-lint] CE070 no-agent-kind-dispatch-in-core: an AST rule in tests/lint/rules/ce070_no_agent_kind_dispatch_in_core.py, wired in tests/lint/runner.py. It flags any Compare node (==, !=, is, in) and any match/case pattern whose operand is a concrete AgentKind.<MEMBER> (or AgentKind.<MEMBER>.value). Scope: src/coder_eval/ except agents/ and plugins/. The sentinel members NONE and UNKNOWN are allowed because they mean 'no agent', not 'this agent'. Construction such as parse_agent_config(type=AgentKind.CLAUDE_CODE) in cli/evaluate_command.py:189 is not a comparison, so the rule does not flag it. Today the rule finds one existing hit: config.py:203 (agent_type == AgentKind.CLAUDE_CODE.value in validate_api_keys). Fix that hit with a polymorphic hook, or record it as a documented noqa: CE070 with a reason. The rule enforces the CLAUDE.md contract 'no closed enum or dispatch to edit'. Per-agent behaviour must go through the agent SPI or an AgentConfig method, for example AgentConfig.rate_key(). It must not go through a branch in orchestration/. Prevents: The medium Architecture finding at src/coder_eval/orchestration/batch.py:57: if agent is not None and agent.model and agent.type == AgentKind.DELEGATE: adds a Delegate-only branch to the core pre-flight. It also prevents the follow-on Code Quality finding at the same line (the double None check and one yield split across two arms, radon B(7)), because that finding exists only because of the branch.
• [ce-lint] Extend CE065 (tests/lint/pricing_mirror.py) so it mirrors the model-id normalizer, not only the rate table. Move the normalizer data out of _normalize_model into module-level constants in src/coder_eval/pricing.py: the routing prefixes, _BEDROCK_REGION_PREFIXES, the anthropic. vendor prefix, and any character-substitution rule such as Delegate's _->- once that rule moves into the shared seam. render_pricing then emits these constants into evalboard/lib/pricing.generated.ts, and normalizeModel in evalboard/lib/pricing.ts reads them from that file. It must not keep its own hand-copied _ROUTING_PREFIXES / _REGION_PREFIXES arrays. After this change, a new normalization rule in Python that is not regenerated fails make verify as CE065 drift. Also add an assertion to the generator: is_priced and calculate_cost must reach the model id only through _normalize_model, and pricing.py must export no other public function that rewrites a model id (an AST check for a public def in pricing.py that returns model.replace(...) / removeprefix(...)). This blocks a second seam such as delegate_rate_key. Prevents: The Harness Quality finding at src/coder_eval/pricing.py:62: delegate_rate_key is a second normalization seam outside the mirrored _normalize_model, so Python and run.json price gpt_5_6_luna while the evalboard shows — or splits the model in two. It also prevents the API Surface finding at pricing.py:62 (the stale 'hyphenated ids' block comment that describes the helper), because the helper would be folded into the single seam.
• [ce-lint] Remove the sharp edge first, then add the rule. Add one helper to pricing.py, for example price_turn(model, ...) -> float | None. It wraps calculate_cost and logs one warning per distinct id on a miss, naming both the raw id and the normalized key (No pricing for model %r (rate key %r); turn cost left unset). Then add CE071 agents-price-through-price-turn (tests/lint/rules/ce071_agents_price_through_price_turn.py, wired in runner.py). It forbids importing or calling calculate_cost / _lookup_rate from src/coder_eval/agents/. Today these call sites call calculate_cost directly, and most have no miss diagnostic: antigravity_agent.py:237, codex_agent.py:2292/2348/2384, delegate_agent.py:1068, opencode_agent.py:430, pi_agent.py:585 (claude_code_agent.py:1569/1593 logs on a miss). The rule also stops an agent from wrapping the id at its own call site (calculate_cost(delegate_rate_key(...))), because id mapping then has only one entry point. Prevents: The Error Handling finding at src/coder_eval/agents/delegate_agent.py:1075: an unpriced Delegate turn drops cost to None with no per-turn log. This silent miss is what let the underscored-id defect ship. The rule also prevents the per-call-site wrapper part of the Architecture finding (delegate_agent.py:1069), and it closes the same silent-miss gap in the other agents that do not log.

Harness improvements (not statically reachable):

• Add a pre-flight vs runtime pricing parity contract test, tests/test_pricing_preflight_parity.py. It is parametrized over every registered AgentKind (taken from the plugin registry, not a hand list), and for each kind over a set of model-id spellings: a canonical dashed id, the underscored backend form, a routing-prefixed form, and an unknown id. For each case it asserts that check_pricing_coverage (orchestration/batch.py _run_models) reports the id as unpriced exactly when that agent's runtime cost path (driven through a stub turn) returns total_cost_usd is None. A Delegate-only happy-path test cannot catch a guard that was deleted or widened. This matrix catches it in both directions. Why not static: Pre-flight and runtime agree only if two code paths compute the same answer for the same input. This needs both paths to run, including each agent's turn-finalization logic. An AST check cannot evaluate what id an agent passes to the rate card at runtime. Prevents: The Test Health finding at tests/test_cost_accounting_paths.py:93 (only the Delegate side of the batch.py:57 guard is tested; nothing pins that claude-code ids are not rewritten). It also prevents the root defect class behind the Architecture finding at batch.py:57: pre-flight and runtime disagree about whether a model is priced.
• Add a shared per-agent cost ContractCase table, modeled on CE036's live_verdict_contract. Each agent declares cases over (echoed model_used, configured config.model): (echoed, None), (None, configured), (echoed, configured), each with a dashed and an underscored id. Each case gives the expected total_cost_usd. One generic test runs every agent through those cases. Add a lint-marked test in tests/test_custom_lint.py that fails when an agent registers without cost cases, the same way CE036 fails for a live criterion without ContractCases. Why not static: The contract is about the value an agent computes from a backend frame and its config, through the expression state.model_used or self.config.model. Only executing that expression shows whether each operand is normalized. A static rule can require that the cases exist. It cannot check that they pass. Prevents: The Test Health finding at tests/test_delegate_agent.py:1017: the underscored-id test uses the same id for the echoed model and the configured model, so the configured-model fallback at delegate_agent.py:1069 is never tested on its own.
• Add a cross-language normalizer golden-vector fixture, for example tests/fixtures/model_id_normalization.json, that maps a raw id to the expected rate key: eu.anthropic.claude-opus-4-8 -> claude-opus-4-8, openrouter/deepseek/deepseek-v4-pro -> deepseek/deepseek-v4-pro, gpt_5_6_luna -> gpt-5-6-luna, and so on. Both a pytest test (_normalize_model) and a vitest test (normalizeModel in evalboard/lib/tests/) read this one file, so make verify and make evalboard-verify both fail when the two implementations diverge. Why not static: The constants can be mirrored by generation (see the CE065 extension), but the control flow of normalizeModel stays hand-written TypeScript: prefix order, break-after-first-match, trim. Two implementations in two languages are behaviourally equivalent only if both run on the same inputs. Prevents: The Harness Quality finding at src/coder_eval/pricing.py:62: Python/run.json and the evalboard resolve underscored Delegate ids differently (the Tokens-to-USD view, the thinking-cost simulator and tallyModels show — or split one model in two).
• In tests/test_cost_accounting_paths.py, add a caplog assertion that the pre-flight warning and the check_pricing_coverage return value name the model id exactly as the task YAML wrote it (for example gpt_9_imaginary). If the key is also shown, it must use the form gpt_9_imaginary (rate key gpt-9-imaginary). Update the _run_models / check_pricing_coverage docstring so the return contract says whether it returns authored ids or rate keys. Why not static: Whether a user-facing message names the string the user typed is a semantic property of runtime output. It needs a resolved task and captured logs. No AST pattern tells an authored id from a derived key. Prevents: The API Surface finding at src/coder_eval/orchestration/batch.py:58: the pre-flight warning and the return value name the rewritten rate key gpt-9-imaginary, which does not appear in the user's YAML. This also drifts from the documented return contract ('unpriced model ids').

Top 5 Priority Actions

1. Remove the hardcoded agent.type == AgentKind.DELEGATE branch at src/coder_eval/orchestration/batch.py:57. Put the _ to - mapping in the shared pricing lookup instead (_normalize_model / _lookup_rate in src/coder_eval/pricing.py:62, try the exact key first, then the hyphenated key), or use a polymorphic AgentConfig.rate_key() hook. Then delete the delegate_rate_key wrapper at src/coder_eval/agents/delegate_agent.py:1069 and restore the single agent-agnostic yield, which also fixes the double None-check in _run_models.
2. Add an else: logger.warning(...) at src/coder_eval/agents/delegate_agent.py:1075 that names the raw model id and the rate key. Without it, an unpriced Delegate turn (a route-chosen model or an unknown echoed id) sets total_cost_usd to None with no log, which is how the underscored-id defect shipped without notice.
3. Port the rate-key normalization to evalboard/lib/pricing.ts:37 (normalizeModel), so the board and Python/run.json price gpt_5_6_luna the same way. Extend the CE065 parity check to diff the normalization helpers as well as the rate table, so the next normalization rule cannot drift silently.
4. Add an inverse test in tests/test_cost_accounting_paths.py:93 that shows a non-Delegate agent with model: claude_sonnet_5 stays unpriced in check_pricing_coverage. Give _resolved an agent= parameter so the test keeps TaskDefinition validation. Also parametrize tests/test_delegate_agent.py:1017 over (echoed=None, config="gpt_5_6_luna") and (echoed="gpt_5_6_luna", config=None), so the configured-model fallback is pinned.
5. Make the pre-flight at src/coder_eval/orchestration/batch.py:58-84 report the model id the user wrote (for example gpt_9_imaginary (rate key gpt-9-imaginary)), not only the rewritten key, and update the _run_models/unpriced_models docstring. Also correct the stale "hyphenated ids" comment at src/coder_eval/pricing.py:37, or delete it when the mapping moves to the shared seam.

---

Stats: 0 🔴 · 0 🟠 · 1 🟡 · 7 🔵 across 8 axes reviewed.