[CMGS]

Problem

The failure streak and cooldown were keyed by account name only. A refusal that only concerns one model therefore benched the account for every model it serves. OpenRouter's peak-load refusal for one model ("exceeds the maximum usage size allowed during peak load") is an example, and it often arrives as an error event inside a 200 stream.

On a single-account pool, that means every caller gets no healthy upstream account serves model type ... for 30 s after three such refusals, including callers of unrelated models. In a 2h15m window, one model's refusals tripped the only Responses account 36 times and produced 211 no healthy upstream refusals for callers of other models.

Change

• gw_state::health_key(account, model). The health store records failures, successes and cooldowns per account and served model (after variant and fallback resolution).
  • AccountPool::select_healthy takes the served model and skips only the accounts cooling down for it.
  • The engine call, its one-shot failover and the realtime handshake all pass the served model.
• account_cooldown alerts. The alert subject and the account_health decision now name account/model.
• /internal/accounts. It reports health: cooling while any configured model the account serves is cooling, and lists those models in a new cooling_models field. The control plane reads only health, so it is unaffected.
• Docs. architecture.md, providers.md, configuration.md, multi-instance.md and api.md are updated.

Why per-model keys rather than a last-account exemption

Never cooling the last healthy account would keep sending the failing model to an upstream that refuses it. It would also still bench the account for every model on multi-account pools. Keying by served model isolates the refusal where it happens, and it needs no special case for pool size.

The cost: an account-wide fault (a revoked key, an unreachable host) now has to trip each model's streak on its own, failure_threshold failures per model. Each of those failures still fails over to another account within its request, so the delay is bounded.

Hot-path cost

• Account selection now builds one short key String per candidate account, plus the Vec holding them.
• A successful call builds one key for record_success.
• Nothing else changes per request.

Tests

• handler a_model_cooling_down_leaves_its_only_account_serving_other_models. A single-account pool with two models, failure_threshold: 1, and an upstream that refuses one model. The test runs twice: once with a plain 503, and once with the refusal as an error event inside a 200 stream, reached through a variant so the served name differs from the requested one. The refused model cools down while the other is still served by the same account; the cooling model is then refused with no healthy upstream. With account-only keys, the test fails at the check that the account is still available for the other model.
• state a_cooldown_benches_the_account_for_one_model_only. select_healthy picks the account for one model and skips it for the cooling one.
• e2e account_cooldown_and_recovery. It now also asserts cooling_models.
• Updated tests. Existing health tests now use the per-model key.

Gates:

• cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings and cargo test --workspace are clean: 775 passed, 4 ignored.
• The GW_TEST_REDIS_URL / GW_TEST_PG_URL suites were not run. RedisHealth code is unchanged; only the key it receives changes.

Deploy note

Health keys change shape, so a cooldown active at deploy time is forgotten. Old Redis keys expire on their own TTL.