Repository navigation
perf: check an account's models concurrently in /internal/accounts - #175
Merged
Merged
Conversation
Per-model health turned the view's one lookup per account into one per served model, awaited in sequence: 25 accounts and 102 models took 17.7 ms against a local Redis, 7.3 ms with each account's lookups joined. Also rewraps the HealthStore paragraph in architecture.md.
CMGS
added a commit
that referenced
this pull request
Oct 27, 2026
…rves (#174) * fix: cool an account down per served model, not for every model it serves A model-level upstream refusal (OpenRouter's peak-load refusals for one model) tripped the account's cooldown, so a single-account pool answered no healthy upstream for every other model too. The failure streak and cooldown are now keyed by account and served model; /internal/accounts lists the cooling models. * test: cover an in-stream refusal on a variant-served model in the per-model cooldown test * review: keep the prompt_cache assignment where it was in SelectAccount * perf: check an account's models concurrently in /internal/accounts (#175) Per-model health turned the view's one lookup per account into one per served model, awaited in sequence: 25 accounts and 102 models took 17.7 ms against a local Redis, 7.3 ms with each account's lookups joined. Also rewraps the HealthStore paragraph in architecture.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Review follow-up to #174, targeting its branch. Merge it into
fix/model-cooldownbefore #174 is squash-merged.Change
With per-model health,
/internal/accountschecks one health key per (account, served model) instead of one per account, and it awaited each lookup in sequence. On a Redis-backed fleet, that turns one round trip per account into one per served model. This PR joins each account's lookups. The behavior and the response shape are unchanged.Measured against a local Redis 7, using the live-matrix config (25 accounts, 102 models), median of 5 calls with the health cache expired between calls:
/internal/accountsIt also rewraps the
HealthStoreparagraph indocs/architecture.md, where one line ran past 100 columns.Review of #174 (no other change needed)
HealthStorecall site uses the per-model key: selection, failover, success and failure recording, the realtime handshake and the admin view. Realtime only selects; it records no health.AccountPool::serves: protocol strings compare exactly (Protocol::from_wirehas no aliases), and the provider check uses the same config fields.Stringper candidate account and one per recorded success.join_allalready allocates. Avoiding the strings would mean a nested-map health store and a new trait shape, which is too much for tens of nanoseconds.note_failuremoves the key into the alert after the decision borrows it, so there is no clone.Gates
cargo fmt --checkandcargo clippy --workspace --all-targets -- -D warningsare clean.cargo test --workspacewith real Postgres 17 and Redis 7 gives 775 passed, 0 failed, 4 ignored. That run covers theRedisHealthsuites that fix: cool an account down per served model, not for every model it serves #174's description lists as not run, and fix: cool an account down per served model, not for every model it serves #174's head passes the same 775.