inference-proxy: back the credential cache with Redis so replicas share short-TTL bearer→virtual-key resolutions

[cubeorgdev]

What problem does this solve?

Every inference request through services/inference-proxy must turn the caller's gateway
bearer JWT into an upstream virtual key. JWT verification is local, but the
bearer → virtual key exchange is a real round trip: resolve_dynamic
(services/inference-proxy/src/main.rs:1158) POSTs to HARNESS_GATEWAY_RESOLVER_URL, and
control-api's resolve_gateway_key (services/control-api/src/lib.rs:4317) does a Postgres
query plus a KMS envelope decrypt before answering.

We already cache the result, but only in-process: CredentialCache (main.rs:50-123) is a
HashMap + LRU behind a Mutex, TTL HARNESS_PROXY_CACHE_TTL_SECONDS (default 60s).

The proxy runs 4 replicas by default and autoscales to 30
(deploy/helm/values.yaml:62-64). Because the cache is per-process:

• steady-state resolves scale as active_sessions × replicas / TTL — an active agent
  session makes far more than 30 requests per TTL window, so it realistically touches every
  replica and pays the resolve on each one;
• resolution_locks (main.rs:1790) single-flights only within a process, so one cold
  key can be resolved up to replicas times concurrently;
• the sharp edge is cold start. Every rolling deploy and every scale-out event drops all
  L1 caches at once, and the resulting burst hits Postgres and KMS, behind the bounded
  state.gateway_kms_permits pool. Raising the TTL makes this strictly worse — larger
  working set, same cliff.

Redis is already provisioned and reachable for exactly this: ElastiCache in
deploy/tofu/aws-ecs/redis.tf (described "Blue session and invalidation cache",
enable_redis defaults to true), HARNESS_REDIS_URL is already injected into the
inference-proxy task definition (deploy/tofu/aws-ecs/ecs.tf:133), and the Helm
NetworkPolicy already has Redis egress knobs (deploy/helm/values.yaml:265-266).
Nothing in the Rust code reads it — deploy/tofu/aws/main.tf:142 even says so.

Proposed solution

Two independent parts. Part 1 is a config change that can ship immediately; part 2 is the
structural fix.

Part 1 — raise the default L1 TTL

HARNESS_PROXY_CACHE_TTL_SECONDS 60 → 600.

This is safe because the TTL is a backstop, not the safety mechanism. Revocation is
push-based over a durable, ordered, replayable log (see below), expiry is re-checked on
every cache read, and a broken invalidation stream makes cached_mapping bypass the cache
entirely. Cuts steady-state resolver/DB/KMS load ~10x on its own.

Part 2 — Redis as an optional shared L2

• L1: today's CredentialCache, unchanged.
• L2: Redis, GET on L1 miss, SETEX on resolve; an L2 hit populates L1.
• Lookup order: L1 → L2 → in-process single-flight → resolve_dynamic.

Key and value

• Keep the session-scoped key {blue_oauth_session_id}:{user_id}
  (GatewayIdentity::cache_key, main.rs:278) — do not key on the token or its hash;
  invalidation targets user/session, and every JWT minted for one session must share one
  entry. Namespace and hash it, e.g. blue:proxy:cred:v1:{sha256(oauth_session_id:user_id)},
  so session ids never enter the keyspace. Bump v1 on any value-schema change.
• Value is Mapping (main.rs:29-43), which needs Serialize/Deserialize; serialize the
  three OffsetDateTime fields as RFC3339, mirroring ResolveResponse (main.rs:1013).
• User-scoped invalidation (oauth_session_id: None) must reach every session of a user, so
  maintain an index set blue:proxy:cred-idx:v1:{sha256(user_id)} → member keys, written in
  the same pipeline and given the same TTL. SCAN is not acceptable on that path.

TTL

• HARNESS_PROXY_REDIS_TTL_SECONDS, default 600 to match part 1.
• Clamp to min(configured, credential_expires_at - now, gateway_session_expires_at - now).
• A Redis TTL is not sufficient alone. CredentialCache::get (main.rs:67) revalidates
  credential_expires_at and gateway_session_expires_at on every read, and
  proxy_auth_guard (main.rs:1849) rejects JWTs predating session_not_before. Every L2
  hit must re-run all three checks before it is trusted.

Invalidation

gateway_cache_events is a durable ordered log; the SSE handler
(services/control-api/src/lib.rs:4445) streams it by monotonic id, honors
last-event-id, and emits {"resync":true} only on a genuine replay gap (replay_gap,
:4457). Therefore:

• while invalidation_healthy == false, bypass and do not write L2, exactly as L1 does;
• on each SSE credential event, DEL the L2 key(s) alongside the L1 entry — every replica
  sees the event and DEL is idempotent, so no coordination is needed;
• on reconnect, clear L1 as today but do not flush L2 — replay covers the gap, so a
  rolling deploy must not become a global flush;
• flush L2 globally only on an explicit resync, via a generation counter in the key
  prefix (INCR blue:proxy:cred-gen), which is O(1) and needs no key enumeration.

Security

Mapping.upstream_credential is a live upstream virtual key, so this moves plaintext
secrets out of the process:

• require rediss:// outside local dev, enforced the way validate_urls
  (main.rs:753-799) enforces schemes per InternalTransportMode;
• ElastiCache already provides TLS in transit, an auth token, and KMS at rest
  (deploy/tofu/aws-ecs/redis.tf);
• decide whether to additionally encrypt the value with a proxy-held key before SETEX, so
  a Redis compromise alone yields no usable virtual keys. The TTL bounds exposure either way.

Optionality and failure behaviour (hard requirement)

AGENTS.md:104: the governance-only path takes no new hard dependency on Redis. Redis will
be standard in enterprise deploys, but the code must not require it:

• with HARNESS_PROXY_REDIS_URL unset, behaviour is exactly what it is today;
• Redis errors, timeouts, and deserialization failures fail open to L1 + resolver and
  never surface as a 5xx;
• aggressive per-op timeout (HARNESS_PROXY_REDIS_TIMEOUT_MS, default ~50ms) plus a breaker
  that stops calling Redis after sustained failure and retries on a backoff;
• /ready must not start depending on Redis.

Config (bare std::env::var + env_u64/env_usize helpers at main.rs:284-293, read
in main() — the proxy has no config struct):

Var	Default	Notes
HARNESS_PROXY_REDIS_URL	unset	unset ⇒ L2 disabled. Decide whether to just consume the already-injected HARNESS_REDIS_URL
HARNESS_PROXY_REDIS_TTL_SECONDS	600	clamped by credential/session expiry
HARNESS_PROXY_REDIS_TIMEOUT_MS	50	per-op
HARNESS_PROXY_REDIS_KEY_PREFIX	blue:proxy	shared-cluster safety

Metrics — extend Metrics (main.rs:126) and /metrics (main.rs:826-855) with
gateway_proxy_l2_cache_hits_total, ..._misses_total, ..._errors_total,
..._writes_total, and an L2 latency histogram. Keep existing counters unchanged.

Follow-up, not required here: a Redis SET NX lease to single-flight resolves across
replicas, replacing per-process resolution_locks on cold keys.

Deploy / docs surfaces

• deploy/tofu/aws-ecs/ecs.tf:133 — already injects HARNESS_REDIS_URL into the proxy.
• deploy/tofu/aws/main.tf:142-144 — comment claiming Blue never reads it becomes wrong.
• deploy/helm/templates/inference-proxy-deployment.yaml:40-78 — add the env; secret-sourced
  values already flow via envFrom. Populate networkPolicy.redisCidrs
  (values.yaml:265), [] today, or egress is denied.
• deploy/docker-compose.yml:197 — the redis service is behind the legacy-gateway
  profile and wired only to LiteLLM (:184); needs to be available to the default profile.
• tests/e2e/docker-compose.yml:160-182 — proxy env block.
• Docs: apps/docs/next/concepts/gateway-mode.mdx:100-135 env table,
  apps/docs/next/admin/gateway-access.mdx, apps/docs/next/deployment/production.mdx.

Tests

• Unit, in the inline #[cfg(test)] mod tests (main.rs:2139+) via the test_state builder
  (:2291): L2 hit skips the resolver; L2 miss resolves and writes; expired
  credential_expires_at / gateway_session_expires_at in an L2 payload is rejected on
  read; session_not_before still rejects a stale JWT on an L2 hit; an SSE event deletes the
  L2 key; reconnect does not flush L2 but resync does; invalidation_healthy == false
  bypasses and does not write; Redis down ⇒ requests still succeed via the resolver.
• E2E, extending tests/e2e/specs/gateway-m2m.spec.ts (asserts on /metrics; compose
  already runs HARNESS_PROXY_CACHE_TTL_SECONDS: "1"): run two proxy containers against one
  Redis and assert the second serves an L2 hit with no additional resolver call.

Acceptance criteria

• With Redis configured and L1 cold, a repeat request for the same session on a
  different replica is an L2 hit and issues no resolver call.
• Restarting all proxy replicas does not produce a resolver/KMS burst — L2 survives.
• A reconnect with valid last-event-id does not flush L2; a resync does.
• gateway_proxy_l2_cache_hits_total / ..._misses_total / ..._errors_total on
  /metrics.
• Revoking a session stops serving from L2 within the same bound as L1 today.
• Killing Redis mid-load causes zero request failures — only a rise in resolver calls.
• HARNESS_PROXY_REDIS_URL unset reproduces current behaviour; governance-only mode
  still has no Redis dependency.
• Non-rediss:// URLs are rejected outside local dev.

Before submitting

• I searched existing issues and this isn't a duplicate.
• I confirmed this doesn't already exist elsewhere in Blue.