Repository navigation
feat(audit): ingest rate-limit circuit breaker (#831) - #98
Merged
Merged
Conversation
Phase 3 of campus#538. The breaker is the producer side of the audit ingest rate limit (campus PR #835): audit 429s carry Retry-After and the tripped bucket key; this module records them and exposes a gate check for producers. - campus_python.audit.ratelimit: thread-safe per-process breaker — 429 trips the bucket (floor 30s, x2 backoff on consecutive trips, cap 5min), 2xx clears expired trips (two-stage retraction: cooldown expiry opens the gate, first 2xx clears), availability errors (timeouts/refused/5xx) neither set nor clear. - apikey= trips are remembered as the producer's own fallback bucket so identity-less requests can be gated coarsely. - Traces.ingest feeds the module breaker before raising; observation is guarded and can never break ingestion. - check_request(): per-identity gate key when identity is resolvable, degrading to the learned fallback bucket or wildcard.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Producer side of the audit ingest rate limit — the circuit breaker half of campus#831 (Phase 3 of campus#538). The audit-side enforcement landed in nyjc-computing/campus#835:
POST /traces/429s carry aRetry-Afterheader and the tripped bucket key inerror.details.bucket. Design (ratified): Phase 3 design on campus#538.What
campus_python.audit.ratelimit— a thread-safe, per-process circuit breaker:observe(status, headers, body): a 429 trips the bucket named in the body (floor 30s, ×2 backoff on consecutive trips, cap 5 min; audit'sRetry-Afterhonored up to the cap). A 2xx clears trips whose cooldown has expired — the ratified two-stage retraction: cooldown expiry opens the gate (verifying) without clearing, and the first 2xx ingest fully clears. Availability errors (timeouts, connection refused, 5xx) neither set nor clear — an audit outage must not take producers down.apikey=trips are remembered as this producer's own fallback bucket (an apikey-bucket 429 seen by this process's client is necessarily its own), so identity-less requests can be gated.check_request(client_id, user_id): per-identity gate key (same priority encoding as the server: pair → user → client), degrading to the learned fallback bucket or the wildcard when identity isn't resolvable at request entry.Traces.ingestfeeds the breaker before raising, with the observation wrapped incontextlib.suppress— a breaker failure can never break ingestion.Consumers
The follow-on campus PR wires this into the producer middleware (
campus/audit/middleware): a flag-gated (AUDIT_TRACING_FAIL_CLOSED)before_requestgate that 503s matching requests withRetry-Afterwhile a bucket is tripped. campus-classroom inherits via its dependency on the campus package. This PR is self-contained (no behavior change for existing callers — the breaker only records; nothing gates until a consumer opts in).Testing
tests/unit/test_ratelimit.py(19 tests): trip/check/backoff/cap math, two-stage retraction, availability neutrality, fallback learning,check_requestkey routing, and theTraces.ingesthook (429 trips + raises, 201 doesn't trip).ruffclean on the touched files (remaining repo baseline findings are pre-existing onmain).