Skip to content

Add the usage-truth ledger (PRX-01–06) - #3

Merged
hmziqrs merged 10 commits into
freeoxide:masterfrom
hmziqagent:prx/usage-ledger
Oct 27, 2026
Merged

hmziqrs merged 10 commits into
freeoxide:masterfrom
hmziqagent:prx/usage-ledger

Conversation

@hmziqagent

@hmziqagent hmziqagent commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Implements the PRX usage-truth ledger plan: the proxy becomes the source of truth for per-request usage — every request persisted to SQLite with token counts, cost, and caller attribution, queryable through an authenticated, read-only admin API. Also adds named client keys, per-provider egress routing for geo-restricted upstreams, and a cargo-dist release pipeline.

What's in it

  • PRX-01 — client identity: [auth.clients] with argon2-hashed secrets, Authorization: Bearer on proxy routes, loopback auth = "none" exemption (non-loopback binds refuse to start without client keys), RequestReceived.caller attribution, rate limiting keyed by key id when authenticated (499b22a, fix bf5c53f).
  • PRX-02 — SQLite ledger: rusqlite (bundled) EventBus backend; one requests table keyed by request_id with usage/cost/latency/body_hash columns; indexes on (timestamp), (provider, timestamp), (caller, timestamp); dedicated writer task behind a bounded channel (a full channel drops and counts, never backpressures a response); WAL; configurable retention pruning; [storage] config path; NoopBus default preserved when [storage] is absent (5e4126a).
  • PRX-03 — /usage admin API: fills the llm-proxy-api placeholder with GET /usage/summary, GET /usage/requests, GET /usage/keys — strictly read-only, no request bodies ever. A separate [auth.admin] token guards every route; served from a dedicated loopback listener ([server.admin_bind]) by default, with admin_on_main_listener = true as an explicit, loudly-flagged opt-in (045e2cb).
  • PRX-04 — per-provider egress: [provider.egress] proxy = "socks5h://host:port" (or http/https) builds a per-provider reqwest client; scheme allowlist validated at startup; no egress configured means today's exact single-client behavior. Bind-out/source-address selection stays out of scope by plan (3ecb2f3).
  • PRX-05 — release pipeline: cargo-dist config + generated release workflow producing musl x86_64/aarch64 binaries and the shell installer, alongside (not modifying) the existing CI (0c2614f). On PRs the release workflow runs its plan job only; the full artifact pipeline triggers on version tags.
  • PRX-06 — secret-abuse test family: tests proving client and admin tokens never appear in events, tracing output, error responses, or the raw SQLite file bytes beyond their hashed-at-rest rows; /usage responses carry body_hash at most; both mounts refuse unauthenticated and wrong-token calls; caller attribution never leaks one caller's key id to another (a551619).
  • CI tooling: the unused-deps job pins cargo-shear --version 1.13.0 because cargo-shear ≥ 1.14 pulls ra_ap_* crates requiring rustc 1.98, above the repo's pinned 1.96.1; bump both together (e90fb06).

Verification

  • Full workspace gates (cargo fmt --all --check, cargo clippy --workspace --all-targets --all-features --locked -- -D warnings, CI's lint-inheritance check, cargo test --workspace --all-features, cargo build --locked) ran after every implementation and fix round, driven from exit codes — never an agent's claim.
  • Every committed area was audited by a fresh, read-only judge; the plan's three exit gates were re-verified independently with file:line evidence.
  • A final independent verification pass re-ran all gates (including the CI-only locked build) and checked 38 delivery claims with seven fresh read-only verifiers: 35 verified in code, 2 matching the pre-declared known-open list, 1 wording correction (the release workflow's PR trigger above), and 18 new review-grade notes — none a correctness or security failure. The run's audit log is available on request; it is kept out of the branch by design.
  • CI on this PR: quality and unused-deps both green.

Known open items (deferred deliberately — reviewer's call)

  • passthrough_auth × client-keys: for passthrough providers the inbound client key is forwarded as the upstream credential, and config validation does not reject the combination (crates/llm-proxy-server/src/routes/core_pipeline.rs:452, crates/llm-proxy-core/src/provider_config.rs:1466). Needs a decision — reject at startup, or a distinct header. Until it lands, the abuse family proves token custody over events/logs/responses/SQLite but not upstream transmission.
  • Release pipeline cannot ship artifacts yet: reqwest still pulls default-tls/openssl-sys, so the musl jobs fail at the openssl-sys build script (crates/llm-proxy-provider/Cargo.toml:23, reproduced with cargo check --target x86_64-unknown-linux-musl → exit 101). Fix is default-features = false + rustls — a TLS backend swap, left for explicit review. Do not push a release tag before this lands.
  • Admin token has no failure budget (crates/llm-proxy-api/src/auth.rs:104) — online-guessable under the admin_on_main_listener posture (where /usage also sits outside the per-IP rate limiter); client auth already has such a gate.
  • model_upstream stays NULL for streaming requests: the stream path emits ctx.core.model (core_pipeline.rs:2043; the resolved value exists unused in ctx.stream_metrics.upstream_model), and /usage group_by=model groups on the requested model only.
  • Smaller deferred items, each with evidence in the run's audit log (available on request): retention starvation under channel saturation and unindexed retention scans (sqlite.rs:504); no TimeoutLayer on /usage (api/routes.rs:64); double rate_limited count on gated rejections (routes/mod.rs:341); ~0 latency_ms on auth rejections (routes/mod.rs:362); two-snapshot /usage/summary reads (api/reader.rs:363); egress proxy URL userinfo echoed in startup error text (provider_config.rs:1044); model-discovery traffic bypassing the per-provider egress client (catalog_service.rs:137).

Reviewer notes from the final verification pass

  • apps/llm-proxy/src/platform.rs carries three pre-existing-test repairs that rode along with PRX-01 (macOS #[cfg] gating of a config-dir import, a strengthened shell-metacharacter escape assertion, and a previously-missing insta snapshot) — forced by tests already failing on master before this branch; documented in the run's audit log (available on request).
  • The z-proflow run log was removed from the branch (b2537ee); .z-proflow/ stays gitignored and nothing under it ships.
  • The ledger indexes currently serve retention DELETEs only in part: every /usage query filters/sorts on COALESCE(received_at, completed_at), which the received_at-only indexes cannot serve — read-path scans are the same pattern as the deferred retention item.

Not covered

  • musl artifacts are first exercised by CI on a release tag (config-only locally, and blocked on the rustls swap above); cargo shear runs in CI.
  • quota-stats (CodexBar-compatible) aggregate: optional per plan, skipped.
  • Bind-out/source-address selection: out of scope per plan (exit geography comes from the tunnel endpoint).

Introduce [auth] config (mode = "none" | "client-keys") with argon2id
password-hash verifiers for named client keys. A require_client_auth
layer on the proxy routes verifies Bearer tokens (digest-cached,
spawn_blocking, failed attempts charge the peer's IP bucket); key ids
are stamped onto RequestReceived events as `caller` ("anonymous" under
the loopback no-auth exemption) and key the rate limiter for
authenticated callers. Non-loopback binds without client keys refuse
to start, and invalid verifiers fail closed.

Also clears the pre-existing clippy/insta baseline: cfg-gates the
macOS-only desktop-entry helpers, fixes the contradictory
shell-metacharacter escape test and commits its missing snapshot, and
adds narrowly-scoped #[expect(clippy::too_many_lines)] on the
wire-translation functions. Ignores .z-proflow/ and tracks its
STATE.md run journal.
require_client_auth's doc claimed that once a peer's IP bucket was
empty further failures skipped the CPU-costly argon2 verification, but
the code ran the full verification on every attempt and the limiter
only charged the bucket afterwards — the claimed CPU-exhaustion bound
did not exist. Implement the bound instead of weakening the comment:
add RateLimiter::would_allow, a non-consuming peek at the peer's
refilled balance, and consult it before authenticate() so a peer with
an exhausted failure budget gets the uniform 401 without paying argon2.
Successful authentications are never charged (they are limited under
the key id in prepare_request); failures still drain the bucket via
is_allowed so the gate trips. Doc comment now states the exact flow,
including that rate_limit_rpm = 0 means no bound applies.
Implement the EventBus trait over SQLite (rusqlite, bundled): a
dedicated writer task behind a bounded channel persists every request
event to a requests table — token usage, cost, caller, latency, stop
reason, error kind, body hash — with indexes on (timestamp),
(provider, timestamp), and (caller, timestamp). A full channel drops
the event and increments a counter rather than backpressuring
responses; the database runs in WAL mode; the writer drains and exits
via the new EventBus::close after the server's graceful drain.

New [storage] config (path, max_age, max_rows, prune_interval) builds
the bus in load_toml_state; absent config keeps the NoopBus default.
Retention pruning runs on a timer against the age and row caps.
Fill the llm-proxy-api placeholder with the read-only query surface
over the SQLite ledger: GET /usage/summary (window plus group_by
provider|model|caller), GET /usage/requests (keyset-paginated raw
rows, never request bodies), and GET /usage/keys (per-caller counts
and last-seen). Queries run on read-only connections with bound
parameters, and costs sum in rust_decimal so SQLite never coerces
them to float.

Admin auth is a separate [auth.admin] token stored as SHA-256 and
compared in constant time; every rejection is a uniform 401 that
never echoes the token. The routes mount on a second loopback
listener ([server].admin_bind, default main port + 1) by default;
admin_on_main_listener = true is the named, loudly-warned opt-in
posture. Config validation rejects admin without [storage], a bind
shared with the main listener, and empty tokens.
Honor [provider.egress] proxy in provider TOML: the transport builds
a per-provider reqwest client (.proxy with socks5h/socks5/http/https,
same pool and timeout settings as the shared client), and both
upstream send sites in the core pipeline pick it via
upstream_client_for. Providers without egress keep today's exact
shared client. Config validation rejects empty, scheme-less, and
non-allowlisted URLs at load time, and the transport re-validates the
scheme before handing the URL to reqwest (which silently accepts
scheme-less and ftp URLs). The URL's userinfo is redacted in Debug
output and never logged, and an unusable proxy URL is a startup error
before the listener binds — never a silent direct fallback. reqwest
gains the socks feature (handshake in-tree, no new crates).
Configure cargo-dist 0.32.0 (dist-workspace.toml): shell installer,
x86_64- and aarch64-unknown-linux-musl targets, CARGO_HOME install
path, GitHub hosting, no updater. The release workflow is generated
verbatim by dist generate --mode ci — plan, build, host, and announce
jobs on version-tag pushes, with dist plan exercised on pull requests;
ci.yml is untouched. Add the [profile.dist] cargo requires (dist
passes --profile dist unconditionally) inheriting the release
posture. Pin CC_aarch64_unknown_linux_musl to musl-gcc in
.cargo/config.toml: dist builds aarch64-musl on arm64 runners with
only musl-tools, whose sole compiler is musl-gcc, while the cc crate
probes only aarch64-linux-musl-gcc for that target.
Nine tests in three new binaries, all driving the real surfaces
(build_router, admin_router, the client-auth layer, SqliteEventBus,
the /usage endpoints) with every negative assertion paired with a
positive control, so a passing scan proves it read real content.
They pin: client and admin tokens, PHC verifiers, and provider keys
never appear in events, captured tracing, error responses, or the
raw db/-wal/-shm file bytes; /usage rows carry the sha256 body
digest but never body content even when the mock upstream reflects
the request; the admin surface answers missing, wrong, and
non-admin credentials with an identical uniform 401 on both mounts;
one caller's key id is never observable by another; and anonymous
attribution survives the loopback exemption without swallowing or
leaking a stray bearer token. tracing-subscriber joins the server
and api dev-dependencies for the log-capture assertions.
@hmziqrs
hmziqrs merged commit 754e2a8 into freeoxide:master Oct 27, 2026
7 checks passed
@hmziqagent
hmziqagent deleted the prx/usage-ledger branch October 27, 2026 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants