Skip to content

fix(deploy): mount the gateway signing key into the worker - #127

Merged
tomi-stipancik merged 3 commits into
mainfrom
fix/worker-gateway-jwt-mount
Sep 16, 2026
Merged

tomi-stipancik merged 3 commits into
mainfrom
fix/worker-gateway-jwt-mount

Conversation

@cubeorgdev

@cubeorgdev cubeorgdev Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Fixes #126

The worker crash-loops in gateway mode. Found it on the EKS cluster built from the Helm guide — 35 restarts, 0/1 available:

Error: reading gateway JWT private key: No such file or directory (os error 2)

The worker runs the same binary and the same blue.yaml as the Control API. That config declares gateway.inference_jwt.private_key_file, and load_gateway_jwt_key_ring (lib.rs:1460) runs on every startup where gateway.type is set. The chart mounted the Secret into the API only, so the worker died on the missing file — and cleanup_gateway_request_logs and process_gateway_revocations died with it.

It can't just skip the gateway block: process_gateway_revocations (lib.rs:8528) drives the provisioner, so it needs the envelope. It only doesn't need this one key — state.gateway_jwt has two consumers and both are API request handlers (gateway_auth.rs:123 mints tokens, :158 serves /gateway/jwks).

What changed

  • worker-deployment.yaml mounts the gateway-jwt Secret the same way deployment.yaml:143,163 does, gated on blue.enableInferenceProxy. Its volumes:/volumeMounts: were wrapped in with .Values.blue.config.existingConfigMap, so that becomes the if or pattern the API template already uses.
  • HARNESS_GATEWAY_JWT_PRIVATE_KEY_FILE is set on the worker too. The API sets it, so without it a config file pointing somewhere else would break the worker again while the API kept working.
  • verify-deployment.sh renders the worker on its own and asserts the mount. The existing greps are file-wide and pass on the API's copy.
  • One values.yaml comment: the Secret reaches two pods now, not one.

Verification

scripts/verify-deployment.sh passes. Reverting only the template makes the new assertion fail, so the guard works. Rendered all four permutations of existingConfigMap × enableInferenceProxy plus includePreviousKey: the two non-gateway ones are byte-identical to main, and the gateway ones match the API's shape. helm lint clean.

Not tested against a live cluster — the deployed one is mid-investigation and I didn't want to change it.

Follow-ups, not in here

  • The real fix is splitting the envelope so only the signing process requires key material; that's a code change plus a release, so it doesn't help anyone on 0.1.0. This does.
  • docs(docs): quickstart as a TL;DR and Helm guide as the verified EKS walkthrough #125 documents the flow that trips this. Land this with or before it — step 10's "Only the API reads it" is the assumption that produced the bug.
  • Unrelated, found alongside: the HPAs are inert on any cluster this guide builds. EKS Auto Mode ships no metrics-server and the guide never installs one. Worth its own issue.

🤖 Generated with Claude Code

The worker runs the same binary and the same blue.yaml as the Control API,
so it loads the gateway envelope at startup and that envelope declares the
signing key. The chart only mounted the Secret into the API, so the worker
exited on a missing file and crash-looped, taking the gateway log cleanup
and revocation jobs with it.

It never signs anything; the mount just satisfies the load. Pinning the
path by env keeps the two pods from disagreeing if a config file moves it.

Fixes #126

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@cubeorgdev

cubeorgdev Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

This pull request was created from a Blocks session.

View in dashboard | View on desktop

@AlejandroEsquivel
AlejandroEsquivel marked this pull request as ready for review September 16, 2026 00:55
@AlejandroEsquivel
AlejandroEsquivel requested a review from a team as a code owner September 16, 2026 00:55
@tomi-stipancik
tomi-stipancik enabled auto-merge (squash) September 16, 2026 05:22
@tomi-stipancik
tomi-stipancik merged commit 1fbeb04 into main Sep 16, 2026
28 checks passed
@tomi-stipancik
tomi-stipancik deleted the fix/worker-gateway-jwt-mount branch September 16, 2026 05:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Worker crash-loops in gateway mode: signing key mounted into the API only

2 participants