Skip to content

fix(pool): renew and persist existing sandbox deadlines - #2

Merged
neotype merged 1 commit into
production/ce85c8bfrom
fix/sandbox-renewal
Sep 17, 2026
Merged

neotype merged 1 commit into
production/ce85c8bfrom
fix/sandbox-renewal

Conversation

@neotype

@neotype neotype commented Sep 17, 2026

Copy link
Copy Markdown
Owner

Bounded production backport, based on the currently deployed fdf73c4 tree (production branch merge has the identical tree). Add sandbox-token POST /renew, serialize with reap/release, reject expired/archived claims, persist before acknowledgement and roll back on persistence errors. No upgrade to main or host network schema. Validation: go test -race ./... and go vet ./... pass; golangci-lint --new-from-rev=fdf73c4 reports 0 new issues. Full baseline lint has 3 existing staticcheck findings (two netfilter comparisons, deprecated ReverseProxy.Director). Added manager persistence/auth/expiry/concurrency and 10 HTTP cases. Production deployment and isolated cross-TTL probe pending.

@neotype
neotype merged commit 91bf867 into production/ce85c8b Sep 17, 2026
@neotype

neotype commented Sep 17, 2026

Copy link
Copy Markdown
Owner Author

本地构建与生产发布(2026-09-17)

本地验证

pnpm test with disposable PostgreSQL: 697 passed, 10 skipped (browser/guest environment-dependent tests). pnpm test:quality-gate: 74/74. pnpm typecheck, pnpm lint, pnpm fmt:check, pnpm build, git diff --check: pass.

Cocoon GOWORK=off go test -race ./... and go vet ./...: pass. golangci-lint run --new-from-rev=fdf73c4 ./...: 0 new issues; full baseline lint still has 3 existing staticcheck findings, not claimed green.

Five Nest Kustomize roots render; kubeconform reports 0 invalid/errors (encrypted Secrets and missing CRD schemas explicitly skipped). Local pre-commit and shellcheck pass.

Remote Pigeon/Nest CI jobs did not start: check annotations report account payment/spending-limit failure. Remote CI is not claimed passing.

发布与探活

Local Linux/amd64 Dockerfile builds pushed to GHCR, deployed immutable source tags; Pod imageIDs match:

  • Agent sha256:7815e7fc769e7486ede1d82feabcb8681bb69a8029e6171628881af50f6baed6
  • sandbox-service sha256:ef844ef33a883741654bbe9ff6e2c63c93f6a4b0f0d790527077a4194dd294e2

Deployed dist/index.js SHA256 matches local builds:

  • Agent 2b930a7a875d93f18dcb40ea487b650cf4bd524133c0c79dd39626eea83087de
  • sandbox-service ad05e929747ef035dfb090313bda1042c3cc3e8482445857e650f7f90eb2791e

Host sandboxd reports the production backport revision; static binary SHA256 91a1aba5e3b11f561363f8526a0a8855c199e6ed126f0370f82066a00ff9908d, health passes. Previous binary backed up at /usr/local/bin/sandboxd.before-renew-20260917. Before deployment: Agent active turns=0, Gateway lane depth=0, sandboxd claimed=0, sandbox-service leases=0.

Both new Pods Ready with 0 restarts. Agent/sandbox health endpoints each 3x HTTP200. Agent synthetic non-owner /turn: HTTP200, one final, one done, zero errors; no WeChat delivery or external modification. Authenticated /renew without a VM: HTTP200, renewed:false, renewAfterMs:600000. Backend missing bearer returns401; node bearer cannot renew sandbox (404). All six Flux Kustomizations Ready at Nest revision above; no Pigeon Warning events at sampling.

范围

Only active-turn TTL loss is fixed. Long idle retention and sandbox-service restart lease recovery remain deferred. No recovery/publication of the already-lost Langfuse commit is included. The physical short-TTL probe below is not a 30-minute real-clock soak; 30-minute/40-minute behavior is covered by fake-clock regressions separately.

Physical cross-TTL probe

Isolated deployed SandboxManager, TTL=90s, real 100-second guest command, 30-second renewal heartbeat (3 successful renewals). The probe crosses the original backend deadline and checks the same VM, file and HEAD, then releases only its synthetic fixture. This exercises sandbox-service/Cocoon renewal physically; the Agent periodic failure/continuation path is separately covered by runtime regressions.

{"phase": "created", "id": "sb_a2b7479a737e54bd", "head": "ffecb7e0e844bd6f36a194baf27296222ad95322", "originalDeadline": "2026-09-17T10:19:53.798884954+08:00"}
{"phase": "verified", "sameVm": true, "id": "sb_a2b7479a737e54bd", "head": "ffecb7e0e844bd6f36a194baf27296222ad95322", "filePreserved": true, "elapsedMs": 100317, "heartbeats": 3, "originalDeadline": "2026-09-17T10:19:53.798884954+08:00", "renewedDeadline": "2026-09-17T10:21:34.142765748+08:00", "verifiedAt": "2026-09-17T02:20:04.087Z"}
{"phase": "cleanup", "released": true}

After cleanup: backend claimed=0; deployed Agent active turns=0, renewal failure events=0. Only the dedicated rollout GPU wake lease is released; other leases are left untouched.

Live auth against a known synthetic sandbox: node token renew=404, own sandbox token renew=200, release=204; fixtures cleaned up and claimed=0. The first auth smoke incorrectly expected release=200; corrected the probe to the existing 204 contract and reran successfully. No production application change was needed for that probe assertion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant