Skip to content

fix: Add one-shot retained-image warm recovery - #979

Merged
Pigbibi merged 5 commits into
mainfrom
fix/bounded-warm-recovery
Nov 5, 2026
Merged

Pigbibi merged 5 commits into
mainfrom
fix/bounded-warm-recovery

Conversation

@Pigbibi

@Pigbibi Pigbibi commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

AI assistance

Codex (GPT-6), with official GitHub Codex exact-head review requested separately.

What changed

Add a separate, one-shot recovery operator for an explicitly approved warm revision at minimum one and 100% traffic. The ordinary protected build workflow and its minimum-zero baseline remain unchanged.

The operator reuses the card-bound immutable application digest, creates exactly one minimum-zero replacement, activates the exact 99/1 split, requires two fresh strict startup observations five seconds apart within 300 seconds, and then promotes replacement to 100%. Failure or an unknown activation/promotion outcome permits one distinct warm-100 compensation. Resource-version preconditions, frozen identity/spec/digest checks, request collision refusal, expiry, and private attempt state prevent unsafe continuation and write replay. Compensation binds outstanding writes to their complete desired spec including traffic; an ambiguous activation requires a confirmed warm-100 CAS update with an advanced resource version before restoration is reported.

Independent v2 service/template minimum and warm provisioning readbacks accompany the operation. The existing rollout document describes the private card contract, protected execution boundary, failure handling and later owner-card bounds. A manual-only recovery workflow uses the existing protected environment and workload identity, verifies a private GCS card against the owner-approved checksum/operator/environment, refuses workflow reruns, and invokes the operator. No automatic trigger or public artifact is added.

How verified

  • 48 offline recovery regressions passed; 111 recovery/shared waiter/tag tests passed on final head 5c9366a. Three adverse persistence regressions first reproduced the official P1 finding and now pass; failure-state recording cannot suppress compensation.
  • Ruff, Actionlint, recovery workflow YAML/card-fetch checks, protected deployment shell contract, Cloud Run startup validation (11 routes), QPK pin/lock consistency, documentation check and Gitleaks directory plus exact commit-range scans passed.
  • Exact-head Linux CI passed: 6630 tests, 6 skips, 368 subtests; the isolated TQQQ job also passed. The preceding local full candidate passed 6624 tests and 368 subtests, with 6 skips and two macOS-only baseline cases deselected after reproducing both at the exact unchanged main base. Final deltas are covered by the current focused tests and full Linux CI.
  • Official current-head Codex review completed with has_any_findings=false and has_unresolved_findings=false. All three earlier findings were reproduced, fixed and resolved; there are no unresolved review threads. Native merge checks and approval-label provenance remain independently enforced.
  • Documentation check retains the existing README outline advisory.

Why

The ordinary workflow always rebuilds and rejects a retained warm minimum-one serving baseline. Recovery needs a separately reviewed entry that preserves the immutable image and explicitly implements bounded activation plus failure restoration.

Risk

High / R2: source-only deployment operator and its synthetic tests/documentation. Authority covers engineering and normal gated merge only. Actual use requires a later explicit owner card in the existing protected execution environment. No deployment, traffic update, Secret/IAM/Scheduler change, environment sync, risk/execution switch, order, deletion, peer operation, business HTTP invocation or recovery retry is performed by this PR.

Co-Authored-By: Codex <noreply@openai.com>

Pigbibi commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Please perform a deep review of the complete R2 diff at head f8d83da against base 8aa8e8e. Focus on Cloud Run v1 resourceVersion concurrency, retained image/source/UID/spec/generation binding, exact minimum-one baseline exception, one new minimum-zero replacement, fresh strict health twice five seconds apart within 300 seconds, expiry, readback-only handling of unknown writes, one distinct warm-100 compensation, request/restart refusal, redaction and independent v2 minimum/provisioning evidence. Verify the actual v1/v2 API contracts and all failure branches. This is source-only engineering; do not deploy, invoke cloud/business endpoints, or operate any peer. Please bind the provider machine-readable review state to this exact head.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-05T23:21:13.780958Z 5c9366a Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Co-Authored-By: Codex <noreply@openai.com>
@Pigbibi
Pigbibi marked this pull request as ready for review November 5, 2026 23:02

Pigbibi commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

Final acceptance candidate: head 4e69584 against unchanged base 8aa8e8e. Marking the draft ready requests the official current-head review. The final commit additionally requires template-spec equality with the verified warm revision, rejects a late control-plane convergence read, and normalizes unexpected provisioning status to Unknown. All 36 recovery tests pass; full Linux CI remains unmodified. The earlier local full run had 6613 passes and only two macOS environment failures, both now reproduced at the exact unchanged main base. The final local suite is running with those two cases deselected, not weakened. Please assess the complete final diff under the R2 failure/concurrency/redaction scope already requested. No operational action is authorized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4e695844b3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/cloud_run_warm_recovery.py Resolved
Co-Authored-By: Codex <noreply@openai.com>

Pigbibi commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Please deeply review the complete final R2 candidate at head bd51623 against base 8aa8e8e. The prior P1 is reproduced and fixed with three adverse persistence regressions; failed persistence no longer prevents the one authorized warm-100 compensation or removes the attempt claim. Also review the new manual-only recovery workflow: existing protected environment/WIF, masked private GCS locator plus owner checksum, exact card bytes/operator/environment, frozen dependencies, no rebuild/image push, no automatic triggers or reruns, no public artifacts. Full scope remains retained warm/min1 -> one pinned-digest min0 replacement -> exact 99/1 -> two strict checks five seconds apart within 300 seconds -> replacement100; fail-stop plus one guarded compensation. Check actual API contracts, concurrency/expiry/unknown outcomes and privacy. The focused 107 tests, Ruff, Actionlint, YAML parser, documentation and exact-range Gitleaks pass. Final full local and Linux suites are running. No deployment, traffic, Secret/IAM, or peer action is authorized by this review.

Co-Authored-By: Codex <noreply@openai.com>

Pigbibi commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Final exact-head R2 acceptance candidate: 8cf607a, unchanged base 8aa8e8e. In addition to the reproduced/fixed P1 and protected manual workflow, the final small commit bounds failure compensation to the last confirmed service generation or its one possible ambiguous-write successor. An injected unrelated generation advance now stops without any compensation write. All 45 recovery regressions pass; the preceding full Linux CI passed, and the unchanged full suite runs again on this exact head. Please review the complete final diff under the API/health/unknown-outcome/concurrency/privacy scope above and bind the provider review state to this head. No cloud or peer operation is authorized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8cf607ae58

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/cloud_run_warm_recovery.py Resolved
Comment thread scripts/cloud_run_warm_recovery.py Resolved
Co-Authored-By: Codex <noreply@openai.com>

Pigbibi commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Please perform the final complete R2 acceptance review at head 5c9366a against unchanged base 8aa8e8e. Both prior concurrency P1 findings are reproduced and fixed: explicit outstanding PUT intent now binds successor generations to the full desired spec including traffic; ambiguous activation requires one distinct CAS warm-100 compensation and an observed advanced resourceVersion before restoration is confirmed. A no-op without version advancement remains unconfirmed, with no retry. The three added adverse cases failed on the exact reviewed head and pass after the fix. All 111 recovery/waiter/tag tests pass, along with Ruff/Actionlint/docs and exact-range Gitleaks. Prior full Linux suite passed 6627 tests; the unchanged full CI runs again on this head. Review the whole protected workflow/operator and the actual Cloud Run optimistic-concurrency contract, particularly the compensation fence and delayed activation. No operational or peer action is authorized.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 5c9366a71c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@Pigbibi Pigbibi added the agent-approved An agent cleared this high-risk change on the current head; not an owner confirmation label Nov 5, 2026 — with ChatGPT Codex Connector
@Pigbibi
Pigbibi merged commit 826c9fb into main Nov 5, 2026
17 of 22 checks passed
@Pigbibi
Pigbibi deleted the fix/bounded-warm-recovery branch November 5, 2026 23:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-approved An agent cleared this high-risk change on the current head; not an owner confirmation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant