Skip to content

Validate the first production AMI seed under the Terraform lock - #933

Merged
ejc3 merged 2 commits into
mainfrom
fix/fleet-prod-first-seed
Nov 20, 2026
Merged

ejc3 merged 2 commits into
mainfrom
fix/fleet-prod-first-seed

Conversation

@ejc3

@ejc3 ejc3 commented Nov 20, 2026

Copy link
Copy Markdown
Owner

This addresses the AMI retention prerequisite in #922 and #930. The production activation guard and disabled default remain in place.

Changes

  • Before Terraform creates the production AMI pointer, re-read the planned staging version and manifest under the backend lock. Verify that the current and rollback images in each region are private, available and owned by the fleet. Match regional copies to their native EC2 source IDs in either copy direction. The initial pointer is protected from replacement.
  • Limit the verifier's assumed session to image metadata and the named parameter reads. The external EC2 check uses AWS CLI v2 because the jumpboxes' system botocore omits copy lineage fields. An existing production pointer retains publisher ownership and does not depend on a later staging version.
  • Acquire the publication mutex on standalone replication before reading or pruning. Preserve both production releases in the regular publisher and the build-region factory collector, including originals behind regional copies. A malformed existing production pin stops cleanup before deregistration or snapshot deletion.

Verification

  • 382 focused offline tests passed on ARM, covering release admission, publisher and dispatcher retention, and production gating. Terraform 1.10.3 formatting and whole-root validation passed.
  • Tested saved full and targeted plans using the locked providers and a loopback fixture. Changing the staging version or losing an image after planning rejected the first seed before creation. Fresh plans succeeded. An unrelated consumer update succeeded after staging changed once the production pointer existed, and the follow-up plan was empty.
  • Ran the proposed probe and verifier on the actual administration host with read-only credentials. The production parameter was absent. The verifier accepted staging generation 23 and its predecessor 22, parameter version 24, including the four east AMIs and linked four west AMIs.

Defer release-version, regional availability and copy-lineage checks until Terraform is creating the protected production parameter. Preserve the current and rollback AMIs in both retention paths and serialize manual replication with the backend lock. Exercise stale saved plans and malformed retention pointers in the offline suites.
@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 9 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 7839552f-d50b-44f0-9586-3dd943b4a4dc
Commits

Reviewing files that changed from the base of the PR and between 839e106 and dd5c5f7.

Files selected for processing: 10

  • README.md
  • fcvm-fleet-ami-pipeline.tf
  • fcvm-fleet-ami-pipeline/dispatch.py
  • fcvm-fleet-ami-pipeline/publish.py
  • fcvm-fleet-prod.tf
  • scripts/fcvm-prod-ami-seed.py
  • scripts/test-fcvm-ami-publish.py
  • scripts/test-fcvm-ami-release.py
  • scripts/test-fcvm-fleet-ami-pipeline.py
  • scripts/test-fcvm-stale-plan.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-20T17:18:50.648082Z dd5c5f7 Manual request
🔒 Security Review ✅ Completed 2026-11-20T17:20:52.430247Z dd5c5f7 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@ejc3

ejc3 commented Nov 20, 2026

Copy link
Copy Markdown
Owner Author

@codex review

@ejc3

ejc3 commented Nov 20, 2026

Copy link
Copy Markdown
Owner Author

@greptileai review

@ejc3

ejc3 commented Nov 20, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@ejc3

ejc3 commented Nov 20, 2026

Copy link
Copy Markdown
Owner Author

@codex security review

@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@ejc3

ejc3 commented Nov 20, 2026

Copy link
Copy Markdown
Owner Author

I reviewed the production seed and cleanup changes from 97aec9a through b07f530. Two issues found during review were fixed before this PR opened.

  • Under Terraform 1.10.3, the former unconditional staging dependency blocked a targeted production consumer apply after staging advanced. I reproduced the failure and verified the corrected replay creates only the downstream consumer, with no second production SSM write.
  • Publisher and factory cleanup now reject a present but malformed production pointer before AMI or snapshot deletion. AMIs referenced as the current or previous production release stay protected.
  • On ARM I ran 10 focused cases against the archived head, covering first-seed inputs, publisher retention and malformed metadata, and factory cleanup. All 10 passed. The exact initial seed lookup rechecks the planned stage version, payload, and per-region image lineage at apply. Production was dormant, so this was a fixture-based review and no live production resource was changed.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: b07f530fa1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: b07f530fa1

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@ejc3

ejc3 commented Nov 20, 2026

Copy link
Copy Markdown
Owner Author

Independent CLI code and security review of 97aec9a08fc13e22ed11697573c268dd439be42e..b07f530fa146c0ced05e1682a8968a2396612da3 found no issues. I ran codex exec -s read-only --ephemeral in a detached ARM worktree, with instructions to review that exact diff and its Terraform first-create, IAM, regional AMI lineage, and cleanup paths. The reviewer also passed 28 focused offline tests and 5 error or incomplete-inventory probes. It inspected the saved-plan cases. Independent Terraform replay results are in the earlier review receipt.

@ejc3

ejc3 commented Nov 20, 2026

Copy link
Copy Markdown
Owner Author

@codex review
@coderabbitai review
@greptileai review

Please review dd5c5f7, especially the validated image and region inputs to the first-seed verifier.

@ejc3

ejc3 commented Nov 20, 2026

Copy link
Copy Markdown
Owner Author

@codex security review

@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: dd5c5f7075

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: dd5c5f7075

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@ejc3
ejc3 merged commit 55474d1 into main Nov 20, 2026
6 checks passed
ejc3 added a commit that referenced this pull request Nov 21, 2026
)

* fcvm fleet prod: lift the activation guard once #922 is done

fcvm_fleet_prod_enabled refused true until staging and dev credentials could
no longer change fcvm.dev DNS or present a certificate for a production name
(#922). #929 deletes the two parent-zone tokens and revokes the staging
certificates for fcvm.dev names; this removes the validation so the switch
can be turned on. The default stays false: turning production on is its own
change.

Tested: python3 -S -B scripts/test-fcvm-dev-prod.py (OK); terraform 1.10.3
fmt -check.

* fcvm fleet prod: turn production on

fcvm_fleet_prod_enabled defaults to true. #922 is complete: both
parent-zone DNS tokens are deleted at Cloudflare, the five staging
certificates for fcvm.dev names are revoked and proven from signed CRLs,
staging serves its subzone certificate over IPv4 and IPv6, and fcvm-arm's
fleet role is gpubox-only. #930 lifted the guard. Every prod block is
still gated on the switch, so turning production off again is this one
line.

The apply creates the prod stack (fcvm-dev-prod.tf, fcvm-fleet-prod*.tf).
The prod AMI pointer is seeded from staging's current release, behind
#933's apply-time check that its images exist. fcvm.dev's page stays
unpublished until A6 (fcvm_fleet_prod_page_public).

Tested: test-fcvm-dev-prod, -fleet, -fleet-prod-alarms,
-coordinator-release, -dev and -tests-quiet pass (the two default
assertions now say true); terraform 1.10.3 fmt -check and validate pass.

* Pin why the prod guard can go: no token here writes fcvm.dev DNS

Codex on #930 reviewed the branch's old base, where the two parent-zone
DNS tokens and dev-server-role's read grant still existed. #936 removed
them, and this branch now carries main. test-fcvm-dev-prod.py fails if any
cloudflare_account_token or cloudflare_api_token names the fcvm.dev zone
or holds DNS Write, so a token like that cannot come back while the guard
is gone.

Tested: the new test fails against the branch's old fcvm-dev.tf (with the
tokens) and passes on this tree; test-fcvm-dev-prod and test-tests-quiet
pass.

* Keep the fleet activation guard until the staging budgets match

Restore the input validation for all Terraform plans, update its error for the remaining billing check, and test both the guard and the retired DNS permission references.

* Check retired DNS permissions in tokens with hyphenated labels

---------

Co-authored-by: ejc3 <ejc3@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant