Skip to content

Activate the production FCVM fleet after the staging billing check - #939

Merged
ejc3 merged 10 commits into
mainfrom
fcvm-fleet-prod-enable
Nov 21, 2026
Merged

ejc3 merged 10 commits into
mainfrom
fcvm-fleet-prod-enable

Conversation

@ejc3

@ejc3 ejc3 commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

Turns production on: fcvm_fleet_prod_enabled defaults to true, and the last validation goes. That validation required three days of staging cost records (#930), and the owner waived it on 2026-11-21: the coordinator's spend meter forces sleep at the $750/day ceiling, and the AWS budget is the backstop. The owner gives the same waiver in the jumpbox session. Every other #922 condition is recorded on #922 and #930.

Every prod block is still gated on the switch, so turning production off again is this one line. The apply creates the prod stack, and seeds the prod AMI pointer from staging's release behind #933's check. The fcvm.dev page stays unpublished until A6.

After the apply, from fcvm: the prod smoke (IPv4-only and IPv6, every mode, broadcast, and an unverified-visitor probe), then the Spot reclaim, rollback-by-digest and budget-trip drills. A6 follows, which the owner has pre-approved once those pass.

Tested: test-fcvm-dev-prod, -fleet, -dev, -fleet-prod-alarms, -coordinator-release and -tests-quiet pass, and terraform 1.10.3 fmt -check and validate pass.

ejc3 added 2 commits November 20, 2026 16:08
fcvm_fleet_prod_enabled refused true until staging and dev credentials could
no longer change fcvm.dev DNS or present a certificate for a production name
(#922). #929 deletes the two parent-zone tokens and revokes the staging
certificates for fcvm.dev names; this removes the validation so the switch
can be turned on. The default stays false: turning production on is its own
change.

Tested: python3 -S -B scripts/test-fcvm-dev-prod.py (OK); terraform 1.10.3
fmt -check.
fcvm_fleet_prod_enabled defaults to true. #922 is complete: both
parent-zone DNS tokens are deleted at Cloudflare, the five staging
certificates for fcvm.dev names are revoked and proven from signed CRLs,
staging serves its subzone certificate over IPv4 and IPv6, and fcvm-arm's
fleet role is gpubox-only. #930 lifted the guard. Every prod block is
still gated on the switch, so turning production off again is this one
line.

The apply creates the prod stack (fcvm-dev-prod.tf, fcvm-fleet-prod*.tf).
The prod AMI pointer is seeded from staging's current release, behind
#933's apply-time check that its images exist. fcvm.dev's page stays
unpublished until A6 (fcvm_fleet_prod_page_public).

Tested: test-fcvm-dev-prod, -fleet, -fleet-prod-alarms,
-coordinator-release, -dev and -tests-quiet pass (the two default
assertions now say true); terraform 1.10.3 fmt -check and validate pass.
@coderabbitai

coderabbitai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 45 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 3fb2dd65-6d74-48fd-93b8-74c534dd7ebf
Commits

Reviewing files that changed from the base of the PR and between d8dbead and e743da1.

Files selected for processing: 3

  • fcvm-dev-prod.tf
  • scripts/test-fcvm-dev-prod.py
  • scripts/test-fcvm-fleet.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

ejc3 added 4 commits November 21, 2026 02:11
Codex on #930 reviewed the branch's old base, where the two parent-zone
DNS tokens and dev-server-role's read grant still existed. #936 removed
them, and this branch now carries main. test-fcvm-dev-prod.py fails if any
cloudflare_account_token or cloudflare_api_token names the fcvm.dev zone
or holds DNS Write, so a token like that cannot come back while the guard
is gone.

Tested: the new test fails against the branch's old fcvm-dev.tf (with the
tokens) and passes on this tree; test-fcvm-dev-prod and test-tests-quiet
pass.
Restore the input validation for all Terraform plans, update its error for the remaining billing check, and test both the guard and the retired DNS permission references.
ejc3 added 4 commits November 21, 2026 02:55
…t on the activation branch

Retain the parent-zone permission regression test and check that the enabled Terraform input accepts its default. Remove premature activation dates from the Terraform description and Python tests.
@ejc3 ejc3 changed the title fcvm fleet prod: turn production on Activate the production FCVM fleet after the staging billing check Nov 21, 2026
@ejc3
ejc3 changed the base branch from fcvm-fleet-prod-lift-guard to main November 21, 2026 03:04
@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

Holding as a draft. #930 merged with the activation guard narrowed to the budget condition: production stays refused until three complete UTC days of matching staging costs are recorded on #922. The jumpbox's billing check posts them from 2026-11-23 00:17 UTC. Once they're recorded, I'll rebase this onto main so it lifts that last validation and sets the default to true.

@ejc3
ejc3 marked this pull request as ready for review November 21, 2026 03:07
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-21T03:09:11.551966Z e743da1 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

This supersedes my 03:05 comment: #939 is not waiting for Nov 23. The owner waived the three-day staging budget record on 2026-11-21. The $750/day ceiling is enforced by the coordinator's spend meter, and the AWS budget is the backstop. The head (e743da1) sets the switch on and removes the last validation, and the PR is ready.

@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review
@greptileai review

@coderabbitai

coderabbitai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

The activation change has passed all five CI jobs. The latest verified daily comparison covers November 20. The billing check is scheduled for November 23 at 00:17 UTC and retries as AWS publishes data. I could verify the earlier DNS, certificate, IAM and staging checks. The direct approval for the billing exception is still being confirmed.

The coordinator samples EC2 inventory and prices. On its first inventory failure it has no burn sample. Later failures can leave it using an older instance list. A $750 target with a 15-minute projection can therefore overshoot. The AWS budget separately uses completed billing days. These are the cost controls in the candidate build. Coordinator spend sampling

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant