Skip to content

Retire the fleet’s parent DNS tokens and isolate propagation tests - #936

Merged
ejc3 merged 3 commits into
mainfrom
fcvm-dev-retire-parent-zone
Nov 21, 2026
Merged

ejc3 merged 3 commits into
mainfrom
fcvm-dev-retire-parent-zone

Conversation

@ejc3

@ejc3 ejc3 commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

Part of #922. Applying this change revokes the two named staging/dev parent-zone DNS writers at Cloudflare. The staging certificate secret, legacy secret and all of its versions, and scene token stay available. Revocation of the five identified certificates is a separate operation tracked in #922.

Changes

  • Remove the coordinator and dev DNS-test Cloudflare token resources, their Secrets Manager copies, the obsolete dev role read and the token output. The cutover already moved staging runtime DNS-01 renewal to its own authoritative server.
  • Delegate dnstest.fcvm.dev to a Route 53 public test zone. The dev instance role uses temporary credentials restricted to the existing dev source IPs, this zone ARN, CREATE/DELETE, and probe-prefixed A/AAAA/TXT records. No test credential can change the parent delegation. From either metal box, python3 scripts/fcvm-dns-probe.py times all four authoritative nameservers before querying the two public recursors, then cleans up. Measurements start at CREATE; the order avoids caching a transient NXDOMAIN. This measures Route 53 publication and recursive visibility; use a newly launched staging hostname to time the fleet coordinator itself. The child zone costs USD 0.50/month plus query charges.
  • The previous cold-bootstrap command required the retiring token and a legacy CNAME. The recovery helper now verifies a current stage-secret backup or the protected /run/fcvm-fleet/tls/tls.json already held on a staging host, rebuilds the omitted certificate metadata, checks trust, the narrow SAN, key and scene token, and labels only a verified empty-container restore. The strict 30-day threshold applies to cutover readiness. An explicit restore accepts an unexpired leaf inside the renewal window so the coordinator can renew through its delegated authority. An idle staging tile also retains the matching three fields on its EBS root. Default invocation still verifies without writing. On the new staging host, byte comparisons against the current secret succeeded for the cert, key and scene token; the recovery test covers that renderer JSON shape.
  • Keep the read-only certificate inventory and CRL verifier from fcvm.dev: read-only audit of staging's parent-zone wildcards with a strict CRL proof (#922) #929/Verify missing staging wildcard certificates from pinned public PEMs #932. The prototype revoker from the first commit could not revoke the three certificates whose private keys are missing. The updated PR contains no mutating revocation script.

Rollout

  1. Verify fresh host and tile coverage and the five-certificate operation’s independence from the old tokens.
  2. From the normal locked Terraform workflow, first target the new delegation, zone-ID TXT record and dev role policy (and their dependencies). Check the existing tokens still exist at the issuer. Run the propagation probe from a metal dev box and verify it creates, resolves and deletes its own record. Confirm IAM denies NS, apex, parent and production changes. The live probe remains pending until the test zone deploys.
  3. Review the full saved plan before retiring the two old tokens, confirm the issuer-side removals and check the existing protected endpoints. Run the post-apply Terraform check. Track certificate/CRL evidence and production activation separately in Isolate fcvm.dev production DNS and certificate authority from staging and dev credentials before activation #922/Protect retired fleet DNS permissions and retain the production budget guard #930.

Validation

202 tests passed across ten focused suites on ARM. The negative-cache and near-expiry regressions each failed before their fix, and the expired-leaf recovery check remains fail-closed. Terraform 1.10.3 format and backend-free validation passed with existing unrelated provider warnings. Pyflakes and git diff --check passed. No live infrastructure was changed by this source update.

…nt wildcards (#922)

The deferred retirement step of #922, for the owner's approval after
staging's cutover to its subzone certificate (#923).

- fcvm-dev.tf: delete cloudflare_account_token.fcvm_fleet_dns and the
  fcvm-fleet/cloudflare-dns-token secret and version (deleting the
  resource deletes the token at Cloudflare), the dns_token output entry and
  the dns_write local.
- fcvm-dns-test.tf: delete the dev boxes' DNS propagation token, its secret
  and dev-server-role's read grant (constellation-account.tf), with its
  AGENTS.md entry, test and CI step.
- test-fcvm-dev.py: no cloudflare_account_token or cloudflare_api_token in
  any .tf file may name the fcvm.dev zone, and the removed names stay gone.
- scripts/fcvm-revoke-parent-wildcards.py: lists the certificates for names
  in fcvm.dev outside the fleet subzones held in fcvm-fleet/tls-wildcard's
  versions, and with --revoke SERIAL... revokes exactly those serials, each
  signed with its own key (RFC 8555 section 7.6, reason superseded). A
  serial that is not listed is refused before any request. Proof of
  revocation is fcvm-parent-wildcard-audit.py --check --expect (#929).

Tested: test-fcvm-revoke-parent-wildcards.py (7 tests) against a local ACME
server that verifies each JWS with the certificate's key, for EC P-256,
P-384 and RSA keys. It also covers revoking only the named serials and
refusing an unlisted one. The token test fails on main's fcvm-dev.tf.
test-fcvm-dev, -fleet, -fleet-account, -constellation-account, -agents-md,
-tests-quiet, -parent-wildcard-audit and -dev-prod pass. terraform 1.10.3
fmt -check and validate pass.
@coderabbitai

coderabbitai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 43 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: cc887970-4e7d-4586-9c0d-b406277e3017
Commits

Reviewing files that changed from the base of the PR and between 55474d1 and 69db951.

Files selected for processing: 11

  • .github/workflows/lambda-tests.yml
  • AGENTS.md
  • constellation-account.tf
  • fcvm-dev.tf
  • fcvm-dns-test.tf
  • scripts/fcvm-dns-probe.py
  • scripts/fcvm-stage-certificate.py
  • scripts/test-fcvm-dev.py
  • scripts/test-fcvm-dns-test.py
  • scripts/test-fcvm-fleet.py
  • scripts/test-fcvm-stage-certificate.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-21T01:17:24.793889Z 69db951 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

I’ll use this PR for the jumpbox-side changes coordinated in #922. Please leave the branch with me while I add the isolated propagation workload, remove the old issuance dependency, and keep revocation in the separately inventoried five-certificate operation. The running staging cutover and its host/tile checks still gate retirement.

@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

The five-serial runbook cannot complete. collect() finds two leaves from the legacy secret. Passing the five independently pinned serials fails at the unknown-serial check before any revocation. Passing just the two leaves three production-name certificates valid. Also, the CRL proof needs the independently pinned public chains for the other three leaves via --public-chain.

The administration host already established domain authorization with Let’s Encrypt for all three SANs before the coordinator switched its challenge handler. After the replacement host and both viewer paths pass, the documented Certbot account-key revocation can cover all five pinned public leaves. This uses the distribution client and requires no custom signing implementation or legacy private-key extraction. Please remove the unused revoker in favor of documenting the actual workflow. The staging-certificate verification/restore fix also belongs on this branch because its current cold-bootstrap path needs one of the tokens being destroyed. Domain authorization and pinned certificate baseline

…copies

Delegate dnstest.fcvm.dev to Route 53 and grant bounded CREATE and DELETE for probe records to the dev instance role. Add the timed probe, cleanup and IAM tests, and the rollout procedure.

Replace the parent-token cold issuance helper with verified restoration from the actual renderer JSON. Remove the duplicate key-based revoker from this token change and retain the independent certificate audit.
@ejc3 ejc3 changed the title fcvm.dev: retire the parent-zone DNS tokens and revoke staging's parent wildcards (#922) Retire the fleet’s parent DNS tokens and isolate propagation tests Nov 21, 2026
@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

@codex review

@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

@greptileai review

@coderabbitai

coderabbitai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. What shall we delve into next?

Reviewed commit: 7c06c80d60

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…S recovery material

Measure each DNS result from CREATE but start recursive lookups only after every authoritative server returns the record. Cover negative caching, near-expiry renderer recovery and expired backups. Keep the 30-day cutoff for readiness, and document the retained tile's certificate files.
@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

@codex review

@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

@greptileai review

@coderabbitai

coderabbitai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: 69db951665

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ejc3
ejc3 merged commit 0750ba2 into main Nov 21, 2026
6 checks passed
ejc3 added a commit that referenced this pull request Nov 21, 2026
Codex on #930 reviewed the branch's old base, where the two parent-zone
DNS tokens and dev-server-role's read grant still existed. #936 removed
them, and this branch now carries main. test-fcvm-dev-prod.py fails if any
cloudflare_account_token or cloudflare_api_token names the fcvm.dev zone
or holds DNS Write, so a token like that cannot come back while the guard
is gone.

Tested: the new test fails against the branch's old fcvm-dev.tf (with the
tokens) and passes on this tree; test-fcvm-dev-prod and test-tests-quiet
pass.
ejc3 added a commit that referenced this pull request Nov 21, 2026
…t guard (#930)

* fcvm fleet prod: lift the activation guard once #922 is done

fcvm_fleet_prod_enabled refused true until staging and dev credentials could
no longer change fcvm.dev DNS or present a certificate for a production name
(#922). #929 deletes the two parent-zone tokens and revokes the staging
certificates for fcvm.dev names; this removes the validation so the switch
can be turned on. The default stays false: turning production on is its own
change.

Tested: python3 -S -B scripts/test-fcvm-dev-prod.py (OK); terraform 1.10.3
fmt -check.

* Pin why the prod guard can go: no token here writes fcvm.dev DNS

Codex on #930 reviewed the branch's old base, where the two parent-zone
DNS tokens and dev-server-role's read grant still existed. #936 removed
them, and this branch now carries main. test-fcvm-dev-prod.py fails if any
cloudflare_account_token or cloudflare_api_token names the fcvm.dev zone
or holds DNS Write, so a token like that cannot come back while the guard
is gone.

Tested: the new test fails against the branch's old fcvm-dev.tf (with the
tokens) and passes on this tree; test-fcvm-dev-prod and test-tests-quiet
pass.

* Keep the fleet activation guard until the staging budgets match

Restore the input validation for all Terraform plans, update its error for the remaining billing check, and test both the guard and the retired DNS permission references.

* Check retired DNS permissions in tokens with hyphenated labels

---------

Co-authored-by: ejc3 <ejc3@users.noreply.github.com>
ejc3 added a commit that referenced this pull request Nov 21, 2026
)

* fcvm fleet prod: lift the activation guard once #922 is done

fcvm_fleet_prod_enabled refused true until staging and dev credentials could
no longer change fcvm.dev DNS or present a certificate for a production name
(#922). #929 deletes the two parent-zone tokens and revokes the staging
certificates for fcvm.dev names; this removes the validation so the switch
can be turned on. The default stays false: turning production on is its own
change.

Tested: python3 -S -B scripts/test-fcvm-dev-prod.py (OK); terraform 1.10.3
fmt -check.

* fcvm fleet prod: turn production on

fcvm_fleet_prod_enabled defaults to true. #922 is complete: both
parent-zone DNS tokens are deleted at Cloudflare, the five staging
certificates for fcvm.dev names are revoked and proven from signed CRLs,
staging serves its subzone certificate over IPv4 and IPv6, and fcvm-arm's
fleet role is gpubox-only. #930 lifted the guard. Every prod block is
still gated on the switch, so turning production off again is this one
line.

The apply creates the prod stack (fcvm-dev-prod.tf, fcvm-fleet-prod*.tf).
The prod AMI pointer is seeded from staging's current release, behind
#933's apply-time check that its images exist. fcvm.dev's page stays
unpublished until A6 (fcvm_fleet_prod_page_public).

Tested: test-fcvm-dev-prod, -fleet, -fleet-prod-alarms,
-coordinator-release, -dev and -tests-quiet pass (the two default
assertions now say true); terraform 1.10.3 fmt -check and validate pass.

* Pin why the prod guard can go: no token here writes fcvm.dev DNS

Codex on #930 reviewed the branch's old base, where the two parent-zone
DNS tokens and dev-server-role's read grant still existed. #936 removed
them, and this branch now carries main. test-fcvm-dev-prod.py fails if any
cloudflare_account_token or cloudflare_api_token names the fcvm.dev zone
or holds DNS Write, so a token like that cannot come back while the guard
is gone.

Tested: the new test fails against the branch's old fcvm-dev.tf (with the
tokens) and passes on this tree; test-fcvm-dev-prod and test-tests-quiet
pass.

* Keep the fleet activation guard until the staging budgets match

Restore the input validation for all Terraform plans, update its error for the remaining billing check, and test both the guard and the retired DNS permission references.

* Check retired DNS permissions in tokens with hyphenated labels

---------

Co-authored-by: ejc3 <ejc3@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant