Skip to content

fcvm-arm: read the fcvm release logs and coordinators, and run the prod Spot drill (#942) - #943

Merged
ejc3 merged 5 commits into
mainfrom
fcvm-dev-observe-and-drill
Nov 21, 2026
Merged

ejc3 merged 5 commits into
mainfrom
fcvm-dev-observe-and-drill

Conversation

@ejc3

@ejc3 ejc3 commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

Changes

  • Give fcvm-arm read access to the two production release log groups and the two fleet CI roles. In the fleet account, limit ECS and log inspection to the staging and production coordinators in their home region.
  • Fix the production promotion's AccessDenied error. Lambda checks the underlying $LATEST ARN on an unqualified invocation. The CI role's explicit deny now exempts that ARN for its two release functions. The allow statement still names only those two unqualified functions.
  • Provision the FIS service-linked role, the experiment role and a single-host production Spot interruption template. The dev role invokes a fixed fleet Lambda. Its handler checks both host health and availability-zone diversity, records each request in S3, and limits stop to an experiment belonging to this template.
  • Document the rehearsal commands and add CI regressions for IAM scope, request recovery, and drill admission. Deployment leaves the production budget and running hosts intact. Track the actual rehearsals in fcvm fleet prod: Spot reclaim and budget-trip drills before fcvm.dev goes public #942.

Verification

Targeted Python suites, Terraform formatting and validation, and the repository's GitHub checks cover these changes. Live Lambda DryRun reproduced the original explicit denial. With the corrected policy shape the intended call returned HTTP 204, while a different function, a numbered version and an alias were denied. DryRun executes no function code.

Summary by CodeRabbit

  • New Features
    • Added a guarded production Spot interruption drill. Operators can start, check, and stop experiments; starts require healthy production hosts across at least two Availability Zones and prevent overlapping drills.
  • Documentation
    • Added a procedure for verifying readiness, running the two-minute interruption drill, checking its result, and stopping an active experiment. The procedure notes that an interruption notice already delivered may still take effect.

…od Spot drill (#942)

The owner, 2026-11-21: "If you can't read something ask for perms."
fcvm-arm verifies the fleet but could not say why a prod promotion
failed (release Lambda logs denied), see the coordinators' ECS state or
logs, or run the Spot reclaim drill that gates fcvm.dev going public.

- constellation-account.tf: dev-server-role, pinned to fcvm-arm's
  instance, may read the logs of /aws/lambda/fcvm-* in this account:
  FilterLogEvents, GetLogEvents, DescribeLogStreams. Logs only. A
  function's configuration holds its environment, so
  GetFunctionConfiguration stays denied.
- fcvm-fleet-account.tf, observe-the-coordinators-and-run-the-spot-drill
  on fcvm-fleet-terraform:
  - ECS describe and list for clusters, services and tasks (no task
    definition, which names the secrets);
  - log reads on /ecs/fcvm-fleet-coordinator and
    /ecs/fcvm-fleet-prod-coordinator;
  - fis:StartExperiment on the Spot drill template only, plus
    Get/List/Stop experiments.
- fcvm-fleet-prod-drills.tf, behind fcvm_fleet_prod_enabled: an FIS
  experiment template that sends one running Purpose=fcvm-fleet-host,
  Fleet=prod instance in us-east-1 the two-minute Spot interruption
  notice. Its role may interrupt only instances with those two tags.

Tested: test-fcvm-fleet-account.py (25 tests), with new ObserveTests:
- the observe policy's exact action set, no task definition, and the
  drill template as the only StartExperiment target;
- the drill interrupts one running prod host only;
- the dev box reads only fcvm Lambda logs.
test-fcvm-fleet (the drills file joins the every-prod-block-is-gated
check), test-constellation-account and test-tests-quiet pass.
terraform 1.10.3 fmt -check and validate pass.
@coderabbitai

coderabbitai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Explain Change →

Walkthrough

This change adds a production Spot interruption drill using FIS and a guarded Lambda handler. It adds scoped permissions for operating and observing the drill, receipt and experiment handling, operator instructions, and automated tests.

Changes: Production Spot drill

Layer / File(s) Summary
FIS template and guarded Lambda
fcvm-fleet-prod-drills.tf, scripts/test-fcvm-fleet-account.py, scripts/test-fcvm-fleet.py
Adds a production FIS experiment targeting one running Spot host with the required tags and a two-minute interruption notice. Adds scoped experiment and Lambda roles, drill state storage, and a Lambda with one reserved concurrent execution. Tests inspect the resource configuration and production file set.
Drill request and receipt handling
scripts/fcvm-prod-spot-drill.py, scripts/test-fcvm-prod-spot-drill.py, .github/workflows/lambda-tests.yml, README.md
Adds start, status, and stop handling with host health checks, request receipts, experiment ownership checks, and recovery behavior. Adds offline tests, runs them in the workflow, and documents the operator procedure.
Scoped operator and CI permissions
fcvm-fleet-account.tf, constellation-account.tf, fcvm-fleet-prod-ci.tf, scripts/test-fcvm-fleet-account.py, scripts/test-fcvm-coordinator-release.py
Adds scoped coordinator ECS and Logs access, production drill Lambda invocation, and instance-pinned access to named production logs and CI role policies. Extends the production CI deny exception to include release-function $LATEST ARNs. Tests check these permissions and scopes.

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant Operator
  participant Lambda
  participant S3
  participant EC2
  participant FIS
  Operator->>Lambda: Send start request with request ID
  Lambda->>S3: Read drill receipts
  Lambda->>EC2: Check production Spot host health and AZs
  Lambda->>S3: Save request reservation
  Lambda->>FIS: Start fixed experiment with request ID
  FIS-->>Lambda: Return experiment details
  Lambda->>S3: Save experiment receipt
  Lambda-->>Operator: Return experiment state and selected AZs
Loading

Possibly related issues

  • ejc3/aws#942 — Tracks the production Spot reclaim drill that this change provisions and supports.

Possibly related PRs

  • ejc3/aws#917 — Defines the production fleet resources and Fleet=prod tag used by the drill’s FIS target.
  • ejc3/aws#930 — Shares the production activation gate and budget guard used by the fleet Terraform resources.
Merge Risk: Minimal · up to 96688

The change adds a guarded production Spot interruption drill and scoped read access for fcvm-arm, and it fixes the $LATEST deny exception for the release functions. No concrete defect was identified. One minor, unverified question remains about ECS ListTasks authorization for Fargate services. It would only limit read-only visibility and does not block merging.

Pre-merge checks: Passed 2

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names the two main areas of the PR: read access for fcvm-arm to fcvm release logs and coordinator state, and the production Spot interruption drill. The changeset adds IAM read grants, coord…
Full details: Title check

Explanation

The title names the two main areas of the PR: read access for fcvm-arm to fcvm release logs and coordinator state, and the production Spot interruption drill. The changeset adds IAM read grants, coordinator ECS and log access, and the prod Spot drill Terraform, Lambda handler, README procedure, and tests. The title covers the main change, though it is a long compound title and omits the production policy fix for the AccessDenied errors. The title also includes a PR number (#942), while the objectives reference PR #943, which is a minor mismatch and not a title-content issue.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-21T05:38:56.857110Z 966886e Manual request
🔒 Security Review ✅ Completed 2026-11-21T05:41:17.589913Z 966886e Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3e07d853ec

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread fcvm-fleet-account.tf Resolved
The prod promotion now names its failure: every invocation of the prod
release functions returns AccessDeniedException (ejc3/fcvm run
44869951743). The CI role's code allows lambda:InvokeFunction on exactly
those functions, so the live role differs from the code. Seeing how
takes IAM reads fcvm-arm does not have.

- constellation-account.tf: dev-server-role, pinned to fcvm-arm's
  instance, may GetRole, GetRolePolicy, ListRolePolicies,
  ListAttachedRolePolicies and SimulatePrincipalPolicy on fcvm-stage-ci
  and fcvm-prod-ci only. Read only.

Tested: test-fcvm-fleet-account.py (26 tests, with
test_the_dev_box_reads_and_simulates_only_the_fcvm_ci_roles);
terraform 1.10.3 fmt -check and validate pass.
@ejc3

ejc3 commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner Author

Added one read-only statement: iam:GetRole, GetRolePolicy, ListRolePolicies, ListAttachedRolePolicies and SimulatePrincipalPolicy on fcvm-stage-ci and fcvm-prod-ci only, pinned to fcvm-arm.

The prod promotion now names its failure. ejc3/fcvm run 44869951743 shows all 11 invocations of the prod release functions returning AccessDeniedException. CloudWatch shows 0 invocations of either function, so IAM refused the calls before they ran, although fcvm-prod-ci's policy in code allows lambda:InvokeFunction on exactly those functions. This grant lets fcvm-arm see how the live role differs, instead of asking for it.

ejc3 added 2 commits November 21, 2026 05:32
…nctions

Exclude their resolved $LATEST ARNs from the explicit deny. Keep the invoke allow on the original function ARNs and test the exact exception.
…tics

Provision FIS service-linked permissions and serialize production drill requests in a fleet Lambda. Persist request receipts, require healthy hosts in two zones, and return the experiment time and target zones. Limit the dev role to invoking the guard and reading coordinator diagnostics. Exercise the handler and IAM boundaries in CI and document the rehearsal.
@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

@codex review

@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

@greptileai review

@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

@codex security review

@coderabbitai

coderabbitai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Action not completed

Head commit changed.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: c5f684d55b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Handle the access denial FIS returns when a deleted experiment has no ownership tag. Preserve the receipt and propagate errors if an experiment is still listed or the scan is incomplete. Cover these cases with paginated regression tests.
@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

@codex review

@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

@greptileai review

@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

@codex security review

@coderabbitai

coderabbitai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Action not completed

Pull request is closed.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: 966886e5aa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 966886e5aa

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@ejc3
ejc3 merged commit 0514503 into main Nov 21, 2026
5 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant