Repository navigation
fcvm-arm: read the fcvm release logs and coordinators, and run the prod Spot drill (#942) - #943
Conversation
…od Spot drill (#942) The owner, 2026-11-21: "If you can't read something ask for perms." fcvm-arm verifies the fleet but could not say why a prod promotion failed (release Lambda logs denied), see the coordinators' ECS state or logs, or run the Spot reclaim drill that gates fcvm.dev going public. - constellation-account.tf: dev-server-role, pinned to fcvm-arm's instance, may read the logs of /aws/lambda/fcvm-* in this account: FilterLogEvents, GetLogEvents, DescribeLogStreams. Logs only. A function's configuration holds its environment, so GetFunctionConfiguration stays denied. - fcvm-fleet-account.tf, observe-the-coordinators-and-run-the-spot-drill on fcvm-fleet-terraform: - ECS describe and list for clusters, services and tasks (no task definition, which names the secrets); - log reads on /ecs/fcvm-fleet-coordinator and /ecs/fcvm-fleet-prod-coordinator; - fis:StartExperiment on the Spot drill template only, plus Get/List/Stop experiments. - fcvm-fleet-prod-drills.tf, behind fcvm_fleet_prod_enabled: an FIS experiment template that sends one running Purpose=fcvm-fleet-host, Fleet=prod instance in us-east-1 the two-minute Spot interruption notice. Its role may interrupt only instances with those two tags. Tested: test-fcvm-fleet-account.py (25 tests), with new ObserveTests: - the observe policy's exact action set, no task definition, and the drill template as the only StartExperiment target; - the drill interrupts one running prod host only; - the dev box reads only fcvm Lambda logs. test-fcvm-fleet (the drills file joins the every-prod-block-is-gated check), test-constellation-account and test-tests-quiet pass. terraform 1.10.3 fmt -check and validate pass.
Walkthrough
Merge Risk:
|
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3e07d853ec
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The prod promotion now names its failure: every invocation of the prod release functions returns AccessDeniedException (ejc3/fcvm run 44869951743). The CI role's code allows lambda:InvokeFunction on exactly those functions, so the live role differs from the code. Seeing how takes IAM reads fcvm-arm does not have. - constellation-account.tf: dev-server-role, pinned to fcvm-arm's instance, may GetRole, GetRolePolicy, ListRolePolicies, ListAttachedRolePolicies and SimulatePrincipalPolicy on fcvm-stage-ci and fcvm-prod-ci only. Read only. Tested: test-fcvm-fleet-account.py (26 tests, with test_the_dev_box_reads_and_simulates_only_the_fcvm_ci_roles); terraform 1.10.3 fmt -check and validate pass.
|
Added one read-only statement: The prod promotion now names its failure. ejc3/fcvm run 44869951743 shows all 11 invocations of the prod release functions returning |
…nctions Exclude their resolved $LATEST ARNs from the explicit deny. Keep the invoke allow on the original function ARNs and test the exact exception.
…tics Provision FIS service-linked permissions and serialize production drill requests in a fleet Lambda. Persist request receipts, require healthy hosts in two zones, and return the experiment time and target zones. Limit the dev role to invoking the guard and reading coordinator diagnostics. Exercise the handler and IAM boundaries in CI and document the rehearsal.
|
@codex review |
|
@coderabbitai review |
|
@greptileai review |
|
@codex security review |
Action not completed
|
|
Codex Review: Didn't find any major issues. 👍 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Handle the access denial FIS returns when a deleted experiment has no ownership tag. Preserve the receipt and propagate errors if an experiment is still listed or the scan is incomplete. Cover these cases with paginated regression tests.
|
@codex review |
|
@coderabbitai review |
|
@greptileai review |
|
@codex security review |
Action not completed
|
|
Codex Review: Didn't find any major issues. Another round soon, please! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
Changes
$LATESTARN on an unqualified invocation. The CI role's explicit deny now exempts that ARN for its two release functions. The allow statement still names only those two unqualified functions.Verification
Targeted Python suites, Terraform formatting and validation, and the repository's GitHub checks cover these changes. Live Lambda DryRun reproduced the original explicit denial. With the corrected policy shape the intended call returned HTTP 204, while a different function, a numbered version and an alias were denied. DryRun executes no function code.
Summary by CodeRabbit