Skip to content

fcvm-fleet-terraform: gpubox-only allow list instead of AdministratorAccess (#922) - #934

Merged
ejc3 merged 2 commits into
mainfrom
fcvm-fleet-gpubox-scoped-role
Nov 21, 2026
Merged

ejc3 merged 2 commits into
mainfrom
fcvm-fleet-gpubox-scoped-role

Conversation

@ejc3

@ejc3 ejc3 commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

Part of #922. fcvm-arm's dev-server-role may assume fcvm-fleet-terraform in the fleet account. That account holds staging's certificates and tokens, and will hold production's. The role had AdministratorAccess, which reaches any secret in three ways:

  • reading it;
  • passing a role that can read it;
  • changing what a role-holding workload runs: a launch template, /fcvm-fleet/ami-release, or a host through Run Command.

gpubox.sh also launched with the fleet's own fcvm-fleet-host profile, which reads the certificate and the agents' Access token. The owner chose factory-only AMI builds on 2026-11-21, so fcvm-arm needs this account for gpuboxes alone (#922 comments).

Change

  • fcvm-fleet-account.tf. The AdministratorAccess attachment is removed, and gpubox-only replaces it. Every allowed call is limited to us-west-2 and us-east-1:

    • describes and the quota read, plus GetParameter on /fcvm-fleet/ami-release and -copies;
    • RunInstances only with aws:RequestTag/Purpose = fcvm-fleet-gpubox, plus the image, subnet, security group, network interface, volume and spot request a launch uses;
    • six allowed instance tag keys at launch and only LeaseUntil on existing gpuboxes;
    • TerminateInstances and ssm:SendCommand (AWS-RunShellScript) on gpubox-tagged instances only;
    • iam:PassRole of fcvm-fleet-gpubox, to EC2 only;
    • S3 get, put and delete under the asset bucket's gpubox/ prefix, which covers copy's staging objects (the signed URLs carry this role's permissions) and the admission lock and ledger.

    An explicit deny backstops the allow list: secretsmanager:*, kms:Decrypt, IAM writes, ssm:PutParameter, ssm:StartSession, launch-template edits and ModifyInstanceAttribute. The G admission deny stays as it is.

  • The new fcvm-fleet-gpubox role and instance profile. They grant SSM managed-instance core and release-artifact reads. An explicit deny limits parameter reads to AWS public service parameters and the two fleet release pointers in us-east-1.

  • AGENTS.md and the account file's header describe the narrowed role.

Apply order

  1. This PR, which creates the profile and narrows the role.
  2. gpubox: launch as fcvm-fleet-gpubox, which reads no secret; AMI builds are factory-only fcvm#1979, where gpubox.sh launches with fcvm-fleet-gpubox. I merge it as soon as this apply finishes.

Between the two steps, gpubox up is refused because the old script passes the host role. Nothing else is affected. fcvm's build-ami.sh and measure-boot.sh stop working from fcvm-arm, which is the decision.

Verification

  • test-fcvm-fleet-account.py has 17 tests. The new GpuboxRoleTests check:
    • no AdministratorAccess is attached, and only the two policies are;
    • every allowed Action is on the gpubox list;
    • instance actions need the gpubox tag;
    • PassRole is only for the gpubox role, and S3 is only under gpubox/;
    • the deny names secrets, IAM writes and what workloads run;
    • the gpubox role reads no secret.
  • These mutations each turn the suite red:
    • the admin attachment restored;
    • a secret read allowed;
    • a launch without the gpubox tag;
    • PassRole of the host role;
    • the whole bucket allowed;
    • a gpubox role that reads a secret.
  • test-constellation-account, test-fcvm-fleet, test-agents-md and test-tests-quiet pass.
  • Terraform 1.10.3 fmt -check and validate (backend-free init) pass.
  • After the apply: gpubox.sh up, run, copy and down from fcvm-arm with #1979. A secretsmanager get-secret-value on fcvm-fleet/cloudflare-dns-token from the same session must return AccessDenied.

…Access; gpuboxes run without secrets (#922)

The fleet account holds staging's and, once enabled, production's
certificates and tokens. fcvm-arm's dev-server-role may assume
fcvm-fleet-terraform there, and that role held AdministratorAccess. An
administrator reaches any secret: by reading it, by passing a role that
can, or by changing what a role-holding workload runs (a launch template,
the AMI pointer, a host through Run Command). gpubox.sh also launched
with the fleet's own host profile, which reads the certificate and the
agents' Access token. The owner chose image-factory-only AMI builds
(2026-11-21), so fcvm-arm needs this account for gpuboxes alone.

- fcvm-fleet-account.tf: drop the AdministratorAccess attachment. Add the
  gpubox-only policy:
  - describes, the quota read, and GetParameter on the two AMI release
    parameters, all in us-west-2 and us-east-1;
  - RunInstances only with Purpose=fcvm-fleet-gpubox;
  - CreateTags at launch, or on gpubox-tagged instances, and
    TerminateInstances on gpubox-tagged instances;
  - SendCommand of AWS-RunShellScript to gpubox-tagged instances;
  - PassRole of fcvm-fleet-gpubox to EC2 only;
  - S3 get, put and delete under the asset bucket's gpubox/ prefix (copy
    staging objects, the admission lock and ledger);
  - an explicit deny of secretsmanager:*, kms:Decrypt, IAM writes,
    ssm:PutParameter, ssm:StartSession, launch-template edits and
    ModifyInstanceAttribute.
  The G admission deny stays.
- New fcvm-fleet-gpubox role and instance profile: SSM managed-instance
  core and release-artifact reads only, no secret.
- AGENTS.md and the account file's header describe the narrowed role.
- test-fcvm-fleet-account.py, GpuboxRoleTests:
  - no AdministratorAccess, and only the two policies on the role;
  - every allowed Action is on the gpubox list;
  - instance actions need the gpubox tag;
  - PassRole only of the gpubox role, and S3 only under gpubox/;
  - the deny names secrets, IAM writes and what workloads run;
  - the gpubox role reads no secret.

Apply order: this first (it creates the profile), then fcvm's gpubox.sh
change, which launches with fcvm-fleet-gpubox. Until fcvm merges, gpubox
up is refused, since the old script passes the host role; nothing else
is affected. fcvm's build-ami.sh and measure-boot.sh no longer run from
fcvm-arm.

Tested: test-fcvm-fleet-account.py (17 tests). Six mutations each turn it
red: the admin attachment back, a secret read allowed, a launch without
the gpubox tag, PassRole of the host role, the whole bucket, and a gpubox
role that reads a secret. test-constellation-account, test-fcvm-fleet,
test-agents-md and test-tests-quiet pass. terraform 1.10.3 fmt -check and
validate (backend-free init) pass.
@coderabbitai

coderabbitai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 38 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1d9c44fc-2adc-48dc-830f-e641b25f81ab
Commits

Reviewing files that changed from the base of the PR and between 55474d1 and 254b0a6.

Files selected for processing: 4

  • AGENTS.md
  • fcvm-fleet-account.tf
  • scripts/test-bedrock-haiku.py
  • scripts/test-fcvm-fleet-account.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-21T01:22:35.613703Z 254b0a6 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai review
@greptileai review
@codex security-review

The code review on a473ae1 has completed. Please check the dev role, instance profile, and isolation of staging and production credentials.

@coderabbitai

coderabbitai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

Please tighten the new gpubox instance profile before apply. I checked the live AmazonSSMManagedInstanceCore policy version. It grants both ssm:GetParameter and ssm:GetParameters on every parameter. The static test only inspects the inline policy, so it misses those effective permissions. Preserve SSM registration and command delivery while limiting parameter reads to the intended public or release paths. Add a regression check covering the effective permissions and verify authorized and unrelated parameter ARNs through IAM simulation.

Please also keep the gpubox ownership tag immutable after launch. The TagAndEndOnlyGpuboxes statement currently restricts which instances can be tagged while allowing any request tag key, including Purpose. Limit post-launch tagging to the keys gpubox actually updates. That keeps fleet discovery and cleanup reliable.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: a473ae18b5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ejc3

ejc3 commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner Author

RED-VERIFIED: Both missing-key paths failed with 403 under the old client. fcvm#1979 fixed both at 04b476f4 and merged with the identical tree as 382558a0. All 51 operator-tool tests passed, including preservation of existing unreadable state. The role can keep its object-only S3 grant. The ordinary post-apply GPU-box smoke is tracked separately.

Cross-check with the running tool and 90-day role inventory found a cold-start regression. fcvm fleet/tools/g-admission.sh:148 accepts only NoSuchKey when the admission ledger does not yet exist. The lock read at line 112 has the same assumption if a competing launcher has already released its lock. A caller with the proposed prefix object grant but without bucket listing gets HTTP 403 in both cases. Existing ledger contents can hide this on a happy-path launch. AWS GetObject error behavior

Please preserve these paths in the companion fcvm PR. An exact-key conditional create (If-None-Match: *) can initialize an absent ledger under the existing grant. A bounded retry of the conditional lock acquisition can distinguish a disappearing lock from one still held. Keep errors on unreadable existing state fail-closed. Please exercise an initially missing ledger and a disappearing lock under restricted IAM before marking the migration complete.

Across all 130,733 fleet-account events from sessions containing “gpubox”, 126,089 succeeded across 17 operations. The 36 KMS CreateGrant and 36 ReEncrypt calls in us-west-2 all came via the EC2 frontend and used the AWS-managed alias/aws/ebs key. I checked its policy: it already allows those actions specifically via regional EC2 for this account. Keep broad KMS rights out and verify the current encrypted AMI boots in the live launch smoke. CloudTrail did not record fleet asset-bucket object operations; the ledger finding comes from the client and documented S3 behavior.

@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

I’m taking the current branch through the remaining review fixes. I’ll restrict effective SSM parameter reads on the gpubox instance role, protect resource classification tags, and fix the Haiku role classification test. The linked fcvm change for missing admission objects is already underway on #1979. I’ll push one fast-forward update to this branch and post the verification results here.

Limit post-launch tagging to LeaseUntil and scope launch tags to the instance's six gpubox keys. Deny parameter reads outside AWS public parameters and the two us-east-1 release pointers even with SSM ManagedInstanceCore attached. Cover both IAM boundaries and classify gpuboxes as fleet-account workloads in the Haiku test.
@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

Fixed the effective gpubox parameter grant, writable classification tags and the CI role inventory in 254b0a6. The instance retains SSM registration and command delivery. The launch role accepts the six tag keys used by gpubox.sh at launch and only LeaseUntil afterwards.

On ARM, 168 tests passed across six suites. Ruff, Terraform formatting, the lockfile-pinned backend-free Terraform validation and diff checks passed. The two new boundary tests failed against the preceding implementation.

I also ran 21 AWS IAM simulations against the rendered prospective policies plus AWS's live AmazonSSMManagedInstanceCore document. Singular and plural reads of the two release pointers and an AWS public parameter were allowed. Both actions on unrelated staging and production parameters were explicitly denied. SSM registration and control channels were allowed. A lease extension was allowed, while changing Purpose, Fleet or Owner, adding Purpose to the lease change, or applying a Fleet tag at launch was denied. Gpubox termination was allowed and fleet-host termination was denied.

The missing admission-object behavior is being fixed on the paired fcvm PR, using the existing prefix grant.

@codex review
@greptileai review
@coderabbitai review

@coderabbitai

coderabbitai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: 254b0a6b86

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ejc3
ejc3 merged commit 1b50ba9 into main Nov 21, 2026
6 checks passed
@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

Verified the deployed role from fcvm-arm using its own instance credentials and the ordinary operator script from fcvm#1979. The script launched one g4dn.metal Spot instance in us-west-2c, found SSM online after 102 seconds and reported eight VFIO GPUs, restored storage and the pinned Rust toolchain ready after 154 seconds. The remote command confirmed the dedicated fcvm-fleet-gpubox identity. A file copied to the GPU host and back through the ordinary operator commands matched byte for byte.

All 21 live IAM simulation cases passed. Two calls from the restricted ARM caller confirmed AWS denies the former DNS secret and an unrelated production parameter. On the GPU itself, both GetParameter and GetParameters could read the staging AMI release pointer and an AWS public parameter. Both calls returned AccessDenied for the production release pointer.

The operator requested termination of its only test instance, i-0c9e8a82eb7c0534b, at 01:35:52 UTC. After the host spent 11m43s shutting down, EC2 accepted a forced termination from the same caller under an additional restriction to this specific instance and its unique ownership tags. EC2 confirmed termination, closure of its one-time Spot request and deletion of its root volume, vol-05946aa6f7dd75040.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant