Skip to content

fleet spend: charge an inventory that stays unknown at the launch cap - #1980

Merged
ejc3 merged 1 commit into
mainfrom
fleet-spend-fail-closed
Nov 21, 2026
Merged

ejc3 merged 1 commit into
mainfrom
fleet-spend-fail-closed

Conversation

@ejc3

@ejc3 ejc3 commented Nov 21, 2026 •

Copy link
Copy Markdown
Owner

The spend meter lists every region's instances every 30 s and reports their price as the burn rate. When listings failed, it kept charging the last list it had. Before the first successful listing it reported nothing, so the ledger kept the rate stored across a restart, or none on a first start. An EC2 permission or API fault that lasted could therefore leave the fleet unmetered past its ceiling, or metered on a list missing every launch since. The jumpbox raised this during prod activation (ejc3/aws#939).

  • When a region is stale. It has had no successful listing for STALE_AFTER (300 s, ten passes), counted from the meter's start when it was never listed. While any region is stale, burn_report charges at least the fleet's launch cap, and the degraded reason names the stale regions.
  • The launch cap. launch_cap_usd_per_hour counts, in each host region, the awake hosts plus the replacement a spot interruption notice launches, and with --ec2 aws each tile plus a relaunch in flight, all at the highest on-demand rate.
    • For prod (one host region, two awake hosts, eight tiles) that is 19 × 7.824 = 148.66 USD/h.
    • So an inventory that stays unknown reaches the 750 USD ceiling in about five hours, instead of never.
  • Inside the bound nothing changes. A brief listing failure keeps the last list, and a start before the first listing keeps the stored rate.

Contract: an unknown inventory can't keep the fleet running past its daily ceiling.
Downstream impact: the staging and prod coordinators. It ships through the normal staging deploy and automatic prod promotion. This is also the first prod promotion, which the rollback drill needs (ejc3/aws#942).
Evidence: make test-unit ran 7 tests, 7 passed:

  • a_region_unlisted_past_the_bound_is_charged_at_the_launch_cap
  • regions_go_stale_after_the_bound_from_their_last_listing_or_the_start
  • the_launch_cap_is_every_host_and_tile_the_fleet_may_run_at_the_highest_rate
  • the integration test an_inventory_that_stays_unknown_is_charged_at_the_launch_cap_and_trips, where a meter whose listings always fail trips a 1 USD budget
  • the existing spend and budget-trip tests

Two mutations each turn them red. burn_report ignoring stale regions fails the unit test and the integration test, and a loop that never marks a region stale fails the integration test. make clippy and make fmt are clean.

Summary by CodeRabbit

  • Bug Fixes
    • Fleet spending reports now account for stale instance listings. If inventory remains unavailable beyond five minutes, spending is estimated using the greater of the known burn rate and the configured hourly launch cap.
    • If no inventory listing has succeeded before the stale threshold, the previous ledger rate remains in use until the threshold is reached.
    • The estimated spend can trigger the daily budget limit when inventory listings are stale.

The spend meter lists every region's instances every 30 s and reports
their price as the burn rate. When listings failed it kept charging the
last list it had. Before the first successful listing it reported
nothing, so the ledger kept the rate stored across a restart, or none on
a first start. An EC2 permission or API fault that lasted could leave the
fleet unmetered past its ceiling, or metered on a list missing every
launch since (raised during the prod activation, ejc3/aws#939).

- A region with no successful listing for STALE_AFTER (300 s, ten
  passes), counted from the meter's start when it was never listed, is
  stale (stale_regions). While any region is stale, burn_report charges at
  least the fleet's launch cap, and the degraded reason says which
  regions.
- launch_cap_usd_per_hour: in each host region the awake hosts plus the
  replacement a spot interruption notice launches, and with --ec2 aws
  each tile plus a relaunch in flight, all at the highest on-demand rate.
  For prod (one host region, two awake hosts, eight tiles) that is
  19 x 7.824 = 148.66 USD/h, so an inventory that stays unknown reaches
  a 750 USD ceiling in about five hours instead of never.
- Inside the bound nothing changes: a brief listing failure keeps the
  last list, and a start before the first listing keeps the stored rate.

Tested (make test-unit, 7 tests):
- a_region_unlisted_past_the_bound_is_charged_at_the_launch_cap;
- regions_go_stale_after_the_bound_from_their_last_listing_or_the_start;
- the_launch_cap_is_every_host_and_tile_the_fleet_may_run_at_the_highest_rate;
- the integration test
  an_inventory_that_stays_unknown_is_charged_at_the_launch_cap_and_trips,
  where a meter whose listings always fail trips a 1 USD budget;
- the existing spend and budget-trip tests.
Two mutations each turn them red: burn_report ignoring stale regions
fails the unit and integration tests, and the loop never marking a
region stale fails the integration test. make clippy and make fmt are
clean.
@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-21T04:33:31.133022Z 4201121 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@cloud-world-model

cloud-world-model Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

☁️ Cloud World Model — Infrastructure Simulation Report

Matching GitHub Check

⚠️ CWM received this pull request, but cannot simulate it because the GitHub App installation is not linked to a CWM workspace.

No simulation was run and no credits were consumed. Link this installation, then re-run the check.


Powered by Cloud World Model · Manage settings

@coderabbitai

coderabbitai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Explain Change →

Caution

Review failed

The pull request is closed.

Walkthrough

The spend meter now tracks instance-listing freshness by region. When a region is stale, the reported burn rate is at least the calculated hourly launch cap. Tests cover stale listings, cap calculations, and a budget trip after listing failures.

Changes: Spend meter stale-inventory handling

Layer / File(s) Summary
Staleness and launch-cap policy
src/fleet/coord/spend.rs
MeterConfig now includes a stale threshold and hourly launch cap. burn_report applies the cap as a minimum when any region is stale and includes a stale-listing reason.
Listing freshness in meter runs
src/fleet/coord/spend.rs
Meter runs track successful listing times, determine stale regions, and pass them with the cap to burn_report.
Staleness and cap tests
src/fleet/coord/spend.rs, tests/test_fleet_coord.rs
Tests cover stale and never-listed regions, launch-cap calculations, and a budget trip when instance listing fails.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant FC as Fleet coordinator
  participant Listing as Instance listing
  participant Meter as Spend meter
  participant Report as burn_report
  FC->>Meter: Start with regions and launch cap
  Meter->>Listing: List instances by region
  Listing-->>Meter: Instances or listing error
  Meter->>Meter: Calculate stale regions
  Meter->>Report: Inventory, prices, stale regions, and cap
  Report-->>Meter: Burn rate and stale-listing reason
Loading

Possibly related PRs

  • ejc3/fcvm#1960: Adds the spend meter and UTC-day ledger whose unknown-inventory charging behavior this change updates.
Merge Risk: Moderate · up to 42011

This change makes the spend meter charge at least the fleet's launch cap when instance inventory stays unknown, and that is a clear improvement. The cap can still be low in edge cases: failed instance terminations that leave extra billed tile instances, and several hosts draining at once in a region. Under stale inventory the budget trip could then come later than intended. Consider tightening the cap before merge, or accept the residual risk knowingly.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: 42011210ac

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ejc3

ejc3 commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

NOT-A-DEFECT: CodeRabbit's comment is its "currently processing" notice, with no finding. Cloud World Model ran no simulation (its app is not linked to a workspace). Codex reviewed 4201121 and found nothing.

@ejc3
ejc3 merged commit e3c1974 into main Nov 21, 2026
7 of 17 checks passed
@ejc3
ejc3 deleted the fleet-spend-fail-closed branch November 21, 2026 04:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant