Repository navigation
fleet spend: charge an inventory that stays unknown at the launch cap - #1980
Conversation
The spend meter lists every region's instances every 30 s and reports their price as the burn rate. When listings failed it kept charging the last list it had. Before the first successful listing it reported nothing, so the ledger kept the rate stored across a restart, or none on a first start. An EC2 permission or API fault that lasted could leave the fleet unmetered past its ceiling, or metered on a list missing every launch since (raised during the prod activation, ejc3/aws#939). - A region with no successful listing for STALE_AFTER (300 s, ten passes), counted from the meter's start when it was never listed, is stale (stale_regions). While any region is stale, burn_report charges at least the fleet's launch cap, and the degraded reason says which regions. - launch_cap_usd_per_hour: in each host region the awake hosts plus the replacement a spot interruption notice launches, and with --ec2 aws each tile plus a relaunch in flight, all at the highest on-demand rate. For prod (one host region, two awake hosts, eight tiles) that is 19 x 7.824 = 148.66 USD/h, so an inventory that stays unknown reaches a 750 USD ceiling in about five hours instead of never. - Inside the bound nothing changes: a brief listing failure keeps the last list, and a start before the first listing keeps the stored rate. Tested (make test-unit, 7 tests): - a_region_unlisted_past_the_bound_is_charged_at_the_launch_cap; - regions_go_stale_after_the_bound_from_their_last_listing_or_the_start; - the_launch_cap_is_every_host_and_tile_the_fleet_may_run_at_the_highest_rate; - the integration test an_inventory_that_stays_unknown_is_charged_at_the_launch_cap_and_trips, where a meter whose listings always fail trips a 1 USD budget; - the existing spend and budget-trip tests. Two mutations each turn them red: burn_report ignoring stale regions fails the unit and integration tests, and the loop never marking a region stale fails the integration test. make clippy and make fmt are clean.
|
@codex review |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
☁️ Cloud World Model — Infrastructure Simulation ReportNo simulation was run and no credits were consumed. Link this installation, then re-run the check. Powered by Cloud World Model · Manage settings |
|
Caution Review failedThe pull request is closed. Walkthrough
Merge Risk:
|
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
NOT-A-DEFECT: CodeRabbit's comment is its "currently processing" notice, with no finding. Cloud World Model ran no simulation (its app is not linked to a workspace). Codex reviewed 4201121 and found nothing. |
The spend meter lists every region's instances every 30 s and reports their price as the burn rate. When listings failed, it kept charging the last list it had. Before the first successful listing it reported nothing, so the ledger kept the rate stored across a restart, or none on a first start. An EC2 permission or API fault that lasted could therefore leave the fleet unmetered past its ceiling, or metered on a list missing every launch since. The jumpbox raised this during prod activation (ejc3/aws#939).
STALE_AFTER(300 s, ten passes), counted from the meter's start when it was never listed. While any region is stale,burn_reportcharges at least the fleet's launch cap, and the degraded reason names the stale regions.launch_cap_usd_per_hourcounts, in each host region, the awake hosts plus the replacement a spot interruption notice launches, and with--ec2 awseach tile plus a relaunch in flight, all at the highest on-demand rate.Contract: an unknown inventory can't keep the fleet running past its daily ceiling.
Downstream impact: the staging and prod coordinators. It ships through the normal staging deploy and automatic prod promotion. This is also the first prod promotion, which the rollback drill needs (ejc3/aws#942).
Evidence:
make test-unitran 7 tests, 7 passed:a_region_unlisted_past_the_bound_is_charged_at_the_launch_capregions_go_stale_after_the_bound_from_their_last_listing_or_the_startthe_launch_cap_is_every_host_and_tile_the_fleet_may_run_at_the_highest_ratean_inventory_that_stays_unknown_is_charged_at_the_launch_cap_and_trips, where a meter whose listings always fail trips a 1 USD budgetTwo mutations each turn them red.
burn_reportignoring stale regions fails the unit test and the integration test, and a loop that never marks a region stale fails the integration test.make clippyandmake fmtare clean.Summary by CodeRabbit