fix(scale-down): clean EC2 runners incrementally and isolate failures

[guicaulada]

Scale-down waits for the complete EC2 inventory before removing active runners. With large fleets, listing can fail or time out before any cleanup happens. Orphan cleanup can also terminate a runner whose registration ID tag is missing without checking GitHub, or stop processing other runners after one failure.

Process EC2 inventory in bounded pages of up to 100 instances and clean each page before requesting the next. Look up registrations by valid GitHub runner ID or exact full runner name instead of listing the entire organization/repository. Verify orphan registrations, remove false-positive orphan markers, preserve unverifiable runners, and isolate failures per runner. Existing busy checks, bypass-removal, boot grace, and idle detection remain in place.

Cleanup is stateless. Every invocation starts from current inventory; successfully terminated instances are no longer listed on subsequent runs. Pagination tokens exist only within the invocation. A deadline guard stops starting new work with ten seconds remaining, and a later listing failure retains earlier cleanup. Busy, retained, or failed items can be checked again on the next invocation.

The idle allowance is shared across pages within an invocation and recalculated on the next run. Eviction ordering applies within each page; strict global oldest/newest ordering requires collecting the whole fleet. Repeated scheduled scans reconcile pagination shifts as instances disappear. Custom compute providers without the optional paging capability retain their existing list-based path, including a complete owner-listing fallback when GitHub identity fields are absent. Only successful complete listings are cached.

Both Terraform scale-down paths supply the configured runner-name prefix. EC2 trusts an instance prefix tag only when it matches that configuration. Missing or mismatched tags without a valid GitHub ID cause cleanup to retain the instance; affected instances need their tags or registration IDs corrected before automatic cleanup. An exact-name miss without an ID also retains the instance because a custom script may have registered a different name. Unverifiable identity is reported at error level before unnecessary authentication when possible.

Validation: 395 control-plane tests, 312 compute-provider tests, and seven mocked Terraform tests pass, covering cleanup before later-page failure, fresh scans of reduced inventory, cross-page idle allowance, deadline handling, direct GitHub ID lookup, and isolated failures. Runtime TypeScript, ESLint, Prettier, Lambda bundle build, and diff checks pass. The compute-provider package's separate TypeScript check has existing Vitest configuration typing errors, reproduced with unchanged compute-provider code. No AWS deployment or live termination was performed.

Independent companion PRs: #5464 (SSM cleanup) and #5465 (opt-in registration reconciliation).

Operational logs report per-page counts and final scan totals, successful termination counts, scan completeness, deadline exits, and orphan termination. These observations replace the old full-fleet before/after count on the EC2 path; dashboard migration and the additional per-instance GitHub API cost are documented. EC2 paging honors request cancellation before sending a request.

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

[Copilot]

Copilot review overview

🟡 Changes recommended

Runner-name derivation can target the wrong JIT registration, and existing custom providers can silently lose scale-down behavior.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2

Open (2)

• Preserve list-only provider fallback for scale-down · New
• Prevent runner name prefix overrides from orphaning JIT runners · New

What changed in this PR

Adds incremental EC2 scale-down processing with bounded pagination, direct GitHub runner lookup, deadline handling, and per-runner failure isolation.

Changes:

• Processes EC2 runners page-by-page while sharing idle allowance.
• Verifies runners by GitHub ID or exact name before cleanup.
• Updates tests and scale-down documentation.

File	Description
modules/​runners/​scale-down-state-diagram.md	Documents incremental cleanup flow.
modules/​orchestration-providers/​webhook/​scale-down-state-diagram.md	Mirrors updated scale-down documentation.
lambdas/​libs/​compute-providers/​registry.test.ts	Relaxes capability registry assertion.
lambdas/​libs/​compute-providers/​core/​index.ts	Adds runner names and optional paging.
lambdas/​libs/​compute-providers/​aws/​ec2/​src/​runners.ts	Implements bounded EC2 pages and runner names.
lambdas/​libs/​compute-providers/​aws/​ec2/​src/​runners.test.ts	Tests paging and name extraction.
lambdas/​libs/​compute-providers/​aws/​ec2/​src/​control-plane/​scale-down.ts	Exposes EC2 paging capability.
lambdas/​functions/​control-plane/​src/​test/​compute-provider-contracts/​scale-down.ts	Updates orphan cleanup contract.
lambdas/​functions/​control-plane/​src/​scale-runners/​scale-down.ts	Implements incremental, isolated cleanup.
lambdas/​functions/​control-plane/​src/​scale-runners/​scale-down.test.ts	Covers paging, deadlines, and failures.
lambdas/​functions/​control-plane/​src/​scale-runners/​scale-down-contract.test.ts	Adds GitHub authentication mocks.
lambdas/​functions/​control-plane/​src/​lambda.ts	Supplies Lambda remaining execution time.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Brend-Smits]

Tested on a sandbox deployment (multi-runner example, non-ephemeral Linux config). Normal scale-down worked. Removing the registration-ID tag exercised the exact-name lookup, which de-registered and terminated correctly. Setting a mismatched prefix tag kept the instance with a warning, as described. A few observations:

1. Fleet-count logs are gone on the EC2 path. Found: 'N' active EC2 runners before/after clean-up. is now only logged on the legacy list path, so EC2 deployments no longer emit it (confirmed after deploy). Anyone with log-based dashboards or alerts on this line will lose them. Could we log a total at the end of the paged run (or per page)?
2. Silent exits. When the time limit is hit, the run returns without any log. On the name-lookup path, an orphan is terminated without an info-level log, because the old Terminating orphan runner message was removed. An info log for each would help operators.
3. GitHub API volume. Before, there was one paginated list per owner, cached for the run. Now there is one call per instance, before any busy check. On large fleets with a 1–5 minute schedule this uses the app's rate limit much faster. It may be worth mentioning in docs/rate-limits-and-tuning.md.
4. MaxResults: 10 means about 100 DescribeInstances calls per run for 1,000 instances. These share EC2's throttling limit with scale-up and pool. Would a larger page (around 100) or a setting be better?
5. Exact-name lookup is stricter than the old endsWith(instanceId) match. A runner registered under a custom name (custom start script) that also has no ID tag is now not found. It gets marked orphan and then terminated, and the orphan path does no busy check. Please document the naming requirement, or avoid treating "not found by name" as proof of absence.
6. Kept instances only produce a warning. Instances that can't be verified stay up indefinitely with a warning every run. An error-level log or a metric would make this cost leak visible. Also, the GitHub client and installation token are created before the identity check that throws; checking first saves the calls.
7. Nit: listPage skips the runWithRequestSignal pre-check that the other EC2 calls use.

[guicaulada]

Thanks for testing this in a sandbox and writing up the observations. I addressed all seven in f3c63c5 and rebased onto current main:

1. Added info-level per-page inventory counts and a final scan summary with pages, scanned runners, successful terminations, completeness, and deadline status. These are scan observations rather than an atomic fleet census; the docs explain how to migrate dashboards from the old before/after lines. A later listing failure still emits the partial summary.
2. Added info logs for deadline exits and orphan termination.
3. Documented the per-instance GitHub API cost and scheduling tradeoff in docs/rate-limits-and-tuning.md, including an example of 1,000 runners on a five-minute schedule.
4. Increased the EC2 inventory page size from 10 to 100 while retaining per-runner processing and deadline checks.
5. Changed the safety behavior: an exact-name miss without a registration ID no longer establishes absence on the paged path. The instance is retained even if already marked orphaned. This protects custom-name runners, including busy registrations we cannot identify. The naming/ID requirement and remediation are documented; this deliberately also retains genuinely absent registrations when their identity cannot be proved safely.
6. Unverifiable identities now emit an error-level UNVERIFIABLE_RUNNER code suitable for alerting. Missing identity is checked before creating GitHub authentication/clients.
7. Wrapped listPage in runWithRequestSignal and added a regression proving an already-aborted request never sends DescribeInstances.