Repository navigation
test: process-level fault-injection harness and the contract tests that pass today - #1473
duncanista wants to merge 4 commits into
Conversation
…act tests that pass today
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f0e4939146
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
This comment has been minimized.
This comment has been minimized.
duncanista
changed the title
- fault_injection_test: measure the /next budget from when the INVOKE was queued - harness: shutdown_cleanly asserts the exit came within the SHUTDOWN deadline
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 51f541581a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- oversized trace: wait for /next against the 10 s INVOKE deadline, and assert the rejected trace is never sent again after the rejection is lifted - lifecycle: read the intake before SHUTDOWN so the final flush cannot hide a missing end-of-invocation flush
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 399b90b8e6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…modes in blackhole
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3e32723992
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| .unwrap(); | ||
| deliver_runtime_done(&s.api, "req-1", true).await.unwrap(); | ||
| assert!( | ||
| s.api.wait_for_next_calls(2, secs(10)).await, |
There was a problem hiding this comment.
If either telemetry POST consumes part of the 10-second invocation window, this wait starts a fresh 10-second timer only after both POSTs finish, so /next may arrive after the event deadline and the test still passes. Capture the time when the INVOKE is queued and wait only for the remaining budget, or record /next arrival times; otherwise the PR summary's claim that this test asserts /next before the deadline is inaccurate.
AGENTS.md reference: AGENTS.md:L23-L23
Useful? React with 👍 / 👎.
| self.api.push_shutdown("spindown", SHUTDOWN_DEADLINE); | ||
| bc.wait_for_exit(bottlecap_process::EXIT_BUDGET).await |
There was a problem hiding this comment.
When callers use shutdown directly, an exit between the advertised 2-second SHUTDOWN deadline and EXIT_BUDGET still returns a successful status. The current direct callers in the 503, truncated-response, and port-conflict tests only assert that status, so those tests can pass after Lambda would have killed the process; this remains fresh evidence despite shutdown_cleanly now checking its elapsed time.
Useful? React with 👍 / 👎.
TL;DR: Tests that run the real
bottlecapbinary against a fake Lambda API and an intake that fails on purpose. This PR adds the harness and the 10 checks that pass today.Why: Our worst bugs (hangs past the invocation deadline, SIGKILL at shutdown, lost data on intake errors) show up only at the process level, so unit tests cannot catch them. Each later fix adds its own test here.
Overview
Jira: SVLS-9978
Stacked on #1472 (
jordan.gonzalez/test-fixtures/feature).Contract tests that run the real
bottlecapbinary against a fake Lambda control plane and aDatadog intake that fails on demand. Each test asserts what Lambda or the customer sees:
/nextbefore the deadline, exit on SHUTDOWN, no panic. Seebottlecap/tests/HARNESS.md.tests/common/: fake Extensions and Telemetry API, fault intake (blackhole, SYN blackhole,slow, 5xx, 429, close mid body, size limits), process wrapper with RSS and CPU, port guard.
maintoday: invoke then shutdown exitscleanly; a 503 storm does not hold
/next; an oversized trace is dropped and later tracesflow; an intake that closes mid body does not hold
/next; 4 config fields with a wrong typecost only that field; ports 8124 and 8126 held by another process.
bottlecap/.config/nextest.toml: the 4 binaries run one test at a time (fixed-portsgroup), because the binary binds fixed ports.
No production code and no Cargo change. The tests for the open bugs come in a later PR, each
ignored with the backlog item that fixes it.
Size: 20 core lines, 2184 lines in total.
Testing
cargo fmt --all -- --check: passcargo clippy --workspace --all-targets --features default -- -D warnings: passcargo clippy --workspace --all-targets --features default,test-mode -- -D warnings: passcargo nextest run --workspace: passcargo test --test <binary>for the 4 harness binaries: pass#[ignore], no backlog reference: pass