[FumingPower3925]

Status: ready for review. loadgen v1.4.14 is released and this PR pins v1.4.14. Squash merge only.

The tag exists and points at the commit this PR tested. gh api repos/goceleris/loadgen/git/refs/tags/v1.4.14 gives commit 9f8993eb24b44e0d053b55a3796211794120c576, which is loadgen main 9f8993e (chore(release): v1.4.14 (#101)). check-tag.sh (finishing step 0) exits 0: git's peeled tag and Go's Origin.Hash are both 9f8993e (finish/check-tag.log, 14:09Z).

• The org admin published the release at 13:55:20Z with gh release create v1.4.14 --target 9f8993e. That is the release: published path, the same way v1.4.13 was cut. It is not a re-run of the failed dispatch 36319142903.
• The release workflow ran from that event: run 36324072983 succeeded. The version stamps agree, the module verifies, and it built and attached 4 binaries (darwin/amd64, darwin/arm64, linux/amd64, linux/arm64). It also requested pkg.go.dev indexing. "Create tag and release" was skipped because the release already existed.
• proxy.golang.org serves v1.4.14 with Origin.Hash 9f8993e and Ref refs/tags/v1.4.14 (finish/release-facts.log).

The pin is v1.4.14. Commit d434126 ran go get github.com/goceleris/loadgen@v1.4.14 && go mod tidy in the root. It changes only the root go.mod require line and the two loadgen lines of go.sum.

• No pseudo-version remains. Before the change, a grep over every go.mod/go.sum finds v1.4.14-0.20260927122822-9f8993eb24b4 (root go.mod:8, go.sum:18-19; that is the positive control). After it, the same grep, a grep for any loadgen pseudo-version and a grep of the whole tree all find nothing (finish/pseudo-grep-{before,after}.log).
• The code is the one the earlier suites and CI tested. diff -r of the module-cache trees loadgen@v1.4.14 and loadgen@v1.4.14-0.20260927122822-9f8993eb24b4 is empty (68 files). As a control, v1.4.13 vs v1.4.14 differs in 33 files (finish/source-identity.log).
• go mod tidy -diff is clean in all 23 modules. All 14 modules that resolve loadgen now resolve v1.4.14: the root, the 12 servers/* adapters and fieldguard. The 9 refapp/debugvars modules do not depend on it. go mod verify passes (finish/bump-proof-v1.4.14.txt, finish/tidy-diff-all.log).
• benchmark_config.loadgen_version and the split index read the root require line (the three sites under "Why the pseudo-version could not merge"). A run from main after this merge therefore publishes v1.4.14.

Rebased on main 02e4172 (#412 at 3263830, then #411 at 02e4172). The rebase was clean.

• The only file both sides touch is servers/servers.go: this PR changes one comment line there and feat(bench): make the SEND_ZC on/off A/B answerable: SUT /debug/vars, iowait-free CPU, per-scenario engine egress, per-column SUT env (celeris#585) #411 adds 14 lines elsewhere in the file.
• feat(bench): make the SEND_ZC on/off A/B answerable: SUT /debug/vars, iowait-free CPU, per-scenario engine egress, per-column SUT env (celeris#585) #411 moved report.SchemaVersion to 5.17. This PR touches no schema constant and no test that pins one, so RULE 48 is not triggered.
• RULE 81: after the rebase, go build ./..., go vet ./..., mage -compile and go vet -tags mage . all pass (finish/build-after-rebase.log).
• Commits: a4b2d0f (the loadgen bump and the budget change, formerly 991d13d), 53f0ed8 (review round 1, formerly 5c63890) and d434126 (the tag pin).

Merge with squash. The repo also allows rebase merges (ruleset 16196082). A rebase merge would put a4b2d0f and 53f0ed8, which pin the pseudo-version, on main as separate commits.

The release-order guard did not happen in the planned order. "Release-order gate" below says so plainly and lists what protects main until this PR merges.

Why the pseudo-version could not merge. Every place that records the loadgen version reads the root go.mod require line:

• benchmark_config.loadgen_version from the mage ingest: mage_bench.go:1859 (goModRequireVersion);
• benchmark_config.loadgen_version from the runner: cmd/runner/main.go:1925 (modRequireVersion, which reads go.mod at :1980);
• the split index's loadgen_version: mage_publish.go:360 → report/split.go:98.

What changes

1. loadgen v1.4.13 → v1.4.14 (9f8993e) in every module that resolves it.
   • The root go.mod is the only module that requires loadgen. The 13 modules that replace the root (servers/* and validation/properties/fieldguard) resolve it through the root. The 9 refapp/debugvars modules do not depend on it.
   • go list -m per module is in the evidence (bump-proof-*.txt; at the tag pin, finish/bump-proof-v1.4.14.txt).
   • loadgen v1.4.14 requires golang.org/x/net v0.59.0, so MVS moves x/net from 0.58.0 to 0.59.0 in the go.mod of the root and of the adapter modules echo, gin, hertz and iris.
   • go mod tidy in the other 18 modules changes nothing: celeris, the refapps and debugvars were already on 0.59.0, and chi, fasthttp, fiber, gnet, gorilla_ws, nbio, stdhttp and fieldguard do not use x/net. In the module graphs of the seven adapters among them x/net moves too (through the root), but none builds an x/net package. hertz's graph also moves x/term 0.45.0 → 0.46.0 (x/net 0.59.0 requires it), and hertz builds no x/term package (round1/graph-diff.log).
   • The root builds one x/net package, x/net/http2/hpack, imported by loadgen's client. Its source is identical in 0.58.0 and 0.59.0.
   • No adapter serves HTTP/2 through x/net, so no server code changes at this re-pin.
     • echo, gin, chi, iris and stdhttp serve h2c with net/http's own HTTP/2 (Server.Protocols with SetUnencryptedHTTP2). That code is net/http/internal/http2 and comes with the Go toolchain, not from x/net. hertz-h2 serves through hertz-contrib/http2.
     • x/net is in the echo and gin builds only because the echo and gin libraries import x/net/http2/h2c for StartH2CServer and UseH2C. The adapters never call either, so the linker keeps only those packages' init functions and variables: 0 x/net/http2.(*serverConn) symbols, 85 net/http/internal/http2.(*serverConn) ones (round1/xnet-symbols.log).
     • Proof (round1/servers-xnet-proof.log): echo, gin, hertz and iris built reproducibly (linux/amd64, -trimpath -buildvcs=false, empty build ID, no symbol table or DWARF) at main cd8dedf and at this head. Two strings must differ and have equal length: the x/net line of the build info and the golang.org/x/net@v0.5x.0/ file names. With those rewritten, .text, .rodata, .data, .noptrdata, .go.type and .go.func are byte-identical for echo, gin and hertz. Only the pc-to-line table (and the module data's offsets into it) differs, because 0.59.0 adds comment lines above linked init functions.
     • iris differs in one function: x/net/html's unescapeEntity, where 0.59.0 writes an if as min() with the same result. Built with 0.59.0 plus 0.58.0's html/escape.go, iris's code and data match the 0.58.0 build exactly. iris links x/net/html only through bluemonday, which iris calls only from WriteMarkdown with Sanitize; the adapter never renders Markdown.
     • Controls: the head built twice is byte-identical; a one-constant adapter change (200 → 201) changes .text; a change in linked x/net code (h2c's init) changes code and data; a change in x/net's HTTP/2 server (serverConn.serve) leaves echo byte-identical.
   • Round 1 also fixes the stale "h2c.NewHandler-wrapped" comments in servers/{chi,echo,gin,iris}/server.go and servers/servers.go, which had described the h2c mode wrongly. Comments only: the four adapter binaries are byte-identical before and after (round1/comment-only-proof.log).
2. churn-close uses the default 5% error budget (probatorium#424 §1). scenarios/static.go drops ErrBudget: 0.5, and the comment says why. The ErrorBudgeter and ErrBudget docs now say that no scenario raises its budget.
3. The H2 comment is corrected (churn-close after loadgen#87: change ErrBudget 0.5 with the loadgen pin bump; lithium's churn-close becomes real churn; latency and teardown discontinuity #424 comment, fix(h2client): real status and body bytes, no lifetime deadline, redial dead connections (#85, #86, #88, #89) loadgen#91 "What changes in probatorium's published numbers").
   • The runner sets Workers = Connections (buildCellConfig), and loadgen multiplies HTTP/2 workers by 4 (bench.go, case cfg.HTTP2).
   • So each H2 cell runs 128 workers on 32 connections: 4 streams in flight per connection, not "default 100".
4. Tests.
   • TestErrorBudgets: every registered scenario gets the default budget.
   • TestChurnCloseErrorBudget: churn-close's budget is checked against measured ratios.
   • TestClassifyCompletedCell gets two new rows, the measured half-failing cell (suspect) and the fixed client's clean cell (ok). Its churn-close rows now take their budget from the registered scenario instead of a copy of 0.5.
   • TestAggregateCarriesLoadgenResultVerbatim is the RULE 93 check (below).
   • Round 1: with churn-close on the default, no registered scenario raises its budget, so every row above ran at 0.05 and no test exercised a non-default budget. The override stays, as churn-close after loadgen#87: change ErrBudget 0.5 with the loadgen pin bump; lithium's churn-close becomes real churn; latency and teardown discontinuity #424's fallback for churn-close. TestErrorBudgets now checks that a scenario declaring ErrBudget: 0.2 gets 0.2. TestClassifyCompletedCell gets two rows with a 0.5 budget: a 0.3 ratio is ok, and over the budget the message and the connect-class overage attribution use 0.5.

Why the churn-close budget is 0.05

#424 names the choice: the default budget (0.05), or a churn-specific budget set from the first post-fix run's genuine refused-dial rates. No post-fix run exists, and the data we have puts the genuine rate at about 0:

• The 0.5 was calibrated against a loadgen artifact, not against the servers. Up to v1.4.13, loadgen wrote the next request into a connection the server had closed as asked, so every close-mode success was followed by one counted EOF error (-close mode counts one error per successful request: the reused slot's read EOF is reported as a failure (errors/(errors+requests) = 0.5) loadgen#87).
  • fix(h1client): never reuse a connection that is closing, so -close stops counting one error per request (#87) loadgen#90's "Measured" table (net/http, close mode, PoolSize=1): main a89f02d had 26,753 requests and 26,746 errors (ratio 0.49993). The fixed client (c913657, shipped in v1.4.14) had 31,419 requests and 0 errors. celeris std shows the same pattern: 28,466 / 28,459, then 34,792 / 0.
  • In docs ae6b949, 42-44 of 45 x86 churn-close cells per dataset are suspect with a delta of 100.000-100.006%. arm64 20260829 has 9 more coin-flip ok cells with a delta of about 100.
• Refused dials were 0 already. connect_errors["churn-close"] is omitted (0) in every published dataset (churn-close after loadgen#87: change ErrBudget 0.5 with the loadgen pin bump; lithium's churn-close becomes real churn; latency and teardown discontinuity #424 §3, r2/impact/published-churn-close-all.txt). So the "refused dials are inherent to churn" premise of the 0.5 never showed up as errors.
• Accept-backlog overflow does not produce errors either. Linux drops the SYN when the accept queue is full, and the client resends it after 1 s. That time lands in a recorded success's latency, not in the error count (churn-close after loadgen#87: change ErrBudget 0.5 with the loadgen pin bump; lithium's churn-close becomes real churn; latency and teardown discontinuity #424 §3).
• Under 0.5, a server that fails half its churn attempts would publish as ok once the artifact is gone. That is the case the new tests pin.
• The first post-bump run is the check. If healthy servers there show genuine refused-dial rates above 5%, a churn-specific budget can be set from that data, as churn-close after loadgen#87: change ErrBudget 0.5 with the loadgen pin bump; lithium's churn-close becomes real churn; latency and teardown discontinuity #424 allows.

Failing-first (evidence control-*.log):

control	result
A: the change as written	the budget tests and the carry test PASS
B: ErrBudget: 0.5 restored	FAIL. TestErrorBudgets: churn-close = 0.5. TestChurnCloseErrorBudget: the 0.49993 row and the 0.10 row are not suspect. TestClassifyCompletedCell/churn-close_half-failing_cell_is_suspect: ErrMsg ""
C: carry test built against loadgen v1.4.13 (main's go.mod via -modfile)	build FAILS: unknown field CloseAborts in struct literal of type loadgen.Result
D: ingest re-encodes result through a typed mirror without close_aborts	FAIL: close_aborts = "", want 4499990. The other fields survive

Round 1, the non-default budget path (round1/mutants.log; go test ./cmd/runner/ ./scenarios/ in a copy of each tree, the worktree never mutated):

mutant	main cd8dedf	round 0 (15d55bc)	round 1 (5c63890)
M1: the gate compares with the default (main.go:1394)	killed	survives	killed
M2: StaticScenario.ErrorBudget ignores ErrBudget (static.go:67)	killed	survives	killed
M3: the overage allowance uses the default (main.go:1401)	survives	survives	killed
M5: the message prints the default (main.go:1396)	survives	survives	killed
M4: the runner hands the classifier the default (main.go:1225)	survives	survives	survives (see "Not in this PR")

RULE 93 (the harness catch-up): no probatorium code has to carry anything, and a test now proves it

What v1.4.14 changes in loadgen.Result. git diff v1.4.13 9f8993e -- results.go adds exactly one field: CloseAborts uint64 json:"close_aborts,omitempty" (goceleris/loadgen#90). TimeseriesPoint and every other type are unchanged. #91's status accounting (#85) and body bytes (#86) change the values of existing fields (requests, errors, throughput_bps), not the shape.

Where the new field goes:

• Per-cell artifacts: already carried.
  • The runner stores the whole *loadgen.Result in its per-cell JSON (cmd/runner cellResultFile.Result).
  • The ingest keeps result as raw JSON (runnerCellFile.Result json.RawMessage → cellRecord.Loadgen, mage_bench.go), so raw/<host>.json carries every field loadgen emits.
  • The Benchmark Tier uploads all of results/ (benchmark-tier.yml, path: results/).
  • TestAggregateCarriesLoadgenResultVerbatim pins this; controls C and D above show it can fail. churn-close after loadgen#87: change ErrBudget 0.5 with the loadgen pin bump; lithium's churn-close becomes real churn; latency and teardown discontinuity #424's post-bump check reads close_aborts from there: about requests on lithium, about 0 elsewhere.
• The published summary (summary.json, schema 5.17 since feat(bench): make the SEND_ZC on/off A/B answerable: SUT /debug/vars, iowait-free CPU, per-scenario engine egress, per-column SUT env (celeris#585) #411): not added, on purpose.
  • churn-close after loadgen#87: change ErrBudget 0.5 with the loadgen pin bump; lithium's churn-close becomes real churn; latency and teardown discontinuity #424 does not ask for it ("not published unless probatorium maps it").
  • Its sibling diagnostic counters are not published either: dial_retries and recvq_high appear nowhere in report/.
  • report.Aggregate reads only named fields, and none changed shape.
  • So SchemaVersion does not move, and RULE 48 is not triggered. After the rebase onto feat(bench): make the SEND_ZC on/off A/B answerable: SUT /debug/vars, iowait-free CPU, per-scenario engine egress, per-column SUT env (celeris#585) #411 (5.17), this PR still touches no schema constant and no test that pins one.
• The fix(refapp): publish every EngineMetrics field and guard the projection by reflection #389/fix(validation): carry every published engine counter to the artifacts, and guard each hop against the published key set (#391) #394 field guards and the tallies cannot see it. They guard celeris EngineMetrics → debugvars → Snapshot. No validation/ package imports loadgen: git grep -l goceleris/loadgen lists only cmd/loadgen, cmd/runner, interleave (test), the mage files, report and scenarios. tools/stresstally tallies go test output.
• No strict decoder rejects the new key. There is no DisallowUnknownFields in the repo. report/schema_v5.json validates the published document, which does not embed the loadgen result.

Which published fields change: the churn-close and H2 series restart at this re-pin

Sources: #424 §2-§4 and its comment, goceleris/loadgen#90 "Impact on what probatorium publishes", and goceleris/loadgen#91 "What changes in probatorium's published numbers".

churn-close, on the 42-44 servers per dataset that close as asked:

• sent_vs_handled_delta_pct falls from about 100 to the genuine rate. The key is omitted at 0.
• cell_statuses moves from suspect to ok.
• The rated sweep now runs, so latency_at_slo["churn-close"] and rated_mode_p99_at_target_rps["churn-close"] appear. Rated profiles run one more churn-close sweep per server.
• saturation_mode_rps and loadgen_cpu_p95["churn-close"] change.
• saturation_mode_rps can also come out low on a server whose full accept queue drops SYNs. loadgen's H1 dial cannot be cancelled, so a SYN retransmission that starts near the end of the window stretches the elapsed time: a 3 s stall after a 30 s window reads about 10% low (h1client: a dial in progress cannot be cancelled, so a SYN retransmission at the end of a run stretches Run's elapsed time and deflates requests_per_sec loadgen#95, open). After fix(publish): BenchAndValidate reads VALIDATE_TARGET to gate ValidateDiff #90 every churn-close request dials, so churn-close is where this shows.
• timeseries.json.gz for churn-close: the per-second errors falls from about rps to about 0.
• Latency now includes the TCP connect and any SYN retransmission: hdr_histogram_b64, the timeseries p99_ms, and the new rated fields.
  • A mode at 1 s or more is accept-backlog drops, or a PAWS-rejected SYN on a server TIME_WAIT. It is not a server latency regression.
  • The server-side teardown changes too: the server keeps one TIME_WAIT per connection, and a server that closes more than 50 ms late gets a client RST.
• connect_errors keeps its meaning.
• throughput_bps scales with successes. Its only published use, network_bound, cannot fire on this small body.

lithium's churn-close becomes real churn. lithium ignores Connection: close, so its published cell has been a keep-alive measurement: 458,510-481,817 rps on x86, 196,704 on arm64. It falls to churn values in saturation_mode_rps, latency_at_slo and rated_mode_p99_at_target_rps. This is not a lithium regression. resources["churn-close"].fd_hwm rises by about rate × 50 ms.

H2: get-json-h2 and post-4k-h2, 32 cells per arch (goceleris/loadgen#91):

• feat(validation): Tier 1 SSE kill-mid-stream slice — broker cleanup oracle #85: a 4xx/5xx, a 1xx-only response or an unreadable :status is now an error. That moves requests, errors, sent_vs_handled_delta_pct and saturation_mode_rps, and can change the cell status. feat(diff): persist ValidateDiff findings as diff.txt + diff.json #91 expects no published change, since every adapter answers 200.
• feat(validation): emit v5 validate-results.json + PublishValidate target #86: throughput_bps changes only for a server that splits bodies or ends them with an empty END_STREAM DATA frame. That is fastapi-h2, which counted 0 bytes. The only published use is network_bound, and no H2 cell comes near it.
• feat(ci): wire ValidateDiff + PublishValidate into validate.yml + soak.yml #89: after a connection dies, workers redial instead of stalling. saturation_mode_rps rises on servers that end connections, the latency of the redialing request carries the dial, failed redials show in connect_errors, and errors per second during an outage rise about 4×.
• fastapi-h2, both rows and both arches: loadgen_cpu_p95 falls (a leaked writeLoop per GOAWAY), and the latency tail falls (no 5-10 ms sleep per redial).
• The slot fix: on 16 of the 64 H2 cells, the hdr_histogram_b64 upper tail (p99.99, max) can rise. Responses that were missing come back, and misattributed ones go away.
• No H2 row changes on the server side: no adapter serves HTTP/2 through x/net (above). Every H2 shift at this re-pin comes from loadgen.
• latency_at_slo and rated_mode_p99_at_target_rps exist for H2 rows only while BenchTier rates every clean cell (~770 of 813) at 30 s passes, not the model's 388 rated cells: this is the whole gap between the 44.5 h / 26.6 h projections and the measured ~69 h / ~53 h runs #418 stands.

Keep-alive H1: small. Only cells that already carry loadgen errors are affected: 15 driver-pg-write cells and one get-simple-1024c cell in ae6b949.

• Error counts go down where main counted one fault twice.
• One connect per connection-level error moves into a recorded success.
• gin-h2 arm64 (0.0845) and fiber-h1 x86 (0.0533) could drop under the 5% budget if their errors were double-counted.

Do not compare these series across this commit.

• diffBenchResults (BenchSince) and the markdown "Regression vs baseline" section do not read loadgen_version.
• Neither runs today: no workflow calls BenchSince, and nothing sets report.Meta.BaselinePath (only tests do). So the first post-bump run cannot trip or pass either comparison on a measurement change.
• The comparison readers actually see is the docs dashboard's over-time charts: the churn-close errors and p99_ms series, and the H2 rows. They need the discontinuity note from churn-close after loadgen#87: change ErrBudget 0.5 with the loadgen pin bump; lithium's churn-close becomes real churn; latency and teardown discontinuity #424's second to-do (docs).
• The version-aware comparison proposed in churn-close after loadgen#87: change ErrBudget 0.5 with the loadgen pin bump; lithium's churn-close becomes real churn; latency and teardown discontinuity #424's comment is still open there. It is needed before anyone wires BenchSince into CI.

Release-order gate

The tag reached the world before this PR merged, and before either guard reached main. Round 1 said a guard had to land on probatorium main before v1.4.14 existed (#424, goceleris/loadgen#97 item 2). The guard was either (a) a dependabot ignore for loadgen or (b) the budget change alone.

• Neither landed. The org admin published v1.4.14 at 13:55:20Z.
• main (02e4172) still pins loadgen v1.4.13, keeps churn-close at ErrBudget: 0.5 and has no ignore for loadgen.
• Since 13:55Z, dependabot can propose loadgen v1.4.14 against that main. Its PR would pair the fixed client with the old 0.5 budget, the combination churn-close after loadgen#87: change ErrBudget 0.5 with the loadgen pin bump; lithium's churn-close becomes real churn; latency and teardown discontinuity #424 exists to prevent.

What protects main until this PR merges:

• No such PR exists. At 14:14Z the open dependabot PRs are deps: bump github.com/lesismal/nbio from 1.6.12 to 1.7.0 in /servers/nbio in the all-go-deps group across 1 directory #413 (nbio, servers/nbio only), ci: bump the all-actions group across 2 directories with 1 update #408 (actions), deps(maven): bump the all-maven-deps group across 1 directory with 2 updates #407 (maven), deps(js): bump uWebSockets.js from v20.70.0 to v20.71.0 in /servers/uws #406 and deps(js): bump fastify from 5.12.4 to 5.12.5 in /servers/fastify in the all-js-deps group across 1 directory #405 (js). None of their diffs has a loadgen line (finish/dependabot-gate-check.txt).
• Nothing merges a dependabot PR on its own. No workflow here enables auto-merge. The Main ruleset (16196082) requires one approving code-owner review and approval of the last push. So a dependabot loadgen PR lands only if a person approves it, and that person is the whole guard.
• CI would not catch it. main's tests pin 0.5, so they pass with the new client. On main 02e4172 plus only go get github.com/goceleris/loadgen@v1.4.14 && go mod tidy, the root builds and vets, and go test ./scenarios/ ./cmd/runner/ (including main's TestErrorBudgets) and the mage-tagged tests pass (finish/main-bump-sim.log). A green dependabot PR would prove nothing here.
• The window closes when this PR merges. Once main pins v1.4.14, dependabot has no newer loadgen to propose until v1.4.15. If a dependabot loadgen PR opens before then, this PR supersedes it and it must not merge. I will comment that on it.

Proposed for later releases: an ignore for loadgen in .github/dependabot.yml (not added here). In the package-ecosystem: gomod entry:

    ignore:
      # loadgen is the measuring instrument: a bump changes what the bench
      # publishes, so it lands only in a hand-made re-pin PR with its harness
      # catch-up (RULE 93) and any budget change (probatorium#424).
      - dependency-name: "github.com/goceleris/loadgen"

• The YAML is trivial, but the policy is not: it removes dependabot's view of loadgen for good.
• It can no longer guard this release, because the tag exists. Merging this PR closes the window just as well.
• Inside this PR it would only take effect after the bump, for v1.4.15 and later. Whether loadgen gets it is the maintainer's call.

Verification

• Finishing suites at d434126 (rebased on 02e4172 and pinned to the v1.4.14 tag; laptop lock timing=none, -p 1, go 1.27.1; finish/suite-v1414-*.log):
  • touched packages: go test -count=1 -p 1 -race -v ./scenarios/ ./cmd/runner/ ./servers/: 108 PASS / 0 FAIL / 3 SKIP (the rust adapter binaries are not built on the Mac)
  • mage layer: go test -count=1 -p 1 -race -v -tags mage .: 129 PASS / 0 FAIL / 1 SKIP (114 at round 1; the new ones come with feat(bench): make the SEND_ZC on/off A/B answerable: SUT /debug/vars, iowait-free CPU, per-scenario engine egress, per-column SUT env (celeris#585) #411 and feat(validation): the celeris#588 capture control -- inject an h2c hang and a WS handshake failure at a known instant, and prove the artifact names them on every engine (probatorium#351) #412)
  • root: go test -count=1 -p 1 -race -v ./...: 1232 PASS / 0 FAIL / 12 SKIP, every package ok (1143 at round 1; the new ones come with feat(bench): make the SEND_ZC on/off A/B answerable: SUT /debug/vars, iowait-free CPU, per-scenario engine egress, per-column SUT env (celeris#585) #411 and feat(validation): the celeris#588 capture control -- inject an h2c hang and a WS handshake failure at a known instant, and prove the artifact names them on every engine (probatorium#351) #412)
  • chi, echo, gin, hertz, iris: go build, go vet ./..., go test -race ./... ok (no test files in these five)
• CI all green at d434126: Test (36325002641: root module, mage-tagged, 12 adapters, refapp, fieldguard, zombie guard, govulncheck, the new ansible loads every playbook and tasks file job from feat(bench): make the SEND_ZC on/off A/B answerable: SUT /debug/vars, iowait-free CPU, per-scenario engine egress, per-column SUT env (celeris#585) #411, ci-ok), Lint (36325002657), Coverage (36325002610), CodeQL (36324999864). The PR-tier matrix is skipped (no cluster-ok label).
• Round 1 suites (laptop lock timing=none, -p 1, go 1.27.1), at 5c63890 with loadgen 9f8993e as the pseudo-version (round1/suite-r1-5c63890-*.log):
  • touched packages: go test -count=1 -p 1 -race -v ./scenarios/ ./cmd/runner/ ./servers/: 108 PASS / 0 FAIL / 3 SKIP (the rust adapter binaries are not built on the Mac)
  • mage layer: go test -count=1 -p 1 -race -v -tags mage .: 114 PASS / 0 FAIL / 1 SKIP
  • root: go test -count=1 -p 1 -race -v ./...: 1143 PASS / 0 FAIL / 12 SKIP, every package ok (round 0: 1141, plus the two new classifier rows)
  • chi, echo, gin, hertz, iris: go build, go vet ./..., go test -race ./... ok (no test files in these five)
• Lint, round 1 (round1/lint.log): go vet ./..., golangci-lint run (v2.13.2) 0 issues, gofmt -l . empty. In the adapter modules, golangci-lint is clean for chi, gin and iris; echo reports one SA1019 (middleware.LoggerWithConfig is deprecated) in chain_handlers.go, which this PR does not touch.
• CI, round 1: all green at 5c63890: Test (root module, mage-tagged, 12 adapters, refapp, fieldguard, zombie guard, govulncheck; run 36323850867), Lint (36323850884), Coverage (36323850865), CodeQL (36323849301), ci-ok, CodeRabbit. The PR-tier matrix is skipped (no cluster-ok label).
• Round 0 (15d55bc, the same tree as 991d13d): the suites above at 85/114/1141, lint and actionlint clean, CI green (Test 36320226265, Lint 36320226256, Coverage 36320226236, CodeQL 36320224492).

Review round 1

finding	what changed
major: echo-h2 and gin-h2 were said to change server-side HTTP/2 code	False, and corrected in "What changes" §1 and the H2 list. No adapter serves HTTP/2 through x/net. Rebuilt at main and at this head, echo, gin and hertz have identical code and data. iris differs only in x/net/html's min() refactor, which the adapter never reaches. The stale adapter comments that said "h2c.NewHandler-wrapped" are fixed.
minor: the per-scenario budget path was untested	Two classifier rows and one TestErrorBudgets check at a non-default budget. They kill M1, M2, M3 and M5; M4 is listed below.
minor: nothing checked that the tag points at 9f8993e	Finishing step 0, check-tag.sh. Its controls at round 1: v1.4.14 exited 2 (no tag yet), v1.4.13 against aad6d6b exits 0, v1.4.13 against 9f8993e exits 1. At the finish, v1.4.14 exits 0 (Status).
minor: a rebase merge would split the bump from the budget change	The two commits are squashed into 991d13d (a4b2d0f after the rebase onto 02e4172), whose tree was identical to 15d55bc's. Status says squash merge only.
minor: the release-order guard was left as "the maintainer's call"	Round 1 made it required before the release, with the two ways to satisfy it. The release came first anyway: see "Release-order gate" and review round 2.
nit: "Only a dial that fails counts"	The comment now reads: the expected close never counts; a failed dial, a reset, a truncated response or a timeout does.
nit: one citation for loadgen_version	All three sites are cited.
nit: goceleris/loadgen#95 not mentioned	Added to the churn-close fields and to "Not in this PR".

Review round 2

Approved. Its release-order minor is answered in "Release-order gate": the tag reached the world before this PR merged, and that section lists what protects main until it does.

Not in this PR, so Refs #424, not Fixes

• The published-docs discontinuity note (churn-close after loadgen#87: change ErrBudget 0.5 with the loadgen pin bump; lithium's churn-close becomes real churn; latency and teardown discontinuity #424 to-do 2).
• The first post-bump run checks (churn-close after loadgen#87: change ErrBudget 0.5 with the loadgen pin bump; lithium's churn-close becomes real churn; latency and teardown discontinuity #424 to-do 3): lithium connect_errors and close_aborts, TIME_WAIT on both hosts, TcpExtTCPSynRetrans, TcpExtPAWSEstab, TcpExtTCPTimeWaitOverflow, and churn-close cells with requests > 0 and throughput_bps == 0 (h1client: the response framing parser leaves the connection at the wrong position for close-delimited bodies, HEAD/1xx/204/304, TE+CL, trailers and HTTP/1.0 loadgen#93).
• The version-aware regression comparison from churn-close after loadgen#87: change ErrBudget 0.5 with the loadgen pin bump; lithium's churn-close becomes real churn; latency and teardown discontinuity #424's comment.
• Follow-ups from #91: measure the H2 request-path cost on the cluster before the release bench (CodSpeed +11% unexplained), stale I3 ledger loadgen#100: measure the H2 request-path cost on the cluster before the v1.6.0 rated bench. The rated bench after this re-pin depends on it.
• h1client: a dial in progress cannot be cancelled, so a SYN retransmission at the end of a run stretches Run's elapsed time and deflates requests_per_sec loadgen#95: the uncancellable H1 dial that deflates churn-close saturation_mode_rps when a SYN retransmission runs past the window. Until it is fixed, a churn-close RPS drop on a server that drops SYNs may be that, not the server.
• Mutant M4 above: nothing tests that executeCell passes scenarios.ErrorBudgetFor(cell.Scenario) to the classifier (cmd/runner/main.go:1225). It survives on main too. It changes nothing today, because every scenario is at the default, but it matters once a scenario raises its budget again. Covering it needs a seam in executeCell, so it is left for a follow-up issue.

Refs #424

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Repository: goceleris/probatorium/.coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 3c3ea64a-fafe-4a8f-bcb9-aa82804bdfe7

📥 Commits

Reviewing files that changed from the base of the PR and between 02e4172 and d434126.

⛔ Files ignored due to path filters (5)

• go.sum is excluded by !**/*.sum, !**/go.sum
• servers/echo/go.sum is excluded by !**/*.sum, !**/go.sum
• servers/gin/go.sum is excluded by !**/*.sum, !**/go.sum
• servers/hertz/go.sum is excluded by !**/*.sum, !**/go.sum
• servers/iris/go.sum is excluded by !**/*.sum, !**/go.sum

📒 Files selected for processing (15)

• cmd/runner/cellclassify_test.go
• go.mod
• mage_bench_ingest_test.go
• scenarios/scenarios.go
• scenarios/scenarios_test.go
• scenarios/static.go
• servers/chi/server.go
• servers/echo/go.mod
• servers/echo/server.go
• servers/gin/go.mod
• servers/gin/server.go
• servers/hertz/go.mod
• servers/iris/go.mod
• servers/iris/server.go
• servers/servers.go

---

_{Comment @coderabbitai help to get the list of available commands.}

[FumingPower3925]

Review round 1 is pushed. The branch was force-pushed, so here is how the commits map:

• 991d13d is f8e76b8 and 15d55bc squashed into one commit. Its tree is identical to 15d55bc's (git diff 15d55bc 991d13d is empty), so every round-0 result still holds for it. Now the loadgen bump and the budget change cannot reach main separately, whatever merge method is used.
• 5c63890 is round 1 (git diff 15d55bc 5c63890):
  • two classifier rows and one TestErrorBudgets check at a non-default budget;
  • the churn-close comment wording;
  • the stale h2c comments in the adapters.

The body's "Review round 1" table maps each finding to its fix. The main correction is that echo-h2 and gin-h2 do not change server-side code at this re-pin. Rebuilt at main and here, echo, gin and hertz have byte-identical code and data. iris differs only in x/net/html's min() refactor, and the adapter never reaches that code.

The Status section now also says that a release-order guard must be on main before the loadgen release is re-run. CI is green at 5c63890. The PR stays a draft until the v1.4.14 tag exists.