Skip to content

ci: CodSpeed label gate and daily main, base-tier and full-size jobs, strict alloc budgets, Python tool pins, probe logs (v1.6.0, Refs #890) - #1124

Merged
FumingPower3925 merged 16 commits into
mainfrom
l18-ci-tests
Oct 27, 2026
Merged

FumingPower3925 merged 16 commits into
mainfrom
l18-ci-tests

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Lane L18 CI-TESTS of v1.6.0. Workflow files, scripts and tests only; no production code. Refs #890, Refs #885, Refs #888.

Method: every item was checked on main (baa43d9) first; "premise" below is that check. Items are quoted by their opening words, with the ordinal that counts every checkbox in the body of the issue (the numbers of #1099 count only unchecked boxes and differ). The gates were added first with the fix left out, so a run shows what each catches (commit 1, run A), then the fixes (run D, head b56c8fd, was the first fully green run; the head is green too).

item (ordinal, opening words) premise on main result commits tick the box on merge?
890.7 "Maintainer decision on the CodSpeed Free-plan budget" TRUE: no label gate, a run per push to main done: performance label gate + daily schedule for main (the loadgen half is not in this repo). Positive arm and cron cannot be shown before merge, see below 2 OPEN until the post-merge CodSpeed schedule check (see "Post-merge verification")
890.97 "CodSpeed main baseline is 8 days stale" TRUE, now 14 days: last success run 38305334372 (2026-10-12T20:37Z) diagnosed from run data, mitigated by 890.7; the cause itself (quota) is not in the tree 2 OPEN: diagnosed only, the cause is the CodSpeed quota
890.92 "test/server/race761_off_test.go ... makes the large-response tests run their full-size shape" TRUE: no step runs it done: job large-response, measured on hosted runners 3 tick
890.93 "zizmor is the one CI tool pinned outside .github/tools/go.mod" TRUE done: requirements file + pip entry + check 4 1, 4 tick
890.96 "test/integration: skipOrFailIfStandbyAborted886 ... keeps its own inline" gate TRUE done as the guard (the second of the two options); ten existing raw skips registered, not routed 1, 4 tick
890.87 "CI never runs the io_uring base tier" TRUE (grep MAX_IOURING_TIER .github/ found nothing) done: job iouring-base-tier 3, 5 tick
890.86 "internal/engine/epoll/adaptive_abort_844_linux_test.go (the pkg844 == "iouring" arm" FALSE: the epoll copy has const pkg844 = "epoll" (:44), so the arm and its t.Fatalf (:273-285) are dead code there; nothing can fail or skip on a host without io_uring nothing to do - tick (premise false, nothing to change)
890.88 "tests (found by PR #1047 review r1, minor, item 6's siblings): keptProbeIOUring" TRUE: six copies, no log done: one Logf per retry round, as the c714ProbeIOUring copies 3 tick
890.95 "test/integration: the epoll arms of three tests skip" TRUE (three sites) done: t.Fatalf on the epoll arms, io_uring arms keep their skips 3 tick
890.91 "driver/postgres TestAllocBudgets/Query_1col_1row" TRUE: 14 allocs/op against 4 done: budget 14 and TESTING_STRICT_ALLOC_BUDGETS=1 on the step 1, 4 tick
890.6 "Make the c714ProbeIOUring probe retry" code half done by #1047 (Logf at detach_capture_alias_linux_test.go:333 and the other copies) not done: what is left is the sentence in the body of merged PR #723 - OPEN: body wording of merged #723 left
888.7 "The inbound oracle counts a server-side read ECONNRESET" assertion half done by #1046 (noteReadErr feeds serverErrs, inbound_sequence_linux_test.go:197-204); the in-tree comment (backpressure_oracle_linux_test.go:1099) already matches not done: what is left is the sentence in the body of merged PR #749 - OPEN: body wording of merged #749 left
890.75 "The ci.yml comment above the step "internal/engine/iouring — race tests at the runner's memlock"" comment half FALSE now: it already says 269.5 to 270.7 s on main and 279.0 s on #987's first run (#1093) not done: what is left is the sentence in the body of merged PR #987 - OPEN: body wording of merged #987 left
890.69 "PR #984's two wire tests use fixed sleeps" sleep half FALSE on main: #1055 replaced the 200 ms and 100 ms sleeps with a PING/ACK barrier and a 25 ms grace (h2_continuation_streams_981_linux_test.go:248, :354); the controls half was owed (#1055: "control A must be rebuilt against the pre-#984 tree") controls re-run here (control A rebuilt on current main, control E repeated), numbers below; the sleeps need no change; the two tests take 1.27 s and 1.10 s under -race on a hosted runner (run A, 6.4 s in the item) - (no change to the tree) tick (controls A and E re-run)
885.21 "(perf-api-security, nit) BenchmarkCloseChurn has extreme outlier samples" TRUE done: the warning (decision A of 10-12) in the doc comment; the 9x is the review's figure, not re-measured 3 tick

Commits: 1 a61d910 gates without their fixes; 2 e0379d2 CodSpeed; 3 b90001a the two jobs, probe logs, epoll arms, warning; 4 234e18b the fixes for 1; 5 b56c8fd the base-tier allow-lists; then e7a797c (DO NOT MERGE, bad inputs for items 87 and 92) and its revert, the CONTRIBUTING sentence, the removal of the temporary CodSpeed twin, 5556e7f (pattern entry in the root base-tier list), 3438221 (probe message as the evidence, shorter timeouts, the CodSpeed schedule caveat), a136253 and a979806 (review round 1: self-test rows first, then the fixes), the merge commit c804f3d and bc707c6 (zoned peer required in the base-tier job, floors from the merged tree). Squash on merge.

CI on the head

Head 3438221 (before review round 1 and the merge): CI run 40422726424 success (all jobs, including the two new ones: iouring-base-tier 12m36s, job 121920842823; large-response 1m32s, job 121920842684), Coverage 40422726408 success, Drivers 40422726407 success, CodSpeed 40422726405 skipped (draft, no label). gh pr checks lists all 11 required names below as pass on this head.

Required checks, before and after

The ruleset "Main — CI" requires these 11 names (gh api repos/goceleris/celeris/rulesets/22358875), read before the change and unchanged after:

Lint, Unit (root + middleware sub-modules), Conformance, Driver Conformance (Postgres + Redis), Build (ubuntu-latest), Build (macos-latest), Vulnerability Check, Cross-compile (windows/amd64), Cross-compile (windows/arm64), Portable fallback (windows-latest), Portable fallback (macos-latest).

name: of every job of ci.yml on main against this head: two jobs added (iouring-base-tier, large-response), none renamed or removed; diff of the two lists is those two lines. Both are NOT required checks (like adaptive, iouring, zc-window): they report on a pull request but do not stop its merge. Making them required is an org-admin ruleset change that this PR does not make.

890.69: controls A and E re-run

The item says: replace the sleeps with server-side conditions (done by #1055), "then re-run controls A and E to confirm the wire limit rows still fail". Control A is the pre-#984 processor (the defect: the CONTINUATION path opened its stream with GetOrCreateStream, which checks neither the stream-ID order, nor the stream's state, nor SETTINGS_MAX_CONCURRENT_STREAMS); control E is no MAX_CONCURRENT_STREAMS admission in the header path. #1055 could not build the old processor.go against main (OnHandlerPanic, MaxHeaderListSize undefined). I rebuilt control A on current main instead of reverting the file: in handleContinuation the call to serveHeaderBlock is replaced by GetOrCreateStream + SetState(StateOpen) + completeHeaderBlock, which is what the pre-#984 code did; control E replaces TryOpenStream in admitHeaders by GetOrCreateStream. Both are scratch worktrees, nothing committed. Docker golang:1.27.3, --cpus 2, --security-opt seccomp=unconfined --ulimit memlock=-1 (so that the io_uring and adaptive arms run, not skip), CELERIS_REQUIRE_IOURING_WORKERS=1, -race -count=1, the two tests TestH2MaxConcurrentStreamsHeadersContinuation981 and TestH2StreamIDOrderHeadersContinuation981:

tree rows result
this branch, unmutated 34 PASS lines 0 FAIL, 0 SKIP (3.4 s)
control A (CONTINUATION path unchecked) 12 FAIL exactly the headers+continuation rows of epoll, io_uring and adaptive: 6 of the MaxConcurrentStreams rows (slots-come-back, refused-id-reused) and the 6 reused-id and lower-id rows of the stream-ID test; every plain-headers row and every std row passes. The defect is detected, on the one path it lives in.
control E (no stream limit in the header path) 12 FAIL all 12 MaxConcurrentStreams rows of epoll, io_uring and adaptive (plain-headers and headers+continuation), the 6 stream-ID rows pass: the same 12 as #1055's control E.

The sleeps are gone and both controls still fail the rows they are for. Logs: scratchpad l18citests/logs/ctl-{celeris,ctlA,ctlE}.log. The item can be ticked for the controls half.

890.7 and 890.97: CodSpeed

Data (gh api repos/goceleris/celeris/actions/workflows/codspeed.yml/runs, then the jobs of each completed run; 682 runs, 250 jobs): the 163 jobs that got a macro runner (2026-09-26 to 10-12) add up to 880 minutes rounded up per job (786 raw); 644 fall in October and the count passes 600 at 13:45Z on 10-12. The last job ran at 20:37Z. None of the 76 jobs created after it got a runner: 35 were cancelled within 3 hours (replaced by the next run of their group), 41 after about 24 hours (the limit for a queued job), and 7 runs are still queued. So the item's "superseded by the next push" is the shorter half of the story: the queue itself stalled. That fits an exhausted monthly quota, and the header of codspeed.yml says it is not proof; the real figure is in CodSpeed's organization settings, which only a maintainer reads.

What changed: pull_request gains labeled, and the job runs only with the performance label (the draft and fork conditions stay); main is measured by a daily cron (17 3 * * *) and no longer by push; a scheduled run cancels an older one still waiting for a runner; the run that adding any other label starts has a concurrency group of its own (a labeled event of another label would otherwise cancel the benchmark run the performance label started a minute before). At 4.8 minutes a job a day costs about 144 minutes a month, against about 440 for the 92 pushes to main of the 30 days to 10-26. CONTRIBUTING.md says the performance label now also starts the run.

Evidence. codspeed.yml targets codspeed-macro-arm64-graviton-ubuntu-22-04, which is not a GitHub-hosted label (the file calls it a self-hosted runner outside GitHub's sandbox), so under the lane's rule I did not dispatch it, did not put performance on this PR and did not run it. What I could show on GitHub without it:

  • The real workflow on this PR, all skipped: 40414278753 (no label), 40414920128 (label spike just added, a labeled event), 40414979246 (label needs-info added), and the pushes 40415458833, 40419553248 and later. This is a draft PR, so the draft clause also applies; the label clause is what the twin shows.
  • A hosted twin (l18-gate-proof.yml, same triggers, same group expression and same if, with spike in place of performance, no draft clause, a 90 s sleep; deleted in the last commit): no label, run 40414278847 skipped. Label spike added: run 40414920862 ran (success). Label needs-info added 27 s later while it ran: run 40414979324 skipped and 40414920862 was not cancelled (completed success). A push with the label present: 40415458668 ran; a push with the label present 12 s after another: the run of the first, 40416207215, was cancelled by the run of the second, 40416230527.
  • Not shown, because it needs the macro runner: that a run with the real performance label starts and runs; that the cron fires (a schedule runs only from the default branch). actionlint accepts both (docker rhysd/actionlint:latest, clean on every head).
  • Unproven and to check after the merge: CodSpeed's GitHub Actions page says pushes to the default branch record the baseline and names workflow_dispatch for backtests; it does not mention schedule. If the first scheduled run is not recorded as the main baseline in the CodSpeed dashboard, push: branches: [main] goes back beside the cron (the header of the workflow says so).

890.92: the full-size shape on a hosted runner

Job large-response (a separate non-required job, not a step of an existing one: it needs a no-race build, memlock and its own timing, and a step in unit would add its 80 s to a required check) builds ./test/server without -race and coverage and runs the eight tests that read lean761(), n836() or raceOn761 by name with memlock raised, -v, a tally by name and a check that the 64 MiB HTTP/1 bodies (12 runs) and the 16 MiB HTTP/2 ones (8 runs) ran and the lean ones did not. Hosted ubuntu-latest, 4 CPUs, 16 GB:

| run | wall | peak RSS of the test process | tally |
|---|---|---|---|---|
| B, job 121892520456 | 66.1 s | 570 MiB | 8 of 8 PASS, 0 SKIP |
| D, job 121899046921 | 78.9 s | 548 MiB | 8 of 8 PASS, 0 SKIP, shape 12/8/0/0 |
| E (bad input, below), job 121899124643: the same binary built with -race | 76.3 s | 1128 MiB | 8 of 8 PASS, shape 0/0/12/8: job RED |

The job's discovery (review r1, m7) fails when a Test function of ./test/server that reads lean761(), raceOn761, n836() or dur836() is not in the list; it does not see a test that reads them only through another helper that is not itself a Test function (the awk looks at the enclosing top-level function). The size run was taken from the pull_request runs of this PR (jobs above), because no dispatchable hosted workflow builds ./test/server without race or coverage (ci.yml has no workflow_dispatch; test-coverage.yml has one but runs with -cover, the lean shape). TestH2CUnmatchedThroughTimeoutSingleflight836 runs its three arms with AsyncHandlers: true here (28.3 s in D against 9.7 s lean under -race in run A). The item's open question was whether the shape is affordable: it is, at about 80 s and 0.55 GiB. Nothing in the full-size shape failed, so no production defect came out of it.

890.87: the base tier

Job iouring-base-tier, CELERIS_MAX_IOURING_TIER=base, memlock raised, two steps: ./internal/engine/iouring under -race (473 s in run B, 471 s in D; step 504 s and 500 s) and the root package plus ./test/server for the names Shutdown|Detach|SSE|Stream|WebSocket|Hijack|Drain under -race (4.6 s and 239 s in B, 4.4 s and 236 s in D; step 258 s and 254 s). base-tier-tally.sh fails on: a SKIP line not allowed by .github/base-tier-skips-{iouring,root}.txt (12 and 19 SKIP lines on the first run: the iouring list names each, with its reason; the root list names three top-level tests and has one pattern entry for the io_uring, adaptive and adaptive-iouring arms of probe-gated tests, honoured only where the arm itself logged the probe's io_uring tier=base kernel=... skip message, not the engine's startup line), fewer top-level PASS than min-pass (554 and 107 measured on the first run; 578 and 118 on the head after the merge with main; floors now 540 and 105), a named test that did not run once and PASS (TestShutdownDetachedPartialWriteIsFlushed806, TestReportTier844), and a cap that did not take effect (the capped probe line io_uring_tier=base CELERIS_MAX_IOURING_TIER="base" must appear once and no engine arm may report tier=high). Its self-test (22 cases after review round 1) is a Lint step.

  • First run, empty allow-lists (B, job 121892520528): the step is red listing exactly the 12 and 19 skips; no test failed, no DATA RACE; TestShutdownDetachedPartialWriteIsFlushed806 passed on the capped probe, the thing the item says only local runs covered.
  • With the lists (D, job 121899047026): green; base-tier-tally iouring: top-level PASS 554 (min 500), SKIP lines 12, not in the allow-list 0, capped probe lines 1, engine arms at tier=base 15 and tier=high 0; root: PASS 107, SKIP lines 19, not in the allow-list 0.
  • A merge with main broke the exact-name root list (run 40419578127, head 08464be, job 121910358601): main had brought in conn: end an unclosed StreamWriter response at handler return, cut a broken-off one, no allocation per chunk (v1.6.0, Refs #889) #1118's nine TestH1Unclosed...889 tests, so the root step saw 117 top-level PASS and 46 SKIP lines, 27 of them not named. A list of names would go red for every test of that kind that a later pull request adds, so the root list got the pattern entry above (the iouring list stays exact names: a new skip in that package needs a line, the repo's usual rule for allow-lists). A first version of the pattern took any tier=base line as evidence; the engine's own startup line says that too (264 such lines in that log), so it now needs the probe's message, with a self-test case (22 cases in all after review r1: parallel-subtest interleaving, the exact probe message, literal names). The root log of that run passes the tally with the pattern (checked under mawk, the awk of ubuntu-latest).
  • Bad input (E, job 121899124505): CELERIS_MAX_IOURING_TIER: bse (the probe counts an unknown value as "none"): the package step is red, 47 top-level tests FAIL, the probe line reads io_uring_tier=none CELERIS_MAX_IOURING_TIER="bse". The root step then ran 23.7 minutes (against about 255 s) and was cancelled while TestShutdownDetachedPushDoesNotHoldTheDrain806/io_uring/Shutdown-background ran: with io_uring "none" the Shutdown tests fail slowly (TestShutdownSendDrainIsBounded took 96.8 s to fail), they do not skip. Step timeouts are now 900 s and 600 s and the job limit 30 minutes. go test failed before the tally ran, so the tally's own cap check is shown by the self-test fixtures (an arm at tier=high, a probe line of an uncapped run), not by a CI run.
  • No production mutant that only the base tier catches was run: the unit test for item 53 of v1.6.0 polish: io_uring engine #884 (TestAcceptedFDIsNonBlockingAndCloseOnExec884) is tier-independent and already in the Unit job. What the job adds is the engine tests running at the base tier.
  • The 19 root skips are the io_uring and adaptive arms of tests whose probe (c714ProbeIOUring, keptProbeIOUring) calls io_uring "usable" only at the high tier with provided buffers, so at the base tier they skip with "no usable io_uring". The job therefore covers the base tier for the tests that start an io_uring engine without that probe, not for all root io_uring arms.

890.93: zizmor

.github/tools/requirements.txt holds zizmor==1.30.0; the Lint step makes a venv and runs pip install -r (pipx run has --spec and no requirements-file option, checked with pipx 1.18.4); dependabot.yml has a pip entry for /.github/tools; check 4 of tool-pins-check.sh fails when a workflow or composite action installs or runs a Python tool (pipx run|install, pip(3) install, python -m pip install, uvx, uv tool run|install) with its own name==version, and when a requirements*.txt under .github/ has no pip entry for its directory (directory: or directories:). 20 new fixtures in tool-pins-check-test.sh (38 in all; review r1 m4 added uvx name@version, uv tool run name@version and the >=, ~=, < pins, with two that must not be flagged).

  • Gate without the fix (run A, Lint job 121891700676): "Tool pins agree" red with .github/workflows/ci.yml:144:run: pipx run --spec zizmor==1.30.0 zizmor .... Locally the same commit's self-test showed the check itself right (every 1-expecting fixture ok) and every "good"-expecting fixture red, because they copy the real tree.
  • With the fix (run D): Lint green, including the zizmor step through the venv.
  • Dependabot CLI v1.96.0, pip updater, local checkout, -d /.github/tools: created | zizmor ( from 1.30.0 to 1.31.1 ) (log: scratchpad l18citests/logs/dependabot-pip.log). The CLI builds its job from its arguments and does not read dependabot.yml; what it shows is that Dependabot's pip updater reads .github/tools/requirements.txt in that directory and proposes a bump. That the YAML entry names that directory is what check 4 enforces.
  • The decision the item asked for (a Python manifest in the repository for one tool) is taken here; it is one three-line file.
  • Scorecard (v5.6.0 built from source, --local, Pinned-Dependencies, Token-Permissions, Dangerous-Workflow): 10, 10, 10 on main and on this branch; the pip install -r in a venv adds no finding.

890.91: allocation budgets

Gate without the fix (run A, job 121891700334): alloc_guard_test.go:60: allocs/op=14 (budget 4), allocs/op 14 exceeds budget 4 (strict mode), alloc guards: top-level PASS 14 (want 15), packages with a wrong tally 1, SKIP lines 0, FAIL lines 2; that is the only failing step of the job and the only failing guard of the 15. With Query_1col_1row at 14 (runs C and D): top-level PASS 15 (want 15) ... FAIL lines 0, and the same in the head run. The other guards sit at 3/4, 4/4, 26/26, 0/0 and so on, exactly at budget in several; none moved in the runs above. The budget is raised to what the driver costs (item's first option; bringing the allocations down is driver/postgres's).

890.96 and 890.95: test/integration

./test/integration runs only through mage, not in CI, so its changes are shown by local runs (docker golang:1.27.3, --cpus 2) and the one guard that can run in CI is run by name in the adaptive job.

  • 890.96, the guard TestAdaptiveSkipRoutingIntegration890 (the counterpart of TestAdaptiveSkipRouting684, AST-based, with a 21-case self-test and a planted regression in the real gate file): with an empty registry it lists 11 raw skips in the three adaptive_*_test.go files, 10 in adaptive_hybrid_test.go and one in adaptive_standby_886_linux_test.go (run A, Adaptive job 121891700701, step red). The one in 886 is an io_uring-availability skip and now goes through skipOrFailUpswitch890, this package's single definition of the gate (skipOrFailIfStandbyAborted886 calls it too). The ten in adaptive_hybrid_test.go are registered with their count and why, because item 95 leaves the adaptive skips to their lane; review r1 (m3) added helpers_test.go to the scan, because startEngine skips on a Listen failure and the adaptive tests and the epoll arms of the slowloris and pipeline tests share it: its three raw skips are registered too (13 in all, debt, same reason); they are the debt, and a further site with a registered message fails the guard (checked by planting one). With the registry (run C, Adaptive job 121896544783): 0 raw skips not admitted, 5 registered (6 entries with the helpers after review r1), step green.
  • 890.95: no failing-first test is possible, since epoll.New only fails on config validation and the skip never fires. Control by hand, same mutation on main and on the branch (cfg.MaxHeaderBytes = 1, rejected by Validate): TestHTTP1PipeliningAsync PASS with two SKIPs on main, FAIL on the branch; TestSlowlorisSynthetic epoll arm SKIP on main, FAIL on the branch, iouring arm SKIP on both. Unmutated on the branch: TestHTTP1PipeliningAsync, TestSlowlorisSynthetic, TestSlowlorisSyntheticConcurrent PASS (the iouring arms skip in docker's seccomp). A dead minSkip field of the slowloris table (never set) is removed.

890.88: probe retry logs

CELERIS_REQUIRE_IOURING_WORKERS=1 CELERIS_MAX_IOURING_TIER=none, one test per copy in docker, branch against main: main prints 0 lines of retry (logger copy); the six copies on the branch print 942 to 957 keptProbeIOUring: retry round N, no usable io_uring ring yet (tier none, provided buffers false) lines in the 10 s of the retry (logger 942, recovery 953, adapters 941, metrics 943, otel 944, test/server 957). The tests then fail as designed ("forbids dropping the io_uring arms"). Logs: scratchpad l18citests/logs/probe-*.log. No CI run can show this: the probe does not retry on a runner that has io_uring.

Files beyond ci.yml, codspeed.yml and tests

The items need these: .github/dependabot.yml (890.93 pip entry), .github/tools/requirements.txt (890.93), .github/scripts/tool-pins-check{,-test}.sh (890.93 check 4), .github/scripts/base-tier-tally{,-test}.sh and .github/base-tier-skips-{iouring,root}.txt (890.87: the tally, its self-test and its allow-lists as data), driver/postgres/alloc_guard_test.go (890.91, a test file), CONTRIBUTING.md (one paragraph: the performance label now also starts the CodSpeed run; no item asks for it, but the coupling is invisible without it; #1092 does not touch the file). Overlap with open PRs checked on exact file lists: #1119, #1123, #1092 share no file with this PR; #1122 shares ci.yml (its hunks are the root-step allow-list, lines 222-240) and #1126 shares codspeed.yml (two uses: pins); git merge-tree of this head with each of #1122, #1126, #1123 and #1119 reports no conflict. Main moved by one commit while this was open (#1118, no shared file); keptProbeIOUring() has no zero-argument call site left on main or in the open PRs' diffs.

Review round 1 (head 3438221), what changed

M1 and M2 as above and below; m1 === NAME/=== CONT handled, with a crafted interleaved log that fails the old tally (4 of the 22 rows fail on the old script, 5 of the 38 on the old check 4); m2 the evidence is now the no usable io_uring message (the head's real root log, 46 SKIP lines, still passes under mawk); m3 helpers_test.go scanned; m4 range and uv pins; m5 below; m6 state == 'open' in the if: of the CodSpeed job; m7 a discovery in the large-response step (it lists the Test functions of ./test/server that read lean761(), raceOn761, n836() or dur836(); checked locally to give exactly the eight, and to report a planted ninth; a helper that is not a Test function and reads them is not seen); nits n0c (counts in this body), n1 (names compared as strings, with a row), n2 (a method of the gate's name is not the gate). Not taken: n0 (the min-pass floors stay weak: they catch a -run that stops matching, not the quiet loss of 50 tests; said here), n3 (the postgres budget has no headroom: intended for a guard).

Release gate (m5). The two new jobs do not block merges, but they do block releases: release.yml calls ci.yml (ci: job) and tag-submodules runs only if needs.ci.result == 'success', so a red iouring-base-tier or large-response blocks the tag. The same holds for the existing non-required adaptive, iouring, zc-window and recv-theft jobs. A merge with --admin can therefore land a red one unnoticed and show it at release time. release.yml is not changed.

Strict allocation budgets (n0b). Several guards sit exactly at budget (4/4, 26/26, 100/100, 1/1 and now postgres 14/14) and the allocation step is part of the required Unit check through unit-gate: a Go or dependency bump that adds one allocation to one of them turns Unit red. That is item 91's intent; the budget is then raised or the allocation removed in the PR that causes it.

Main moved: merge commit c804f3d (origin/main 8936e89, with #1118, #1122, #1123, #1125, #1126 and #1127). Head 3438221 had been tested on a merge ref that predates #1122. The merge was clean, with no conflict marker: #1126 changed one line of codspeed.yml (and one of scorecard.yml), outside every hunk of this PR, and #1127's step change in ci.yml (the "eight top-level PASS" comment of the multi-worker step, and the #885 allow-list names) is outside the hunks of this PR too. The codspeed.yml of the merge keeps both: the diff of the merge commit against its first parent (this branch) in that file is only

-        uses: CodSpeedHQ/action@821a40cd6c80e3cfe387caf55668287addd808da # v5.5.0
+        uses: CodSpeedHQ/action@78102a468f96d8e0ccb92d0ec49425753fc42062 # v5.6.0

and the gate is as before: schedule: - cron: "17 3 * * *", pull_request types with labeled, the concurrency group with the labeled-by-another-label clause, and the job if: (fork, state == 'open', not draft, label performance, labeled event only for performance). scorecard.yml is #1126's (upload-sarif 9b32540, v4.39.1).

Base tier on the merged tree (the job's own two commands, docker golang:1.27.3, --cpus 2, CELERIS_MAX_IOURING_TIER=base, seccomp unconfined, memlock unlimited, -race, then base-tier-tally.sh under mawk):

before the merge (hosted, runs B/D/H) merged tree (docker)
./internal/engine/iouring 554 top-level PASS, 12 SKIP lines, all allowed 577 PASS, 13 SKIP lines; the 13th is TestListenOnAZonedLinkLocalServes885 (#1127), because the container has no fe80:: address; capped probe line 1, tier=base arms 15, tier=high 0; TestShutdownDetachedPartialWriteIsFlushed806 and TestReportTier844 PASS
root + ./test/server 107 PASS / 19 SKIP lines (117 / 46 once #1118 was in) 118 PASS, 46 SKIP lines, 0 not allowed

The tests that #1123 and #1127 add run at the base tier without a skip, except that one. The zoned tests already fail instead of skipping under CELERIS_REQUIRE_ZONED_PEER=1 (the unit job sets it, a hosted runner's eth0 has an fe80:: address), so the two base-tier steps now set it too (bc707c6) and the allow-list does not name the test. The min-pass floors move to 540 and 105 (577 and 118 measured). Discovery of the full-size job still lists exactly the eight tests.

Post-merge verification (890.7, 890.97)

The main baseline of CodSpeed now comes from schedule runs. CodSpeed documents push to the default branch as the baseline and does not mention schedule; the pinned runner versions (v5.5.0 now, v5.6.0 in #1126) accept a scheduled run (RunEvent::Schedule), but whether the server records it as the main baseline is undocumented. The macro-runner quota is exhausted until 2026-10-31, so the first scheduled run cannot get a runner before then. Steps, once the quota is back:

  1. Check that the cron fired: gh run list -R goceleris/celeris -w codspeed.yml -e schedule -L 3 shows a run, and its job Benchmarks (arm64 macro runner) got a runner and succeeded.
  2. In the CodSpeed dashboard (goceleris/celeris), confirm that this run appears as a report on main and is the baseline: open a pull request that touches a benchmark path, add performance, and check that CodSpeed compares it with that run (not with "no baseline" or an older report).
  3. If it does not: either put push: branches: [main] (with the paths: list) back beside the schedule, or keep the cron and have it dispatch the workflow (gh workflow run codspeed.yml --ref main, a workflow_dispatch run, which CodSpeed documents). Both fallbacks are named in the header of codspeed.yml. The dispatch fallback has two costs: each dispatched run gets a concurrency group of its own, so a stalled run is no longer replaced by the next one, and CodSpeed documents workflow_dispatch for backtests, not as a baseline source, so it is no better documented than the schedule; restoring push is the only documented baseline trigger.
  4. Then add the CodSpeed badge (Make the celeris, loadgen, probatorium and docs READMEs attractive for visitors: consistent live badges (Codecov, CodSpeed, Scorecard; drop Go Report Card and stale tier badges) and a stronger first screen #700), after re-reading the baseline at merge time as 890.97 says.

Ready to paste into the #890 tracking list (not edited here):

- [ ] CodSpeed main baseline from the daily schedule (celeris#1124, 890.7/890.97): after the quota resets on 2026-10-31 and the first scheduled run gets a runner, confirm in the CodSpeed dashboard that it is recorded as the main baseline and that a PR labelled `performance` compares against it; if not, restore `push: branches: [main]` in .github/workflows/codspeed.yml or have the cron dispatch the workflow (steps in the PR body of #1124, "Post-merge verification").

Release-note lines

None.

Findings not fixed here

  1. Three sentences in merged PR bodies (890.6, 888.7, 890.75). I did not edit merged bodies; the replacements, from this lane's data:
  2. 890.97's cause is a quota or a runner-group state that only the CodSpeed organization settings show; read the usage there. The macro runner has taken no job since 10-12; seven runs are still queued (they will be cancelled by GitHub at 24 h). The README badge (Make the celeris, loadgen, probatorium and docs READMEs attractive for visitors: consistent live badges (Codecov, CodSpeed, Scorecard; drop Go Report Card and stale tier badges) and a stronger first screen #700) stays out until a run completes on main after the daily schedule starts; README is docs(readme): a first screen for visitors, performance claims with their conditions, and the v1.6.0 text (release-gated) #1092's.
  3. The hosted ubuntu-latest jobs here are advisory. iouring-base-tier adds about 13 minutes of job time to every pull request run; if that is too much it can move to a label or a schedule.
  4. Under CELERIS_MAX_IOURING_TIER=none (or a value the probe does not know) the Shutdown tests of the root package and ./test/server do not skip, they fail slowly (97 s for one): a misconfigured cap costs minutes of CI, not a skip. Not changed here.
  5. 890.86's dead arms: the epoll copy of adaptive_abort_844_linux_test.go carries three pkg844 == "iouring" branches that can never run. Removing them is a cleanup in a file this lane does not need to touch.
  6. test/integration is not in CI at all, and its ten adaptive_hybrid skips are the same trap as item 95's epoll arms (adaptive.New fails on config validation and port holds only).

For the reviewer to attack

  • 890.7: the label gate shares the release-notes label performance; a PR labelled for the notes on a benchmark path starts a macro run (intended, documented in CONTRIBUTING). Dropping the push trigger means a PR compares with a main run up to a day old, and a workflow_dispatch is the manual refresh. The positive arm (real label starts the real job) and the cron are unproven before merge; the twin proves the expression and the group, not the runner label. Is it right to also cancel an older scheduled run, and is 17 3 * * * a good time?
  • The quota story (880, 786, 644, 600 at 13:45Z) is a count I built from job timestamps; CodSpeed's own meter is not public and may differ. The header says "fits", not "is".
  • 890.87: the iouring allow-list is exact names, so a new test in that package that skips at the base tier (iouring: the shutdown gate closes idle and ended HTTP/1 conns and answers requests in progress, ring-wait ceiling, shutdown trials (v1.6.0, Refs #884) #1123 and engines: OnDetach rollback first, io_uring pause wake, zoned bind address, one-dump NIC table, MaxEvents by allowed nodes, upgrade-carry count, backlog hook (v1.6.0, Refs #885 #886 #889 #890) #1127 were the last two; both are merged and add none) turns this non-required job red until a line is added; the pattern for root arms trusts the probe's message. The floors are measured (578 PASS / 12 SKIP for the package and 118 PASS / 46 SKIP for root on the head's CI run 40439807946, jobs 121977172796). The root half covers fewer io_uring arms than the item's wording suggests (19 skips, above). Step 2 runs under !cancelled() even when step 1 failed, on purpose (both reports in one run), bounded by its timeout.
  • 890.7: CodSpeed does not document schedule as a baseline trigger (see "Post-merge verification").
  • 890.96: I registered ten skips instead of routing them. A reviewer may prefer routing adaptive.New's as t.Fatalf and the Listen ones through the gate; that is the adaptive lane's file.
  • 890.91 raises a budget rather than lowering allocations; 14 is one measured number on arm64 and amd64.
  • 890.93: python3 -m venv on ubuntu-latest instead of pipx; check 4 is a regex over pipx|pip|uvx lines and will miss other installers (pip install -c, a requirements URL, conda).
  • Commit 1 is red on purpose (Lint, Adaptive, Unit); its fixture "good" cases are red with it. The history contains a DO NOT MERGE commit and its revert; merge by squash.

…the test/integration skip guard (Refs #890)

Failing-first arm of three items of #890. Each gate is added with the fix left
out, so that CI shows what it catches:

- item 91: TESTING_STRICT_ALLOC_BUDGETS=1 on the allocation-guard step. The
  postgres Query_1col_1row guard measures 14 allocs/op against a budget of 4.
- item 93: tool-pins-check.sh check 4. No workflow installs or runs a Python
  tool with a name==version of its own, and every requirements file under
  .github/ is in a Dependabot pip entry. ci.yml still runs
  `pipx run --spec zizmor==1.30.0`, so Lint is red, and the "good" fixtures of
  the self-test, which copy the real tree, are red with it.
- item 96: TestAdaptiveSkipRoutingIntegration890 with an empty registry lists
  the 11 raw skips of test/integration/adaptive_*_test.go that bypass the
  gate; a CI step in the adaptive job runs it by name.
@FumingPower3925 FumingPower3925 added this to the v1.6.0 milestone Oct 26, 2026
@coderabbitai

coderabbitai Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…and on main once a day (Refs #890)

Items 7 and 97 of #890. The workflow's job minutes passed a 600-minute
monthly count on 2026-10-12 and no job has had a macro runner since; the header
now carries the numbers. A pull request runs the benchmarks while it carries
the `performance` label (the run that adding any other label starts has a
concurrency group of its own, so it cannot cancel the one in flight); main is
measured by a daily schedule instead of every push, and a scheduled run
replaces an older one still waiting for a runner.

Adds a temporary hosted twin of the trigger, group and condition
(l18-gate-proof.yml, label `spike` in place of `performance`) so that the gate
can be shown on GitHub without a macro-runner minute. It is deleted in a later
commit.
… retry logs, epoll arms fail, BenchmarkCloseChurn warning (Refs #890 #885)

- 890.87: job iouring-base-tier runs ./internal/engine/iouring and the root
  package's Shutdown/Detach/SSE/Stream/WebSocket/Hijack/Drain tests with
  CELERIS_MAX_IOURING_TIER=base, asserts the cap took effect and tallies skips
  against an allow-list (empty in this commit: the first run lists them).
- 890.92: job large-response builds ./test/server without -race and coverage
  and runs the eight tests that read lean761(), n836() or raceOn761 by name,
  printing wall time and peak RSS.
- 890.88: the six keptProbeIOUring copies take a testing.TB and log one line
  per retry round, as the c714ProbeIOUring copies do.
- 890.95: the epoll arms of pipeline_test.go and slowloris_synthetic_test.go
  fail where they skipped when epoll.New fails.
- 885.21: the warning about the outlier samples on BenchmarkCloseChurn.
@FumingPower3925 FumingPower3925 added spike Time-boxed exploration to de-risk a larger feature needs-info Waiting on the reporter labels Oct 26, 2026
…t 14, zizmor pinned in a requirements file, the integration skip registry (Refs #890)

- item 91: driver/postgres Query_1col_1row has the budget the driver costs, 14
  allocs/op (measured 14 on linux/arm64 in 20 of 20 runs and on the GitHub
  amd64 runner), and the allocation-guard step stays strict.
- item 93: zizmor==1.30.0 moves to .github/tools/requirements.txt, installed by
  `pip install -r` in a venv; dependabot.yml gets a pip entry for
  /.github/tools. The Dependabot CLI (v1.96.0, pip updater) proposes
  zizmor 1.30.0 to 1.31.1 from that file. Check 4 of tool-pins-check.sh passes
  and so do its fixtures.
- item 96: skipOrFailUpswitch890 is the one definition of the gate in
  test/integration (the standby gate and the io_uring skip of
  adaptive_standby_886 use it); the ten raw skips of adaptive_hybrid_test.go are
  registered with their count and why, which item 95 leaves to the adaptive
  lane.
- item 92: the large-response job also asserts that the 64 MiB and 16 MiB
  bodies ran and the lean ones did not.
- item 87: base-tier-tally.sh has a self-test in the Lint job, and its
  allow-lists are data files.
…Refs #890 item 87)

Run 40414278698 on b90001a: ./internal/engine/iouring took 473 s under -race
with 554 top-level tests passed and 12 SKIP lines, the root package and
./test/server 4.6 s and 239 s with 107 passed and 19 SKIP lines, and
TestShutdownDetachedPartialWriteIsFlushed806 passed on the capped probe. Each
skip is named in .github/base-tier-skips-{iouring,root}.txt with its reason,
the tally has a min-pass floor, and its self-test covers it.
…ed tier cap, a -race build of the full-size job); reverted by the next commit (Refs #890)
@codecov

codecov Bot commented Oct 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…misspelled tier cap, a -race build of the full-size job); reverted by the next commit (Refs #890)"

This reverts commit e7a797c.
…item 7)

Its runs are cited in the pull request: 40414920862 (label spike, ran, not
cancelled by the run another label started, 40414979324 skipped), 40415458668
(a push with the label ran) and 40416207215 (cancelled by the run of the next
push, 40416230527).
@FumingPower3925 FumingPower3925 removed spike Time-boxed exploration to de-risk a larger feature needs-info Waiting on the reporter labels Oct 26, 2026
…rn (Refs #890 item 87)

Run 40419578127 (the head after a merge with main) failed the job: main had
brought in the TestH1Unclosed...889 family, nine tests whose io_uring, adaptive
and adaptive-iouring arms skip with 'io_uring tier=base ...: no usable io_uring'
(117 top-level PASS, 46 SKIP lines, 27 not named). An exact-name list would turn
the job red for every test of that kind that a later pull request adds. The
root list now has a pattern entry for the three arm names, honoured only where
the log says tier=base under the skipped arm; the self-test covers an arm that
logged it, one that skipped for another reason, one the pattern does not name
and a top-level test named like an arm.
…er step timeouts, the CodSpeed schedule caveat (Refs #890 items 7, 87)

- The pattern entry no longer takes the engine's startup line (tier=base) as
  evidence: an arm that started an engine and skipped for another reason fails.
  Self-test case added (17 cases).
- -timeout 900 s and 600 s (about twice the measured 473 s and 239 s), job limit
  30 minutes: a run with the cap lost ended after 23 minutes in run 40416230192.
- codspeed.yml says that CodSpeed documents push as the baseline and does not
  mention schedule, and what to do if the first scheduled run is not recorded.
…1124, m1 m2 m4 n1)

base-tier-tally-test.sh: a probe message of one parallel arm must not excuse
another arm's skip (=== NAME / === CONT lines), parallel arms that each logged
it pass, another tier=base message is no evidence, and a regular expression
character in an allowed name is literal. tool-pins-check-test.sh: uvx
name@version, uv tool run name@version and range pins (>=, ~=, <) are flagged,
an @ that is no version and a redirection are not. Run against the previous
scripts: 3 and 5 rows fail (Refs #890).
…check 4 sees range and uv pins, the integration guard scans helpers_test.go, closed PRs do not start CodSpeed, discovery for the full-size job (review r1 of #1124)

- m1, m2, n1: base-tier-tally.sh credits a probe message to the test that is
  current on RUN, NAME and CONT lines, the evidence is the 'no usable io_uring'
  message the comments name, allowed names are compared as strings. The head's
  real root log (117 PASS, 46 SKIP, 100 NAME/CONT/PAUSE lines) still passes.
- m4: tool-pins-check.sh check 4 flags >=, ~=, <, > pins and uvx/uv tool name@version.
- m3, n2: the skip guard scans helpers_test.go (startEngine, three raw skips,
  registered as debt) and does not take a method for the gate function.
- m6: the CodSpeed job runs only for an open pull request.
- m5: the comments of the two new jobs say they gate releases.
- m7: the full-size job fails when a Test function that reads the lean-shape
  helpers is not in its list.
- M2: codspeed.yml's header names the fallbacks and the date of the quota reset.
(Refs #890)
…the merged tree (Refs #890 item 87)

Main 8936e89 (with #1123 and #1127) run through the job's logic in docker at
CELERIS_MAX_IOURING_TIER=base: the iouring package gives 577 top-level PASS and
13 SKIP lines, the new one being TestListenOnAZonedLinkLocalServes885 (the
container has no fe80:: address); root and test/server give 118 PASS and 46 SKIP
lines, all allowed. The zoned test fails instead of skipping under
CELERIS_REQUIRE_ZONED_PEER=1, as in the unit job, so both steps set it and the
list stays as it was. min-pass floors 540 and 105.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant