Repository navigation
v1.6.0 polish: CI and tests #890
Description
Consolidated on 2026-10-02 from the per-PR 'Follow-ups from #N' bundles, by maintainer decision. From now on, review nits are fixed inside the PR before it merges, so no new bundles are filed. Each line names the bundle and item it came from; the bundle's thread keeps the full context. Everything here stays in v1.6.0.
- Rewrite the ci.yml ramp-quarantine comment (now ci.yml:670-678 on main 24cdb7b) that still says main without the epoll: PauseAccept silently drops connections already waiting in the accept queue — their requests get EOF (8/8, deterministic), and a promotion on a GitHub runner loses ~1/3 of 2048 connections #662 fix fails with read resets; CI: lift the adaptive job's quarantine of TestRampH1Sync and TestRampH1Async once #674 is on main (>= 6 green GitHub-hosted runs) #708 says main passed 12/12, so match CI: lift the adaptive job's quarantine of TestRampH1Sync and TestRampH1Async once #674 is on main (>= 6 green GitHub-hosted runs) #708. (from Follow-ups from #674: stale ci.yml ramp-quarantine comment, stall write-up accuracy, B1 instrument note #724 1) (ci: forbid silent skips in the unit, adaptive and driver steps, run the #592 rig's io_uring half, and lift the ramp quarantine #961)
- Fix the ci.yml comment on the 'celeris#691 io_uring driver tests' step: it accounts for 17 (7+2+2+3+3) of the 19 names in new=; the first group is nine and TestDriverUnregisterWaitForOnCloseThenClose and TestDriverShutdownReleasesDescriptors are uncounted. (from Follow-ups from #696: provider.go close-before-unregister contract text, the #691 CI step's test-group count, and its job not being required #726 2) (ci: SKIP tallies fail on a joined SKIP line, non-race allocation guards, drain-order step with memlock raised, -tags http2legacy step, Redis with a password for the auth tests (v1.6.0 polish items) #1063)
- The skip-forbidding io_uring: UnregisterConn then Close leaks the driver socket — the fd-keyed ASYNC_CANCEL misses once the caller has closed the fd, so onClose never fires and the peer never sees EOF #691 (and io_uring: a worker whose ring setup or first submit fails leaves its SO_REUSEPORT listen socket open, and adaptive retries the build every tick (found by reading) #656) steps live in the non-required 'io_uring init-failure regression' job; either say so where the step is described or make that job a required check (org-admin ruleset change). (from Follow-ups from #696: provider.go close-before-unregister contract text, the #691 CI step's test-group count, and its job not being required #726 3) (done: ci: SKIP tallies fail on a joined SKIP line, non-race allocation guards, drain-order step with memlock raised, -tags http2legacy step, Redis with a password for the auth tests (v1.6.0 polish items) #1063; confirm)
- Add TestADeferredTransplantFinishesOnlyAfterItsGoroutineExits to the Unit job's named -v step so a silent skip would show (ci.yml has no match for it). (from Follow-ups from #698: shutdown leaves a deferred transplant's descriptor open (pre-existing), EPOLLOUT comment premise, CI visibility of the new test, stale comments #727 3) (done: ci: forbid silent skips in the unit, adaptive and driver steps, run the #592 rig's io_uring half, and lift the ramp quarantine #961; confirm)
- Add a CI step (or an entry in a raised-memlock step) that runs -run '^TestAdaptiveSettledRouteRetime592$' . with sudo prlimit --memlock=unlimited, -v, CELERIS_REQUIRE_IOURING_WORKERS=1 and a 6 PASS / 0 SKIP tally; the root Unit step at 8 MiB silently skips its io_uring subtests. (Also tracked as CI: two test groups have never run — the epoll sendfile e2e tests (Workers: 1 fails validation, the helper skips) and the #592 settled-route io_uring subtests (skip at 8 MiB) #709 item 2.) (from Follow-ups from #702: a raised-memlock CI step for the celeris#592 rig, with the io_uring stall assertion #739 1) (ci: forbid silent skips in the unit, adaptive and driver steps, run the #592 rig's io_uring half, and lift the ramp quarantine #961)
- Make the c714ProbeIOUring probe retry (detach_capture_alias_linux_test.go:323 and the ws/sse/otel copies) t.Logf when it fires so CI can show it; narrow fix: clone the request values a detached stream keeps (websocket Conn.Query, SSE Last-Event-ID, Context.Detach, requestid and otel context values) (#714, #717, #718) #723's 'No ENOMEM retry fired' sentence to the counted runs (it fired 3-15 times in ~12 superseded local mutant runs). (from Follow-ups from #723: otel traceparent clone cost, SSE OnConnect and session c.Set views, stale #714/#718 scope, ENOMEM claim wording, deliberate-red record #740 5) (partial: tests: start helpers retry ENOMEM and fail rather than skip; probe retry logs its rounds (checklist #888 items 3, 31, #890 item 6) #1047)
- Maintainer decision on the CodSpeed Free-plan budget (600 macro-runner min/month shared by celeris and loadgen; measured burn lasts ~3.6-4.1 days): gate pull_request runs on the 'performance' label in both repos, and/or move main to a daily schedule; also read real usage in CodSpeed's org settings. (from Follow-ups from #748: decide the CodSpeed budget lever now, the fork-PR exemption for org members, H2 benchmarks, bench-ab.sh result checks #754 1)
- .github/scripts/bench-ab.sh goes straight to benchstat even if the regex matched no benchmark in one ref; collect benchmark names from A.txt, B.txt, A2.txt and exit non-zero naming the missing side if any set is empty or A and B differ; check with a head-only and a no-match regex. (from Follow-ups from #748: decide the CodSpeed budget lever now, the fork-PR exemption for org members, H2 benchmarks, bench-ab.sh result checks #754 4) (ci(bench): refuse empty or mismatched benchmark sets, exclude OneByteGrant911 from CodSpeed #1048)
- In start_context_shutdown_test.go, TestStartContextWatcherDoesNotRepeatADirectShutdown's check races the Shutdown goroutine under the negative control (300/300 FAIL without -race, 296/300 with); fix its doc so it does not present itself as the deterministic detector, which is TestShutdownRunsHooksOnlyAfterListenReturns. (from Follow-ups from #746: the adapted #692 assertion is not forced, the ctx-bound test's unbounded-wait failure mode, CI runs the drain-order test in one shape #777 1) (test(root): harden the lifecycle tests (Refs #890 items 9, 10, 12, 13, 14) #1054)
- TestShutdownWaitForListenIsBoundedByCtx (shutdown_waits_listen_test.go) catches an unbounded wait only through the package -timeout panic (no --- FAIL line, 300 s, aborts the root package), and takes start after context.WithTimeout; run Shutdown on a goroutine with its own 10 s watchdog and take start before WithTimeout. (from Follow-ups from #746: the adapted #692 assertion is not forced, the ctx-bound test's unbounded-wait failure mode, CI runs the drain-order test in one shape #777 2) (test(root): harden the lifecycle tests (Refs #890 items 9, 10, 12, 13, 14) #1054)
- CI runs TestShutdownHooksRunAfterTheDrain only in the Unit root step (no -v, 8 MiB memlock, one io_uring worker, no skip path on ENOMEM); add a named -v step so the multi-worker drain shape is covered, as Follow-ups from #698: shutdown leaves a deferred transplant's descriptor open (pre-existing), EPOLLOUT comment premise, CI visibility of the new test, stale comments #727 item 3 asks for another test. (from Follow-ups from #746: the adapted #692 assertion is not forced, the ctx-bound test's unbounded-wait failure mode, CI runs the drain-order test in one shape #777 3) (ci: SKIP tallies fail on a joined SKIP line, non-race allocation guards, drain-order step with memlock raised, -tags http2legacy step, Redis with a password for the auth tests (v1.6.0 polish items) #1063)
- start_failure_release_linux_test.go:24 says other tests' descriptors 'cannot move' the /proc/stat count, but every celeris server in the binary opens and closes a /proc/stat monitor; reword to say the tests are safe only because they are non-parallel top-level tests. (from Follow-ups from #747: /proc/stat test comments and direction, a dial assertion that cannot fail, the uncommitted fd-delta probe #778 1) (test(root): harden the lifecycle tests (Refs #890 items 9, 10, 12, 13, 14) #1054)
- In start_failure_release_linux_test.go change after != before to after > before (:64, :129), the leak direction, so a monitor closed elsewhere in the window cannot fail the check. (from Follow-ups from #747: /proc/stat test comments and direction, a dial assertion that cannot fail, the uncommitted fd-delta probe #778 2) (test(root): harden the lifecycle tests (Refs #890 items 9, 10, 12, 13, 14) #1054)
- TestFailedStartClosesTheSuppliedListener (start_failure_release_test.go) closes the listener itself before dialling, so its dial assertion cannot fail; dial first, or drop the claim from the test's doc. (from Follow-ups from #747: /proc/stat test comments and direction, a dial assertion that cannot fail, the uncommitted fd-delta probe #778 3) (test(root): harden the lifecycle tests (Refs #890 items 9, 10, 12, 13, 14) #1054)
- The fd-delta probe behind fix(server): a Start that never serves releases the caller's listener, the CPU monitor and the settle re-opener (#737) #747's scenario table lives only in the maintainer's evidence root; commit it (as a test, or in probatorium) or mark the table maintainer-local. (from Follow-ups from #747: /proc/stat test comments and direction, a dial assertion that cannot fail, the uncommitted fd-delta probe #778 4)
- TestHijackKeepsRequestViews has no reuse witness: count per round that B was served in the defect's conditions (epoll: B's connState is A's recycled one; io_uring: from the pool) and require a minimum, so a later pool change cannot make the test vacuous. (from Follow-ups from #773: an AsyncHandlers arm after #774, io_uring multishot strings read before Hijack, the copy on std, the engine-side cost, doc and witness nits #785 6) (partial: tests: hijack request views on AsyncHandlers, forced connState reuse and its witness, inline-run assertions (v1.6.0 polish items) #1087)
- Run the test-only commit aa2331d (TestRegisterConnDeliversBytesQueuedBeforeIt) once on amd64 (cluster celeris-stress target=cluster arches=x86 -race, or a GitHub runner) expecting FAIL; if it passes, force the window harder on amd64. (from Follow-ups from #776: a test for RegisterConn's rollback, an amd64 failing-first run, the 200 ms bound, the Coverage linger flake on pre-#744 branches #787 2)
- Add the six tests of engine/iouring/park_close_fin_test.go (TestParkedWorkerSendsFINFor{AHeaderTimeout,AReadTimeout}Close, TestRunningWorkerSendsFINForAHeaderTimeoutClose, TestParkedAsyncWorkerSendsFINForAHeaderTimeoutClose, TestParkedWorkerSendsFINToMany{ReadTimeout,HeaderTimeout}Closes) to a named CI interlock in .github/workflows/ci.yml reporting want N, ran N, passed N, SKIP lines 0. Still absent on main 24cdb7b; test(iouring): regression arms for #712 (fixed by #793) #767's merged body also lists it as open. (from Follow-ups from #767: a named CI interlock for the #712 tests, and the bare-metal failing-first #795 1) (ci, test, comments: the tick pass leftovers of #890, #884 and #892 #1093)
- Add the io_uring: a parked worker keeps a stale cachedNow; if the park outlasted ReadTimeout, conns it adopts or accepts on waking get a past timestamp and the first checkTimeouts closes them (adaptive: 6/6 promotes 31 s after a demote, 0/8 at 18 s; A/B not run) #713 tests to named CI interlocks (want N, ran N, passed N, SKIP lines 0): epoll twins TestAcceptAfterALongParkIsNotTimedOutEpoll and TestAdoptAfterALongParkIsNotTimedOutEpoll (which can t.Skip silently) and the ten io_uring tests in engine/iouring/park_stale_clock_test.go; or turn the twins' skips into failures under a CELERIS_REQUIRE_* variable. Absent from ci.yml on main 24cdb7b. (from Follow-ups from #768: named CI interlocks for the #713 tests and epoll twins, and why the async standby's drain grew from ~1 s to ~11 s #797 1) (ci, test, comments: the tick pass leftovers of #890, #884 and #892 #1093)
- Triage the flake TestHeldRecvIsReArmedWhenTheHandOffDoesNotHappen/target_refuses on main (sync mode, -race -count=10: 1 FAIL, all 64 conns read_timeout); file a separate issue if it reproduces. No tracker exists in the four repos. (from Follow-ups from #799: assert the h2c arms' outcome, write down the exit contract, a rerunHandOff witness, an unattributed merged-tree failure, two flakes on main #816 5a)
- Triage the flake TestFlapConvergesPollSync (FLAPPOLL PLACEMENT: flap 3 left 8 of 64 conns, want <=2; once in an adaptive run, then 10/10 pass on main and merged tree); file a separate issue if it reproduces. No tracker exists. (from Follow-ups from #799: assert the h2c arms' outcome, write down the exit contract, a rerunHandOff witness, an unattributed merged-tree failure, two flakes on main #816 5b)
- Replace sleep-as-synchronisation in fix(epoll, iouring): send a large response whole, keep pipelined responses in order, never send a body the handler has given back (celeris#761, celeris#802, celeris#817) #805's tests with server-side conditions: TestBackloggedPeerIsClosed's 300 ms sleep (large_response_linux_test.go:418 on main; can fail falsely), and the 2 ms / 100 ms sleeps in TestSplitBodyResponsesKeepTheConnection and TestConcurrentResponsesOwnTheirBodies (CodeRabbit on fix(epoll, iouring): send a large response whole, keep pipelined responses in order, never send a body the handler has given back (celeris#761, celeris#802, celeris#817) #805). (from Follow-ups from #805: the memory bound after the per-request cap, epoll's closing-conn reap by read time, no deterministic H2 refusal test, io_uring's dormant WRITEV path, body nits #818 7) (partial: test: replace timing-based waits in the wire and shutdown tests (Refs #890 items 22, 25, 26, 53, 63, 69) #1055)
- Delete skipIouringAsyncPipelined751 and its two call sites (large_response_linux_test.go:334,400,495-501 on main) now that fix(iouring): an async handler's direct write waits for a ring SEND of the conn's earlier bytes (celeris#751) #800 is merged (65d1755); the trial merge ran the cases 30/30 and 15/15 with the skip disabled. (from Follow-ups from #805: the memory bound after the per-request cap, epoll's closing-conn reap by read time, no deterministic H2 refusal test, io_uring's dormant WRITEV path, body nits #818 9) (ci: forbid silent skips in the unit, adaptive and driver steps, run the #592 rig's io_uring half, and lift the ramp quarantine #961)
- TestWriteBufBackpressureClosesSlowConsumer stays gated on GOTEST_BACKPRESSURE=1 (engine/epoll/backpressure_test.go:67), so CI skips it without a record; give it a CELERIS_REQUIRE_* switch or a CI tally, or document it as a manual test. (from Follow-ups from #805: the memory bound after the per-request cap, epoll's closing-conn reap by read time, no deterministic H2 refusal test, io_uring's dormant WRITEV path, body nits #818 10) (ci: forbid silent skips in the unit, adaptive and driver steps, run the #592 rig's io_uring half, and lift the ramp quarantine #961)
- TestShutdownH2PoolWaitIsBounded (shutdown_h2_pool_linux_test.go, bound = 3*time.Second, unchanged on main) passes on main and allows budget+3s; add a lower bound (handler still held when Start returns, >= budget) and tighten the upper bound so the test pins the wait. (from Follow-ups from #808: std h2c conns get no GOAWAY and outlive the drain, a pool-wait bound test that passes on main, cost wording #820 2) (test: replace timing-based waits in the wire and shutdown tests (Refs #890 items 22, 25, 26, 53, 63, 69) #1055)
- TestShutdownWaitsForFlowControlledH2Response/*/sync (shutdown_h2_pool_linux_test.go:590 on main) synchronises on time.Sleep(300ms); on a slow runner GOAWAY can name last-stream-id 0 and the test fails for the wrong reason. Signal on stream 1's first DATA frame instead. (from Follow-ups from #808: std h2c conns get no GOAWAY and outlive the drain, a pool-wait bound test that passes on main, cost wording #820 5) (test: replace timing-based waits in the wire and shutdown tests (Refs #890 items 22, 25, 26, 53, 63, 69) #1055)
- engine/std/listen_cancel_budget_test.go (~line 133 on main): shutCtx is created before start is taken, so 'el < budget' has a sub-microsecond theoretical margin; create the context after start. (from Follow-ups from #803: overlapping Shutdown calls share the shortest budget, the Engine.Shutdown doc sentence, a test margin nit #821 3) (fix(std): Shutdown sends h2c connections GOAWAY at the start of the drain and tells a live overlapping caller whose budget ended it #968)
- Add the variant-rebuild case (rebuild only the .br/.gz, keep the original's mtime, resume with If-Range) to middleware/static/range_435_test.go and show it fails on e410772. (from Follow-ups from #834: If-Range on pre-compressed .br/.gz variants (rest of #435), the IfRange date-safety comment, an unsatisfiable If-Range test, the stat/open window #846 1c) (fix(static): a pre-compressed variant has its own validators, 304 and range; a strong fs.FS ETag; Root validators from the opened file (celeris#846) #972)
- Add an If-Range mismatch on an unsatisfiable range (must be 200, RFC 9110 13.2.2, not 416) to middleware/static/range_435_test.go for both os and fs.FS paths and to range_engines_435_linux_test.go; mutant M1 (Parse before If-Range) is caught only by one root case today. (from Follow-ups from #834: If-Range on pre-compressed .br/.gz variants (rest of #435), the IfRange date-safety comment, an unsatisfiable If-Range test, the stat/open window #846 3) (fix(static): a pre-compressed variant has its own validators, 304 and range; a strong fs.FS ETag; Root validators from the opened file (celeris#846) #972)
- When h2: an async handler's streamed response can lose its last frames in the write queue (lost wakeup in h2ShardedQueue.DrainTo): the stream never ends for a quiet client #837 lands, put /stream and /stream-head back into the async-route raw-HEAD check in auto_head_options_engines_421_linux_test.go:215-218. (from Follow-ups from #841: HEAD now reaches #837 and #835, HEAD on Detach-streaming GET routes, the per-request isHEAD read on live streams, FullPath sentinels, OPTIONS opt-out #847 1b) (ci, test, comments: the tick pass leftovers of #890, #884 and #892 #1093)
- Add a HEAD case to sse: an OnConnect rejection never reaches the client: no response on epoll/io_uring/adaptive (the client hangs), an empty 200 on std #835's regression test. (from Follow-ups from #841: HEAD now reaches #837 and #835, HEAD on Detach-streaming GET routes, the per-request isHEAD read on live streams, FullPath sentinels, OPTIONS opt-out #847 1c) (fix(api): an error or panic before a taken StreamWriter sends anything is still answered, so an SSE OnConnect rejection reaches the client (celeris#835) #901)
- Add a Lint-job step that fails when the modules of the tools ((cd .github/tools && go list -f '{{.Module.Path}}' tool | sort -u)) differ from the dependency-name entries of the /.github/tools entry in .github/dependabot.yml (:47-51); a missing entry or a package path instead of the module path silently stops Dependabot (verified with Dependabot CLI v1.93.0). Probe fix-r2/scripts/allow-guard-probe.sh already exists. Same gap in probatorium (deps: bump the all-go-deps group across 2 directories with 6 updates #464) and loadgen (perf: x86 multi-socket optimization + AVX2 SIMD (v1.1.0-beta.6) #104), tracked there. (from Follow-ups from #839: an unchecked Dependabot allow list for .github/tools, the Scorecard side of its security trade-off, inline go install pins, a stale evidence label #853 1a) (ci: mage and actionlint are tools of .github/tools run with go tool; a Lint step compares the tool pins with Dependabot, CONTRIBUTING and the workflows (v1.6.0 polish items) #1069)
- Mention that check in the dependabot.yml comment (:44-45). (from Follow-ups from #839: an unchecked Dependabot allow list for .github/tools, the Scorecard side of its security trade-off, inline go install pins, a stale evidence label #853 1b) (ci: mage and actionlint are tools of .github/tools run with go tool; a Lint step compares the tool pins with Dependabot, CONTRIBUTING and the workflows (v1.6.0 polish items) #1069)
- Move the four inline go install pins (ci.yml:38 mage@v1.17.2, ci.yml:80 actionlint@v1.7.12, release.yml:61 and release-checklist.yml:52 mage@v1.17.2) into .github/tools as tool directives with allow entries, run via go tool -modfile, so tools are pinned one way and Dependabot bumps them. (from Follow-ups from #839: an unchecked Dependabot allow list for .github/tools, the Scorecard side of its security trade-off, inline go install pins, a stale evidence label #853 3) (ci: mage and actionlint are tools of .github/tools run with go tool; a Lint step compares the tool pins with Dependabot, CONTRIBUTING and the workflows (v1.6.0 polish items) #1069)
Added 2026-10-02 18:40Z from #860 comment 5958063514, the second review of #843 at f718f05 (posted 4 min before the consolidation closed #860):
-
Pin the write half of UnregisterConn's promise: in TestTeardownMarksTheConnClosedBeforeItLeavesTheWorker784/UnregisterConn, assert unregErr is still empty while the test holds c.mu. Mutant X10 (rmutate_r2.py X10) must fail it (from Follow-ups from #843: the cost of forget's EPOLL_CTL_DEL under the map lock, the test hook on the read path, the repro's thin P arm, a worker starved by one conn's inflow #860 item 5). (test(eventloop): hooks and mutant pins for the teardown orders, the WriteAndPoll polls, shutdown, wake and RegisterConn's ADD #1050)
-
Six of the 13 caller-side reads in WriteAndPoll* are unpinned after a teardown (Phase B/C of WriteAndPoll and WriteAndPollBusy; WriteAndPollMulti :1010/:1046). Commit the review's zz_review2_multi_seam_784 probe, and add a hook before poll(2) for Phase B/C, or state that they are pinned by inspection only (from Follow-ups from #843: the cost of forget's EPOLL_CTL_DEL under the map lock, the test hook on the read path, the repro's thin P arm, a worker starved by one conn's inflow #860 item 6). (test(eventloop): hooks and mutant pins for the teardown orders, the WriteAndPoll polls, shutdown, wake and RegisterConn's ADD #1050, test(eventloop): the slow-path reach test releases the reply from the hook, so its counts no longer depend on scheduling #1053)
-
Add tests for X1 (forget's DEL after releasing w.mu) and X2 (shutdown marking conns closed after closing epfd). Both mutants survive the whole *784 suite. Add X1 before item 1's locking change (from Follow-ups from #843: the cost of forget's EPOLL_CTL_DEL under the map lock, the test hook on the read path, the repro's thin P arm, a worker starved by one conn's inflow #860 item 7). (test(eventloop): hooks and mutant pins for the teardown orders, the WriteAndPoll polls, shutdown, wake and RegisterConn's ADD #1050)
-
read_after_unregister_784_other_test.go:80-87 handles only pair[0] == a; reuse c784TakeNumber's switch with Dup2 (from Follow-ups from #843: the cost of forget's EPOLL_CTL_DEL under the map lock, the test hook on the read path, the repro's thin P arm, a worker starved by one conn's inflow #860 item 8). (test(eventloop): the number-taking helper on every unix, a Loop.Close order test for the portable worker, and the CI list that requires them #1051)
-
test/drivercmp/redistests do not build off Linux:integrated_async_test.gohas no build constraint but usescelerisRedisEnv, whichintegrated_test.godeclares under//go:build linux, sogo vet ./...in that module fails on darwin (same on main 24cdb7b; CI runs on Linux, so CI is unaffected). Giveintegrated_async_test.gothe same//go:build linux. (from fix(api): remove every deprecated public API and basicauth.HashPassword before the public v1.6.0 (celeris#830, celeris#826) #894) (fix(websocket): Conn.Param, IP and RemoteAddr answer from the upgrade request on every engine; every test helper that starts a server fails at once with Start's error (celeris#721, celeris#706) #918) -
TestNoDeprecatedPublicAPI830(deprecated_api_830_test.go, fix(api): remove every deprecated public API and basicauth.HashPassword before the public v1.6.0 (celeris#830, celeris#826) #894 at a519143): (a) its notice half bans everyDeprecated:paragraph outside internal/ for good (doc comment :45-60), and its message (:126) says to remove the identifier "before v1.6.0" or move it under internal/. After the v1.6.0 tag both of those are breaking changes; the v1.x way is to deprecate now and remove in v2. GOVERNANCE.md and CONTRIBUTING.md state no deprecation policy, so this test is the only place one is stated. Keep the removed-names half as a permanent invariant, and turn the notice half into an explicit allowlist (empty at v1.6.0, each entry naming its v2 removal issue) whose message says so. (b) Its markerdeprecationMarker = "Deprecated" + ": "(:20, used at :244) needs a trailing space, so a notice written as// Deprecated:at the end of a line passes the guard although the issue's done conditiongit grep 'Deprecated:'finds it (correctness reviewer's arm S5); also matchDeprecated:at end of line. Test-only: fix both in fix(api): remove every deprecated public API and basicauth.HashPassword before the public v1.6.0 (celeris#830, celeris#826) #894 before it merges. (from fix(api): remove every deprecated public API and basicauth.HashPassword before the public v1.6.0 (celeris#830, celeris#826) #894 review r1: API/perf/security MINOR 1, correctness NIT 1) DONE in fix(api): remove every deprecated public API and basicauth.HashPassword before the public v1.6.0 (celeris#830, celeris#826) #894 (2c81073, from 8ee8257): (a) thedeprecationsAllowedInV1allowlist, empty at v1.6.0, each entry with its notice count and v2 removal issue; (b) the marker isDeprecated:with or without text after it. -
Put the three celeris#905 tests in a by-name CI interlock. Moved into fix(epoll, iouring): a stopped engine no longer hands its loop threads back to the scheduler pinned to one CPU (celeris#905) #907 (round 1, once fix(epoll, iouring): an adopted conn's carried requests get the async dispatch rule under AsyncHandlers (celeris#543) #898 had merged): the Unit job's step
celeris#905 stopped engines leave no thread pinned to one CPU (skipping forbidden)runs all three by name (epoll and io_uring at the runner's memlock, adaptive with memlock raised and CELERIS_REQUIRE_UPSWITCH=1) and requires 3 PASS, no SKIP line and 3 RESULT lines with off_mask_after_stop=0 and locked_off_mask=0, and since round 2 pinned_not_main above 0 and alive_after_stop=0. Tick this when fix(epoll, iouring): a stopped engine no longer hands its loop threads back to the scheduler pinned to one CPU (celeris#905) #907 merges. (from fix(epoll, iouring): a stopped engine no longer hands its loop threads back to the scheduler pinned to one CPU (celeris#905) #907) (fix(epoll, iouring): a stopped engine no longer hands its loop threads back to the scheduler pinned to one CPU (celeris#905) #907) -
The celeris#905 io_uring test checks the restore half of the fix(epoll, iouring): a stopped engine no longer hands its loop threads back to the scheduler pinned to one CPU (celeris#905) #907 fix (the main thread gets its mask back) only in runs where the one io_uring worker ran on the main thread: 8 of 20 first cycles locally, none in CI run 37087402233. More cycles do not raise that: with up to 12 cycles, 5 of 10 runs never had the worker on the main thread (L905
logs/r1-x1/iouring-x10.log). A deterministic check needs the worker started on the main thread on purpose (a test hook in the engine, say).TestSaveThreadAffinityRestoresThePincovers Save and Restore themselves, and epoll and adaptive have a loop on the main thread in 20 of 20 first cycles. (from fix(epoll, iouring): a stopped engine no longer hands its loop threads back to the scheduler pinned to one CPU (celeris#905) #907)
Added 2026-10-03 (from #899, the last pre-rule bundle):
-
A half-done async promotion in a transplant replay is caught only by -race. Add an assertion that does not need the race detector, so that mutants C/D (cs.asyncPromoted not set; evidence lanes-20261002/L616_543/review-correctness-r2/) fail without -race (from Follow-ups from #898 (celeris#543): a promoting replay's RequestCount differs by engine; a replay parse error closes without the error response; a replayed h2c upgrade closes the conn #899 item 3). (epoll,iouring: one handling of an HTTP/1 verdict for the recv path and the transplant replay; a pipelined h2c upgrade behind a direct-body request #1015)
-
CONTRIBUTING.md copies CI's tool pins by hand: mage v1.17.2 (CONTRIBUTING.md:19 after fix(mage): Tools installs h2spec 2.6.0 on every platform and prints the version it ended with; pin the mage and benchstat hints (celeris#838) #910) and golangci-lint v2.13 (:22), while ci.yml:38, release.yml:61 and release-checklist.yml:52 pin mage and ci.yml pins golangci-lint, so the next bump leaves the hints stale with nothing failing. When the item that moves mage into .github/tools (from Follow-ups from #839: an unchecked Dependabot allow list for .github/tools, the Scorecard side of its security trade-off, inline go install pins, a stale evidence label #853 3) lands, point the hint at
GOWORK=off go install -modfile=.github/tools/go.mod toolas probatorium's CONTRIBUTING does (probatorium#485), and have the Lint job compare CONTRIBUTING's golangci-lint version with ci.yml's. (from fix(mage): Tools installs h2spec 2.6.0 on every platform and prints the version it ended with; pin the mage and benchstat hints (celeris#838) #910 review round 1, nit) (ci: mage and actionlint are tools of .github/tools run with go tool; a Lint step compares the tool pins with Dependabot, CONTRIBUTING and the workflows (v1.6.0 polish items) #1069)
Added 2026-10-03 (#907's round-2 reviews, lane L905):
- Codecov counts test-only helper packages as library code: fix(epoll, iouring): a stopped engine no longer hands its loop threads back to the scheduler pinned to one CPU (celeris#905) #907's patch coverage read 3.52%, with 218 of its 219 missing lines in internal/platform/pintest/pintest_linux.go (the Codecov comment on fix(epoll, iouring): a stopped engine no longer hands its loop threads back to the scheduler pinned to one CPU (celeris#905) #907, issuecomment-5964324517), and middleware/internal/testutil is counted the same way. Add both, and any later test-support package, to codecov.yml's
ignorelist, which today holds only test/**, /testdata/ and **/*.pb.go. celeristest is public API and stays counted. (from fix(epoll, iouring): a stopped engine no longer hands its loop threads back to the scheduler pinned to one CPU (celeris#905) #907) (ci(bench): refuse empty or mismatched benchmark sets, exclude OneByteGrant911 from CodSpeed #1048)
Added 2026-10-03 (#910's round-2 reviews, lane CI1):
-
mage toolsaccepts on its second run the h2spec its first run refused (magefile.goTools, fix(mage): Tools installs h2spec 2.6.0 on every platform and prints the version it ended with; pin the mage and benchstat hints (celeris#838) #910 at fe509c8). The early return (:106-111) checks only--version, which the-ldflagsstamp makes say 2.6.0 for any code. So the 2.6.0-stamped v2.2.1+incompatible build that the post-install build-info check refuses (:131-137) stays in GOBIN, and the nextmage toolsreports it "(already installed)" and exits 0. The same branch accepts such a build anywhere on PATH. Fix: in the early path, readh2specBuiltFrom(path)and reinstall when the build info namesgithub.com/summerwind/h2specat a version other thanh2specModVersion. Keep binaries with no Go build info (the release tarballs) and homebrew's build, which records exactlyv1.5.1-0.20200804131034-70ac22940108(go version -m /opt/homebrew/bin/h2spec). Optionallyos.Removethe install before returning a refusal. Failing test: the review'sTestReviewerRefusalIsSticky(lane evidencelanes-20261003/CI1/review-correctness-r1/r1x-tests/magetools/reviewer_rerun_test.go), which FAILs at fe509c8 on darwin/arm64, linux/arm64 and linux/amd64 (r1x-logs/838-{host,linux-arm64,linux-amd64}-P-probe-sticky-head.log). (from fix(mage): Tools installs h2spec 2.6.0 on every platform and prints the version it ended with; pin the mage and benchstat hints (celeris#838) #910 review round 2, minor and nit; a behaviour change, so not fixed in the PR) (mage: Tools reads the build info of an h2spec it finds before keeping it, and removes a refused install (v1.6.0 polish item) #1071) -
.coderabbit.yaml:184(main fa76fe6) tells the review bot, as a fact, "Code from a fork pull request never runs on a self-hosted or CodSpeed macro runner". An organization member's fork pull request runs without approval (CI: the fork-PR approval policy exempts org members, so a read-only member's fork PR runs codspeed.yml on the macro runner without approval #864), and theif:that skips forks sits in the workflow file, which a fork can edit. Reword it as the rule it is, the way codspeed.yml:108-120 already does: such a job must skip forks withhead.repo.full_name == github.repository, and the fork-PR approval policy is what holds a fork that edits the workflow. probatorium#486 corrects the same claim in probatorium's matrix-pr-tier.yml and CONTRIBUTING.md, and loadgen#108 in loadgen's codspeed.yml. (found by the CI: the fork-PR approval policy exempts org members, so a read-only member's fork PR runs codspeed.yml on the macro runner without approval #864 fork-claim audit in fix(mage): Tools installs h2spec 2.6.0 on every platform and prints the version it ended with; pin the mage and benchstat hints (celeris#838) #910's round-2 fix) (ci(bench): refuse empty or mismatched benchmark sets, exclude OneByteGrant911 from CodSpeed #1048)
Added 2026-10-03 (#923's round-1 reviews, lane F3T):
- middleware/cache/slow_leader_913_linux_test.go (fix(cache): the coalesced fill stores the response and releases its followers before any client is written, so a slow leader's client no longer holds them (celeris#913) #923 at 87ecd45, :86-98): the handler only ever returns a cacheable 200, so the test covers only the coalesced fill. fix(cache): the coalesced fill stores the response and releases its followers before any client is written, so a slow leader's client no longer holds them (celeris#913) #923 also takes the leader's write out of the call on every uncacheable path, where followers fall back to
executeAndStore(no-store, non-2xx, 206, a body overMaxBodyBytes, a handler that writes its body and returns an error), and reorders the non-coalesced path to store before it writes (cache.go:139-149). No test checks either. Add one uncacheable row (no-store or 206) and oneSingleflight: falserow to the same harness, with failing-first on base + fix(h2): bound the response bytes a connection holds for the peer's window; over the budget, handlers run on the pool and send as the windows open (celeris#893) #906. The review's probeTestReview923SlowLeaderPaths(7 paths x 4 engines) already shows the shape: all 28 rows FAIL on base + fix(h2): bound the response bytes a connection holds for the peer's window; over the budget, handlers run on the pool and send as the windows open (celeris#893) #906 and meet the fixed semantics on head + fix(h2): bound the response bytes a connection holds for the peer's window; over the budget, handlers run on the pool and send as the windows open (celeris#893) #906, except three native errbody rows whose assertion is over-strict (followers on the leader's sync loop arrive after the call ends). Evidence:lanes-20261003/F3T/review923-corr-r1/probes/review923_paths_probe_test.go,logs/q1/rp-{test,fix,906test,906fix}.probe.log. (from fix(cache): the coalesced fill stores the response and releases its followers before any client is written, so a slow leader's client no longer holds them (celeris#913) #923 review r1, correctness MINOR 1) (fix(cache,singleflight): coalescing correctness and leader-path polish (v1.6.0 checklist items) #1007)
Added 2026-10-03 (#935's round-1 reviews, lane D2):
- redis and memcached: add a portable recording-loop test for Close, like postgres's
checkCallsin fix(postgres): keep a conn's descriptor number until Close has made its last call on it (celeris#859) #935 (driver/postgres/close_after_teardown_859_test.go): a WorkerLoop that records, at each call by number, which socket the number names (by inode). fix(postgres): keep a conn's descriptor number until Close has made its last call on it (celeris#859) #935's end-to-end guards (driver/redis/close_after_teardown_859_linux_test.go, driver/memcached/close_after_teardown_859_linux_test.go) catch onClose releasing the number. They do not catch Close releasing it before UnregisterConn: Close unregisters (driver/redis/conn.go:946, driver/memcached/conn.go:667 at 56a6c1e) before it closes the fd (:953, :674). With that order inverted, UnregisterConn would act on whatever conn has taken the number. The correctness review's arm x9 (Close callsfile.Close()beforeUnregisterConn) passes every postgres: after a worker-side teardown, Close writes Terminate to and unregisters the conn that has taken the closed descriptor number #859 test, PASS 70 / FAIL 0 / SKIP 0 for each driver, and the whole redis and memcached suites, PASS 159 / FAIL 0 / SKIP 0. Evidence:lanes-20261003/D2/review-correctness-r1/logs/ctl-x9-{redis,memcached}-release-before-unregister-arm64.logandsuites-x9-arm64.log; injections inreview-correctness-r1/scripts/make-variants.py. The new test must FAIL under x9 and PASS on main. (from fix(postgres): keep a conn's descriptor number until Close has made its last call on it (celeris#859) #935 review r1, correctness MINOR) (redis: run the PubSub rig tests on the epoll and io_uring engines (v1.6.0 checklist item) #1017)
Added 2026-10-03 (#932's round-1 review, lane D1):
- fix(eventloop): a Write or RegisterConn racing Loop.Close no longer uses the worker's eventfd or epoll fd number after shutdown closed it (celeris#862) #932 wake test:
TestWriteRacingCloseNeverTouchesTheClosedEventfd862(driver/internal/eventloop/wake_close_862_linux_test.go) catches its defect only through the race detector. Its second oracle (two eventfds that take the freed numbers must receive no write) needs a wake to land in the few microseconds between the close and the reuse: on main without-raceit fired in 0 of 10 processes. The likeliest regression of the fixed design is a wake that writesw.wakeFD.FD()itself (FD()is exported and lock-free, andrunalready uses it). The correctness review's mutant WAKEATOMIC (that shape) failed the test in 1 of 10 processes with-race, by the second oracle, and 0 of 10 without (evidencelanes-20261003/D1/review-correctness-r1/logs/run-r1/m-WAKEATOMIC.log,m-WAKEATOMIC-norace.log,base-wake-norace.log). Add a deterministic test: a nil-by-default hook inshutdownbetween the eventfd's close and the return ofw.mu, where the test opens the taker eventfd and then lets a wake that loaded the number before the close go ahead, so the regression fails without-race. fix(eventloop): a Write or RegisterConn racing Loop.Close no longer uses the worker's eventfd or epoll fd number after shutdown closed it (celeris#862) #932 states in the test's comment and its body that coverage rests on-race. (from fix(eventloop): a Write or RegisterConn racing Loop.Close no longer uses the worker's eventfd or epoll fd number after shutdown closed it (celeris#862) #932 review round 1, correctness NIT 1) (partial: test(eventloop): hooks and mutant pins for the teardown orders, the WriteAndPoll polls, shutdown, wake and RegisterConn's ADD #1050)
Added 2026-10-03 (#932's round-2 review, lane D1):
- fix(eventloop): a Write or RegisterConn racing Loop.Close no longer uses the worker's eventfd or epoll fd number after shutdown closed it (celeris#862) #932 register tests:
TestRegisterConnRacingCloseNeverAddsToAClosedEpoll862andTestRegisterConnRacingUnregisterConnLeavesNoEpollEntry862(driver/internal/eventloop/wake_close_862_linux_test.go) run their hook attestHookBeforeAdd, whichRegisterConncalls before it takesc.mu(loop_linux.go:328at dce5a3c). So they do not pin that thec.closedcheck and theEPOLL_CTL_ADDare one critical section underc.mu(:346-361). A refactor that checksc.closedunderc.mu, unlocks, and then issues the ADD passes every eventloop: a Write racing Loop.Close can write the closed eventfd's number in wake(), which another socket may hold by then (data race) #862 test. The correctness review's mutant CHECKOUTSIDE (that shape) passed all three tests in 20 of 20-raceprocesses, 60/0/0, with 0 race reports (evidencelanes-20261003/D1/review-correctness-r2/logs/r2a/mut-CHECKOUTSIDE.log). Add a deterministic pin in the style ofTestTeardownMarksTheConnClosedBeforeItLeavesTheWorker784. A nil-by-default hook inside thec.musection, after the check and before the ADD, holds the section while the test callsLoop.Closeon another goroutine. The test asserts thatClosehas not returned while the hook is held, then releases it and checks that the ADD landed on the worker's own epoll instance. CHECKOUTSIDE must fail it. (from fix(eventloop): a Write or RegisterConn racing Loop.Close no longer uses the worker's eventfd or epoll fd number after shutdown closed it (celeris#862) #932 review round 2, correctness NIT) (test(eventloop): hooks and mutant pins for the teardown orders, the WriteAndPoll polls, shutdown, wake and RegisterConn's ADD #1050)
Added 2026-10-04 (from #938's review round 1, lane M443):
- The 14
//go:build !racefiles never run in CI: everygo testin ci.yml and test-coverage.yml, andmage test, passes-race. They hold the alloc guards of driver/{memcached,postgres,redis}, middleware/{cache,idempotency,overload,ratelimit,timeout} and celeristest (added by refactor!: the engine, protocol and driver codec packages move under internal/ (celeris#443) #938), plus the race_off files. That is how refactor!: the engine, protocol and driver codec packages move under internal/ (celeris#443) #938's commit b shipped green whilemiddleware/timeout'sTestNoTimeoutCommonPathSingleAllocfailed (3 allocs/op, want <=1; fixed in refactor!: the engine, protocol and driver codec packages move under internal/ (celeris#443) #938 by e55fa03). Add a short non-race step to the Unit job, for examplego test -count=1 -run 'TestAllocBudgets|TestAllocBudgetsPreparedExec|TestAllowUnderLimitZeroAlloc|TestNoTimeoutCommonPathSingleAlloc|TestNewContextZeroAlloc' ./driver/... ./middleware/... ./celeristest/, about 10 s in a container. (from refactor!: the engine, protocol and driver codec packages move under internal/ (celeris#443) #938 r1, MINOR) (ci: SKIP tallies fail on a joined SKIP line, non-race allocation guards, drain-order step with memlock raised, -tags http2legacy step, Redis with a password for the auth tests (v1.6.0 polish items) #1063) -
TestServerMixedAsync_ConcurrentNoCrosstalk(server_async_test.go:134) flakes on darwin with "connection reset by peer" on the std engine:go test -count=10 -run 'TestServerMixedAsync_ConcurrentNoCrosstalk$' .failed 2 of 10 at main a43aa1d and 1 of 10 at refactor!: the engine, protocol and driver codec packages move under internal/ (celeris#443) #938's head, on darwin/arm64. It passes on Linux, where CI runs it. (from refactor!: the engine, protocol and driver codec packages move under internal/ (celeris#443) #938 r1, NIT) (test: replace timing-based waits in the wire and shutdown tests (Refs #890 items 22, 25, 26, 53, 63, 69) #1055) - The
api/golden files (ci: check the exported API against golden files in api/, so every API change shows in review (celeris#443) #914) do not show five kinds of breaking change, whichapi/README.mdlists for review by hand: a struct that stops being comparable; the first unexported field added to an all-exported struct; reordered exported fields (both break unkeyed literals); a renamed package clause; methods promoted through an unexported embedded interface of another module. Each fix changes what apidump reports, so each comes with amage apiregeneration; the plan's estimate is 2-3 h for all five. (from ci: check the exported API against golden files in api/, so every API change shows in review (celeris#443) #914's review MINORs, decision D-F in refactor!: the engine, protocol and driver codec packages move under internal/ (celeris#443) #938) (apidump: a golden file names the package clause, so renaming it shows in api/ (v1.6.0 polish item) #1072, apidump: a struct that is not comparable says so, so a struct that stops being comparable shows in api/ (v1.6.0 polish item) #1073, apidump: a struct with an unexported field says so, so the first one added to an all-exported struct shows in api/ (v1.6.0 polish item) #1074, apidump: the fields of an all-exported struct are listed in order, so a reorder shows in api/ (v1.6.0 polish item) #1075, apidump: the methods an unexported interface supplies from another package are listed, so changing it shows in api/ (v1.6.0 polish item) #1076) - Coverage's root step (test-coverage.yml) runs
./internal/engine/iouringin the samego test -race -coveras the rest of ci.yml's unit set, at the runner's 8 MiB memlock, and that package sometimes cannot set up a ring:io_uring_setup: cannot allocate memory (likely RLIMIT_MEMLOCK ...). It failedTestHandoffHasNothingInFlightat refactor!: the engine, protocol and driver codec packages move under internal/ (celeris#443) #938's ee8ae8b (run 37162905087, 2 of 2 attempts) andTestAsyncH2CUpgradeOnPromotionLeavesH1StateToTheGoroutineat refactor!: the engine, protocol and driver codec packages move under internal/ (celeris#443) #938's 998d7bf (run 37169188620, attempt 1; the rerun passed). Neither commit touches that package's code. ci.yml's Unit job runs the package in a step of its own and has not hit it. The likely cause, not proven, is the other test binaries of the samego testholding io_uring memory at the same time. Fix: run./internal/engine/iouringin its own step in test-coverage.yml as ci.yml does, with its own profile and upload, and keep the package-set check in step. (from refactor!: the engine, protocol and driver codec packages move under internal/ (celeris#443) #938 r2, CI) (refactor!: the engine, protocol and driver codec packages move under internal/ (celeris#443) #938) -
middleware/metricsTestRequestSizeLabelsSurviveNextRequest/std(request_size_labels_742_linux_test.go) failed once in the Coverage job (run 37961436166, attempt 1, on fix(h2): outbound flush paths: buffered DATA through the write queue, both windows on every DATA path, no body-sized copy per WINDOW_UPDATE #965's head a08b27a: series[PURGE|tenant-bbbb TRACE|tenant-aaaa],MKCOL|tenant-ccccmissing; attempt 2 and the Unit job passed). The test reads the registry as soon as the third response arrives, and on std the response can reach the client before the middleware observes the request size after the handler chain returns, so the last request's series can be missing. Make the test wait for the series (pollseriesOfto a deadline) instead of reading once. (from fix(h2): outbound flush paths: buffered DATA through the write queue, both windows on every DATA path, no body-sized copy per WINDOW_UPDATE #965 CI) Recurred once more: Coverage job, run 37977744168 on fix(adaptive): test crash on a missing io_uring standby, spurious error-rate reverts, listener gap at a switch #967's head 98d7678 (same series,MKCOL|tenant-ccccmissing). And again: Coverage job, run 37997365616 on ci: forbid silent skips in the unit, adaptive and driver steps, run the #592 rig's io_uring half, and lift the ramp quarantine #961's head f14291b (job 114046716203, same test,stdarm). (test(metrics): wait for the middleware to record before reading series (flake on std) #995)
Added 2026-10-09 (lane C1, PR #961; celeris#684 / celeris#709 follow-ups that the PR leaves out by maintainer decision):
-
The three alloc-count tests in
middleware/sse(TestBrokerFanOutFormatsOnce,TestRingBufferAppendAllocs,TestSlowClientSendNonFullQueueAllocs) skip at runtime under-race("alloc counts unstable under -race / coverage / -short"), and everygo testin ci.yml is-race, so they never run in CI. They are in the root step's allow-list in ci: forbid silent skips in the unit, adaptive and driver steps, run the #592 rig's io_uring half, and lift the ramp quarantine #961. This is not the same set as the//go:build !racefiles in the item above; one non-race step can run both (go test -count=1 -v -run '...' ./middleware/sse/with a 3-PASS tally). (from CI: with io_uring unavailable, 16 ./adaptive tests skip and the adaptive job passes even under CELERIS_REQUIRE_UPSWITCH=1 #684 follow-ups) (ci: SKIP tallies fail on a joined SKIP line, non-race allocation guards, drain-order step with memlock raised, -tags http2legacy step, Redis with a password for the auth tests (v1.6.0 polish items) #1063) -
The three
TestAuth_Password*redis conformance tests (test/conformance/redis) skip on every CI run because the redis service in ci.yml'sdriver-conformancejob has no password (CELERIS_REDIS_PASSWORDunset); they are the only skips that job's tally allows in ci: forbid silent skips in the unit, adaptive and driver steps, run the #592 rig's io_uring half, and lift the ramp quarantine #961. Give the service a password (a second service container, or--requirepassthrough the image's command) and make the three tests required. (from CI: with io_uring unavailable, 16 ./adaptive tests skip and the adaptive job passes even under CELERIS_REQUIRE_UPSWITCH=1 #684 follow-ups) (ci: SKIP tallies fail on a joined SKIP line, non-race allocation guards, drain-order step with memlock raised, -tags http2legacy step, Redis with a password for the auth tests (v1.6.0 polish items) #1063) -
Twelve helpers in
internal/engine/epollend int.Skipf("epoll engine unavailable: %v", err)on anyNewerror (transplant_handoff_failure_test.go:268,transplant_accounting_test.go:183,park_stale_clock_test.go:54,park_retracts_residue_test.go:102,adopt_suspended_loop_test.go:110,sweep_engine_test.go:61,async_largebody_test.go:71,137,async_reqcount_test.go:62,async_reuse_test.go:55,backpressure_test.go:92,transplant_standby_suspend_test.go:104).Newfails only on config validation, so this is the sendfile trap (test: run the epoll sendfile e2e tests, and let CI forbid the celeris#592 rig's io_uring skips (celeris#684) #702) waiting to recur: none fires today, and ci: forbid silent skips in the unit, adaptive and driver steps, run the #592 rig's io_uring half, and lift the ramp quarantine #961's root-step tally would fail the step if one did. Replace each skip witht.Fatalf. (from CI: with io_uring unavailable, 16 ./adaptive tests skip and the adaptive job passes even under CELERIS_REQUIRE_UPSWITCH=1 #684 follow-ups; the count was ten on 9f4d89b) (epoll: a failed New or bind in a test helper fails the test instead of skipping it (v1.6.0 polish item) #1089) -
The SKIP tallies added in ci: forbid silent skips in the unit, adaptive and driver steps, run the #592 rig's io_uring half, and lift the ramp quarantine #961 (root, adaptive, middleware, driver conformance, conformance) grep plain
-voutput for^[[:space:]]*--- SKIP:. A test that writes to stdout without a trailing newline joins the next--- SKIP:line to its own, the line goes uncounted, and the check fails open (the PASS-by-name tallies fail closed in the same case). Not seen in any log; the repo's older tallies share the pattern. Fix: countAction == "skip"events fromgo test -json, or have the tally also grep unanchored--- SKIP:and compare the two counts. (from ci: forbid silent skips in the unit, adaptive and driver steps, run the #592 rig's io_uring half, and lift the ramp quarantine #961 review r1, NIT) (ci: SKIP tallies fail on a joined SKIP line, non-race allocation guards, drain-order step with memlock raised, -tags http2legacy step, Redis with a password for the auth tests (v1.6.0 polish items) #1063) -
No CI job builds or tests with
-tags http2legacy(x/net's own HTTP/2 server instead of the go1.27 net/http wrapper), which x/net offers as the opt-out. fix(std): Shutdown sends h2c connections GOAWAY at the start of the drain and tells a live overlapping caller whose budget ended it #968 adds a build-tag split for the std engine's h2c front end (h2c_base_wrap.go/h2c_base_legacy.go) and its tests pass under the tag locally (go test -tags http2legacy ./internal/engine/std/), but nothing in CI keeps it that way. Add a std-package job with the tag, or declare the tag unsupported in the docs. (from fix(std): Shutdown sends h2c connections GOAWAY at the start of the drain and tells a live overlapping caller whose budget ended it #968 review r1) Data point (fix(std): an h2c Upgrade body is bounded by MaxRequestBodySize; an h2c request's c.Context() ends when the client resets or leaves (celeris#976, celeris#949) #977 round 3): at 3ddb498 the upgrade request of an h2c connection under this tag reached the std bridge as HTTP/1 (c.Protocol()"1.1", no stream-1 cancel):TestUpgradeRequestIsHTTP2949/std/upgradeand the upgrade arms ofTestH2DepartureCancelsRequestContext949failed there, and CI, which runs no job with the tag, did not show it. The fix is in fix(std): an h2c Upgrade body is bounded by MaxRequestBodySize; an h2c request's c.Context() ends when the client resets or leaves (celeris#976, celeris#949) #977. (ci: SKIP tallies fail on a joined SKIP line, non-race allocation guards, drain-order step with memlock raised, -tags http2legacy step, Redis with a password for the auth tests (v1.6.0 polish items) #1063)
Added 2026-10-10 (lane E15, PR #973):
-
The root unit step runs
go test -race -count=1without-vand without a skip tally, so the new engine-facing tests of PR fix(engine): pin loops inside the process's CPU set and off little CPUs, report a failed pin, document the inline handler's pin (#909 #919) #973 (TestLoopsPinInsideTheAllowedSet909ininternal/engine/epoll,TestInlineHandlerChildKeepsTheProcessMask919in the root package) pass or skip there without a trace in the log. The io_uring ones (TestWorkersPinInsideTheAllowedSet909,TestUnpinnedWorkersStayUnpinned909,TestFailedPinIsLoggedOnceAndNotPretended909) and the threeTestStopped*LeavesNoPinnedThreadtests do show in the-vio_uring and celeris#905 steps. Add the epoll 909 tests and the 919 test to the celeris#905 "skipping forbidden" step with a by-name PASS tally. (from fix(engine): pin loops inside the process's CPU set and off little CPUs, report a failed pin, document the inline handler's pin (#909 #919) #973 CI review) (ci, test, comments: the tick pass leftovers of #890, #884 and #892 #1093) -
(lane E15, PR fix(engine): pin loops inside the process's CPU set and off little CPUs, report a failed pin, document the inline handler's pin (#909 #919) #973 CI)
TestUnmatchedDispatchedLikeDefaultRoute852/async-default/notfound-handler-blocks(unmatched_dispatch_852_test.go:134, engine std) failed once in the Coverage job (root unit package set under coverage instrumentation) withafter fast runs: routeAsync(GET /nope) = true, want false (inline until a run blocks)for five routes, and passed on a rerun of the failed job (run 38011596806). The std engine pins nothing, so the failure is independent of fix(engine): pin loops inside the process's CPU set and off little CPUs, report a failed pin, document the inline handler's pin (#909 #919) #973. Reading only: the test expects fast runs to stay underadaptiveBlockingThreshold(2 ms), and a slow runner under instrumentation can exceed it, promoting the route. Not on the known-flake list (test: TestHijackKeepsRequestViews/epoll-async timed out in CI: a GET /w on the epoll engine went unanswered for 60 s after an Async-route Hijack (not reproduced locally) #936, TestAdaptiveImmediatePromote_Epoll fails when one /ping run takes over 2 ms, which -race plus coverage on a hosted runner can cause #752/TestAdaptiveSettledRouteRetime592/epoll/settled fails as NOT_FIXED when its reference window was never pinned (the probe's worker held no /kv conn, ~2^-8 per run); it should VOID and re-arm #790, Codecov); the log islogs/ci-coverage-fail.login the E15 lane evidence. If it recurs, give the test a floor on the fast handler's measured time or run it outside the cover job. (from fix(engine): pin loops inside the process's CPU set and off little CPUs, report a failed pin, document the inline handler's pin (#909 #919) #973 CI) (test: replace timing-based waits in the wire and shutdown tests (Refs #890 items 22, 25, 26, 53, 63, 69) #1055)
Added 2026-10-10 (lane E5B, PR #975 review r1):
- The Coverage job's root package is close to its budget:
ok github.com/goceleris/celeristook 225.7 s on main at 3a89451 (run 38012586092) and 227.4 s at ff7b734 (run 38013576429), against the 300 s timeout, i.e. about 75% of it; PR fix(iouring): an unrecognised protocol prefix closes the conn instead of piling up in detectAccum, and a conn that stalls before detection is reaped by ReadHeaderTimeout (celeris#974) #975 (which adds about 70 server-start subtests indetect_bound_974_linux_test.go) measured 241.7 s at its first head and 237.6 s at a17bf4a (run 38016627590). The time of such tests goes to server starts, not to what they send. Each new root-package test with a server per subtest takes a share of the remaining 60 s; consider a larger timeout for that step or splitting the root package's engine-facing tests into a step of their own (scripts/ci974-r2.shin the E5B lane evidence prints the times). (from fix(iouring): an unrecognised protocol prefix closes the conn instead of piling up in detectAccum, and a conn that stalls before detection is reaped by ReadHeaderTimeout (celeris#974) #975 review r1) Another data point, from fix(std): an h2c Upgrade body is bounded by MaxRequestBodySize; an h2c request's c.Context() ends when the client resets or leaves (celeris#976, celeris#949) #977 (review round 1): at its head ba51d86 the root package took 288.5 s in the Coverage job (run 38021694862) and 269.4 s in the Unit job (run 38021694844) against the 300 s timeout, while main's Coverage runs of the same hours took 226.2 to 261.0 s (runs 38020879487, 38019971169, 38017909062, 38016197769). The PR's own share is about 10 s (14 new top-level tests, 7.8 s, plus about 2.6 s from the 836 std arms it stops skipping); the rest is runner variance and main's growth since (review-r1-perf-api-security/ci-times.shin the STD lane evidence prints the times). (from fix(std): an h2c Upgrade body is bounded by MaxRequestBodySize; an h2c request's c.Context() ends when the client resets or leaves (celeris#976, celeris#949) #977 review r1) (ci(unit): the root package race step gets 600 s, not 300 s #991, ci: run the root package's slowest tests in shard jobs of their own #1018)
Added 2026-10-10 (merges of #972 and #971):
- CodSpeed
BenchmarkOneByteGrant911(internal/protocol/h2/stream/outbound_flush_911_test.go, from fix(iouring): buffer ownership at hijack and close: keep a hijacked multishot buffer, apply held completions, hold what an uncancelled recv can still fill (celeris#868, #869, #880) #962) is a ~1 us wall-time benchmark that flagged -49.5% (4MiB) on fix(static): a pre-compressed variant has its own validators, 304 and range; a strong fs.FS ETag; Root validators from the opened file (celeris#846) #972 and -30.9% (64KiB) on docs: the tier-cap row separates Adaptive's automatic choice from an explicit start; StreamWriter.Flush is not a delivery guarantee on epoll and io_uring HTTP/1.1 #971, two PRs that do not touch h2/stream, at different sizes: noise, but it forces a by-hand merge on every PR. Give it enough iterations per sample or exclude it from the CodSpeed run, then drop the by-hand justification comments. (done: ci(bench): refuse empty or mismatched benchmark sets, exclude OneByteGrant911 from CodSpeed #1048; confirm)
Added 2026-10-10 (lane E10, PR #978):
- The root package's
-racerun has almost no headroom under its 300 s timeout (ci.ymlUnit step andtest-coverage.ymlCoverage step, both-timeout=300s).ok github.com/goceleris/celeristook 241.8 s, 258.2 s and 261.0 s on three recent main Coverage runs (38019971169, 38017909062, 38020879487), and 300.39 s on fix(iouring): shutdown: the send drain follows the shutdown budget, the owed-ops drain survives a full CQ ring, a parked recv-end error is delivered, batched counters flushed #978's head a32f339 (Coverage run 38025453556, passed with 0.4 s to spare; the Unit job of the previous head took 283.3 s). fix(iouring): shutdown: the send drain follows the shutdown budget, the owed-ops drain survives a full CQ ring, a parked recv-end error is delivered, batched counters flushed #978's first head timed out there (Coverage run 38022026595:panic: test timed out after 5m0s, in the middle of a new test). fix(iouring): shutdown: the send drain follows the shutdown budget, the owed-ops drain survives a full CQ ring, a parked recv-end error is delivered, batched counters flushed #978 then cut its root tests to a slim matrix: 283.4 s on its head 09c05ad (Coverage run 38026367303). Every new root test now risks the timeout; raise the timeout, or split the package's slow tests (TestShutdownSendsTheWholeResponse19 s,TestAdaptiveSettledRouteRetime59274 s,TestAsyncChurnNoUseAfterFree60 s,TestBackpressurePauseDoesNotCancelInflightSend50 s,TestRampH1Async/Sync39 s each in the Unit job's-vlog of run 38025377139) into a step of their own. Read from CI logs, not reproduced locally. Review round 2 of fix(iouring): shutdown: the send drain follows the shutdown budget, the owed-ops drain survives a full CQ ring, a parked recv-end error is delivered, batched counters flushed #978 (heads 8beff52, 775ace3, 4259826): Coverage root 267.1 s, 291.4 s and 260.1 s, Unit root 258.2 s and 282.3 s (runs 38029982563, 38033329708, and 4259826 (Unit run 38034389482, Coverage run 38034389978)): the heads differ by a few seconds of tests but the runs by 30 s, so the runner spread alone is a tenth of the timeout. (ci(unit): the root package race step gets 600 s, not 300 s #991, ci: run the root package's slowest tests in shard jobs of their own #1018)
Added 2026-10-10 (lane E10, PR #978 round 3):
- The
internal/engine/iouringpackage's-racerun is as close to its 300 s timeout as the root package's:ok github.com/goceleris/celeris/internal/engine/iouringtook 267.0 s (Unit) on fix(iouring): shutdown: the send drain follows the shutdown budget, the owed-ops drain survives a full CQ ring, a parked recv-end error is delivered, batched counters flushed #978's head 4259826 (before round 3), 277.3 s (Unit) and 276.4 s (Coverage) at 3a726c9 (with a round-3 test that ran twice per engine and took 2 s each time; runs 38042089050 and 38042089049), 270.0 s (Unit) and 269.5 s (Coverage) at 10025b5 (runs 38043064784 and 38043064770), and 269.5 s and 271.9 s at b61ceac (runs 38045532482, 38045532516); the root package took 283.9 s and 267.2 s at 10025b5, and 281.5 s and 292.7 s (Coverage, 7.3 s to spare) at b61ceac. After the rebase onto main 489c2bb (which added root tests of its own, fix(std): an h2c Upgrade body is bounded by MaxRequestBodySize; an h2c request's c.Context() ends when the client resets or leaves (celeris#976, celeris#949) #977) the head 3e0ea84's root Coverage run took 299.06 s (0.9 s to spare) and Unit 286.6 s;internal/engine/iouring271.8 s and 270.6 s (runs 38047075465, 38047075393). fix(iouring): shutdown: the send drain follows the shutdown budget, the owed-ops drain survives a full CQ ring, a parked recv-end error is delivered, batched counters flushed #978's four root drain tests account for about 30 s of that, 20 s of itTestShutdownSendsTheWholeResponse, whose std/epoll/adaptive arms were on main already. A re-run of the Coverage job can time out on this package alone. A new test of a few seconds is now a risk in either package; split their slow tests into steps of their own, or raise the timeout. Also: a test that waits for a client to read through a small receive buffer must take its wait from the slowest host, not the laptop's: fix(iouring): shutdown: the send drain follows the shutdown budget, the owed-ops drain survives a full CQ ring, a parked recv-end error is delivered, batched counters flushed #978's first e2e budgeted 4 s for 1 MiB through 16 KiB windows and failed in CI on that alone (the same transfer is 0.2 s on the laptop, over a second for 160 KiB on a CI runner). Read from CI logs. (from fix(iouring): shutdown: the send drain follows the shutdown budget, the owed-ops drain survives a full CQ ring, a parked recv-end error is delivered, batched counters flushed #978 round 3) (fix(iouring): a conn with an outstanding SEND is not kept on the dirty list, so a stalled client no longer keeps the worker polling (#811) #994, ci, test, comments: the tick pass leftovers of #890, #884 and #892 #1093) - Data point for the root package's
-racetimeout items above (from fix(h2): a header block ending in CONTINUATION is admitted as a HEADERS frame is: stream limit, stream-ID order, stream state #984, lane H2B): the Unit job of PR fix(h2): a header block ending in CONTINUATION is admitted as a HEADERS frame is: stream limit, stream-ID order, stream state #984's head 24ea972 (run 38054744991) failed withpanic: test timed out after 5m0singo test -race -timeout=300sof the root package (running:TestStreamWriterWriteErrorAfterPeerGoneH1_875, an H1 test, so not a hang in the PR's H2 code); the same PR's heads 4342359 and f38b263 passed it (the step took 5:11 and 4:54 of wall time; main's last three Unit runs 5:04 to 5:10). fix(h2): a header block ending in CONTINUATION is admitted as a HEADERS frame is: stream limit, stream-ID order, stream state #984's two wire tests add 4.0 s and 2.3 s under-raceto that package (CI log, run 38054744991). So the package is at its budget on a normal runner and one slow run fails the job; this is the same item as above, not a new flake. (ci(unit): the root package race step gets 600 s, not 300 s #991, ci: run the root package's slowest tests in shard jobs of their own #1018)
Added 2026-10-10 (lane H2B, PR #984 round-2 review):
- PR fix(h2): a header block ending in CONTINUATION is admitted as a HEADERS frame is: stream limit, stream-ID order, stream state #984's two wire tests use fixed sleeps in the root package, whose
-racestep is at its time budget (found by lane H2B, review round 2 of PR fix(h2): a header block ending in CONTINUATION is admitted as a HEADERS frame is: stream limit, stream-ID order, stream state #984; perf minor).TestH2MaxConcurrentStreamsHeadersContinuation981sleeps 200 ms per subtest (h2_continuation_streams_981_linux_test.go:203) andTestH2StreamIDOrderHeadersContinuation981sleeps 100 ms per subtest (:305); with 16 subtests each that is about 4.8 s of the 6.4 s the two tests take under-race. Fix direction: replace the sleeps with server-side conditions (a PING/ACK barrier, since frames are processed in order, plus a grace for the pool goroutine), then re-run controls A and E to confirm the wire limit rows still fail. Under the defectwaitUntil981already runs to its timeout, so detection does not depend on the sleep. (partial: test: replace timing-based waits in the wire and shutdown tests (Refs #890 items 22, 25, 26, 53, 63, 69) #1055)
Added 2026-10-10 (lane ADDR, PR #986 review r1):
- Root package time budget. The Unit job's root step (
go test -race -count=1 -timeout=300s -v $pkgs, the root module) ran the root package in 294.35 s on main 6668c8c, 98% of its budget. On PR fix(engines): format IPv4-mapped peers and unscoped listeners as std does (#925) #986's first run (run 38056881354, attempt 1) it hit the budget:FAIL github.com/goceleris/celeris 300.062s,panic: test timed out after 5m0s, withTestUseMountedMiddlewaresAnswerTheirPaths852in flight, which passes alone. The rerun of the same job (attempt 2) passed in 261.66 s. Same mechanism as the closed probatorium Adaptive engine never promotes to io_uring at high concurrency (1024c) — default engine leaves ~14.5% throughput unused #396. Numbers:unit-timing-925.shandunit-timing-925.txtin the ADDR lane dir. Fix direction: find the slowest root-package tests on main and move them out of the-racestep, or give the package its own budget. Not changed by fix(engines): format IPv4-mapped peers and unscoped listeners as std does (#925) #986. (from fix(engines): format IPv4-mapped peers and unscoped listeners as std does (#925) #986 review r1) (ci(unit): the root package race step gets 600 s, not 300 s #991, ci: run the root package's slowest tests in shard jobs of their own #1018) - CI sets
CELERIS_REQUIRE_DUALSTACK=1on the three race steps that run the engines: an IPv4 client of a dual-stack listener is reported as "[127.0.0.1]:port", and Server.Addr of an IPv6 listener as "[::%0]:port" (epoll, io_uring, adaptive) #925 dual-stack tests (root race step,internal/engine/iouringrace step, the Adaptive job step). Done in PR fix(engines): format IPv4-mapped peers and unscoped listeners as std does (#925) #986 (draft, not merged), commit 12062f6; the root/io_uring Unit job and the Adaptive job already printed--- PASS: TestDualStackRemoteAddrMatchesStd925on their runners (run 38059527577), so the runners have IPv6 loopback. (from fix(engines): format IPv4-mapped peers and unscoped listeners as std does (#925) #986 review r1, resolved in r3)
Added 2026-10-10 (lane ADDR, PR #986 round-3 Opus verify):
- No test pins the accepted-socket argument at the two call sites of
sockaddrString(internal/engine/epoll/loop.go accept site, internal/engine/iouring/worker.go accept site): a mutant passing-1there prints the zone as a number (%12) instead of the interface name for every real link-local peer and every PR test still passes. Add an engine end-to-end test of this shape: pick an up interface with anfe80::address fromnet.Interfaces/Addrs(or add a dummy linkdz0withfe80::1234/64 nodadwhere CAP_NET_ADMIN is available), dial[fe80::...%name]:port, and compare the handler and OnConnect strings with std's AcceptRemoteAddr; skip when no such interface, required in CI behind an env var (GitHub runners' eth0 has an fe80 address). The reviewer's template:lanes-20261009/ADDR/review-r3-verify/zoned_e2e_verify_linux_test.go.tmpl. (epoll, iouring, probe: the AsyncHandlers note, the start-up multishot fields, the probe reason, the diagnostic floor and parseAddr say what is so (v1.6.0 polish items) #1079)
Added 2026-10-10 (lane E10B, PR #987 round-2 CI):
- Data point for the root package time budget item above: PR fix(iouring): the conn table's slot writes take a leaf lock RegisterConn reads under; shutdownDrivers closes the driver descriptors in parallel (#959, #866) #987's head 0bdcf25 (run 38064776993, Unit job 114250096835, attempt 1) failed
Root module — race testswithpanic: test timed out after 5m0s(running:TestStreamWriterWriteErrorAfterPeerGoneH1_875,/adaptive), with the whole step 5:16 wall; the PR changes only the io_uring driver-shutdown path and the conn-slot lock, and the same PR's first run (38061060807, d79f109) passed that step in 5:17 (main's Unit root steps: 5:08 to 5:20). Same mechanism, not new; the PR's iouring package step got its own-timeout=420sin that PR, the root step was left alone. (from fix(iouring): the conn table's slot writes take a leaf lock RegisterConn reads under; shutdownDrivers closes the driver descriptors in parallel (#959, #866) #987 review r1) (ci(unit): the root package race step gets 600 s, not 300 s #991, ci: run the root package's slowest tests in shard jobs of their own #1018, fix(iouring): a conn with an outstanding SEND is not kept on the dirty list, so a stalled client no longer keeps the worker polling (#811) #994)
Added 2026-10-10 (lane E10B, round-2 review; CI and test items):
- The CI timing figures in PR fix(iouring): the conn table's slot writes take a leaf lock RegisterConn reads under; shutdownDrivers closes the driver descriptors in parallel (#959, #866) #987's body are not produced by any script: "279 s on this PR's first run", "263 to 271 s on main's last six runs" and "the new tests add about 9 s" have no extractor under E10B/scripts/. Re-derivable from the logs: 279.037 s (run 38064776993) and 270.508 s, 269.526 s and 270.717 s (main runs 38062783373, 38060237412 and 38054122900); the 263 s figure and the "last six runs" range do not re-derive. Add a small extractor script under the lane directory and cite it, or restate the figures as the script prints them. (partial: ci, test, comments: the tick pass leftovers of #890, #884 and #892 #1093)
- The ci.yml comment above the step "internal/engine/iouring — race tests at the runner's memlock" (PR fix(iouring): the conn table's slot writes take a leaf lock RegisterConn reads under; shutdownDrivers closes the driver descriptors in parallel (#959, #866) #987) says "265 to 280 s (270 s on main, 279 s with the ... tests)", while the PR body says "263 to 271 s on main's last six runs"; align both to the script-backed figures in the item above. (partial: ci, test, comments: the tick pass leftovers of #890, #884 and #892 #1093)
-
TestCloseDriverFDsUsesTheEngineWideBound866(PR fix(iouring): the conn table's slot writes take a leaf lock RegisterConn reads under; shutdownDrivers closes the driver descriptors in parallel (#959, #866) #987) fills all 128 tokens of the process-widedriverCloseSem; if an earlier test's engine is still shutting down (startTestEngine's stop gives up after 10 s), its fill loop can block until the 1-minute watchdog fires, and for about 300 ms it stalls any background shutdownDrivers in the process. Give the test its own bound, or wait for earlier engines to drain before filling. (iouring: driver-descriptor close bound on the live path, shutdown wait without a sleep, the bound test without the shared semaphore #1057) - The final check of
TestCloseDriverFDsUsesTheEngineWideBound866(PR fix(iouring): the conn table's slot writes take a leaf lock RegisterConn reads under; shutdownDrivers closes the driver descriptors in parallel (#959, #866) #987,unix.Fstat(p[0]) == EBADF) can false-fail if another goroutine opens a descriptor in that window and gets the same number; assert that a write top[1]returns EPIPE instead, which does not depend on number reuse. (iouring: driver-descriptor close bound on the live path, shutdown wait without a sleep, the bound test without the shared semaphore #1057) - PR fix(redis): bind PubSub control writes to their conn and release the dropped conn on reconnect #985's pr-body.sh sources tally.sh from another lane (evidence/lanes-20261003/D2/scripts/tally.sh), so the body's tallies are not reproducible from under DREDIS/; copy or vendor the script into the DREDIS lane directory.
Added 2026-10-11 (lane E14, PR #994 review r1; tests the CI runner's configuration does not run):
-
With memlock unlimited and
CELERIS_IOURING_MULTISHOT_RECV=1(tier high, provided-buffer ring),TestAsyncConnRevertsOnRouteDepromotionandTestHoldReleasedWhenDrainStops/worker_loop_rearms_after_drain_stops(internal/engine/iouring) fail 10 of 10 on origin/main'sworker.goand 10 of 10 on PR fix(iouring): a conn with an outstanding SEND is not kept on the dirty list, so a stalled client no longer keeps the worker polling (#811) #994's head: apparatus failures, not product ones.async_revert_test.go:165: conn did not revert+re-promote (still stuck async)(the celeris#364 revert path is unreachable with a buffer ring; the sibling test atasync_revert_test.go:378skips for exactly that reason withif w.bufRing != nil { t.Skip(...) }, this one needs the same guard) andfd_lifetime_engine_test.go:608: the /big response was not held: the test exercised nothing. Reproduce:evidence/lanes-20261009/E14/review-r1-perf/run-ab-flaky.sh(-race -count=5x2 per arm, interleaved; logsreview-r1-perf/logs/flaky-{main,head}-{1,2}.log: PASS=0 FAIL=10 per arm for both tests). CI does not run this configuration (8 MiB memlock, base tier), so nothing is red; it matters to anyone running the package on a laptop with the default unlimited memlock and the env var. Fix shape: skip, with the reason, whenw.bufRing != nil(first test), and make the second hold its premise or skip when the response is not held. RULE 92: searched the goceleris repos for both test names and the mechanism, no tracker. (from PR fix(iouring): a conn with an outstanding SEND is not kept on the dirty list, so a stalled client no longer keeps the worker polling (#811) #994 review r1) (iouring: exit-window witness and arms, ZC shutdown pass counts, buffer-ring skips, rerunHandOff benchmark (v1.6.0 polish items) #1099) -
CI/tests:
go vet ./...does not type-check outside Linux.internal/adaptive/shutdown_switch_877_test.gohas no build constraint and usesmockEngine, which a Linux-only file defines, soGOOS=darwin go vet ./...andGOOS=windows go vet ./...fail;internal/protocol/h2/stream/window_waiters_bench_893_test.gocallssyscall.Getrusage, which Windows lacks, soGOOS=windows go vet ./...fails there andgo test ./internal/protocol/h2/streamdoes not compile on Windows. Fix: a//go:build linux(orunix) line on each file, then drop theinternal/protocol/h2/streamexclusion from thewindows-crossjob's vet step (celeris PR fix(drivers): the drivers and their nine stores compile and run on GOOS=windows through a portable net.Conn fallback (#937) #996) and consider a darwin vet step. Reproduce:evidence/lanes-20261009/WIN937/scripts/windows-build.sh <rev>(thevet:lines oflogs/windows-build-after.log). RULE 92: searched the goceleris repos forGetrusage,shutdown_switch_877_testandgo vet darwin windows test file build constraint, no tracker. (found by lane WIN937, drivers: redis, postgres, memcached and the nine middleware stores built on them do not compile for GOOS=windows, which the README offers (unix.Dup in the fallback loop, int descriptors in each conn.go) #937) (fix(drivers): the drivers and their nine stores compile and run on GOOS=windows through a portable net.Conn fallback (#937) #996) -
CI: the jobs PR fix(drivers): the drivers and their nine stores compile and run on GOOS=windows through a portable net.Conn fallback (#937) #996 adds are not required checks.
Cross-compile (windows/amd64)and(windows/arm64)(they now rungo build ./...andgo vet ./...over the whole root module) andPortable fallback (windows-latest)/(macos-latest)(the only CI run of the portable worker and the drivers' net.Conn dials, the macOS leg with-race) are inci.yml, but required checks live in the org ruleset, which a PR cannot edit; until they are listed there a red Windows leg does not block a merge, and a rename of a job silently un-requires it. Add the four job names to the ruleset's required status checks (org-admin change), keeping the macOS leg. RULE 92: searched celeris, probatorium, loadgen and docs forruleset required checks,branch protection required,required status checks,windows-latest requiredandcross-compile windows required: the only neighbours are the closed Follow-ups from #696: provider.go close-before-unregister contract text, the #691 CI step's test-group count, and its job not being required #726 (the io_uring: UnregisterConn then Close leaks the driver socket — the fd-keyed ASYNC_CANCEL misses once the caller has closed the fd, so onClose never fires and the peer never sees EOF #691 step's job not being required) and the open item in this list for the 'io_uring init-failure regression' job, which is the same org-admin change. (from PR fix(drivers): the drivers and their nine stores compile and run on GOOS=windows through a portable net.Conn fallback (#937) #996 review r2 decision, lane WIN937, drivers: redis, postgres, memcached and the nine middleware stores built on them do not compile for GOOS=windows, which the README offers (unix.Dup in the fallback loop, int descriptors in each conn.go) #937) Done (verified 2026-10-25, maintainer decision U-14): the required checks of main (ruleset 22358875) already include bothCross-compile (windows/*)jobs and bothPortable fallbackjobs; the separate item on theio_uring init-failure regressionjob stays open. -
TestDetachedCloseLeavesNoResidualHeap/io_uring/invalid-utf8(middleware/websocket/detach_residual_heap_linux_test.go:172, the client'sconn.Readwaiting for the close frame, no read deadline) hangs the whole package until-timeoutin a golang:1.27 arm64 container with--security-opt seccomp=unconfined --ulimit memlock=8388608and nothing else running: 2 of 2 runs on main b31759a (go test -run TestDetachedCloseLeavesNoResidualHeap -count=2 -timeout 100s ./middleware/websocket,panic: test timed out after 1m40s), and in the same shape in a larger run of fix(std): a detached stream (SSE) is not cut at ReadTimeout or WriteTimeout (#855) #998's branch (475 s, killed). It is not caused by fix(std): a detached stream (SSE) is not cut at ReadTimeout or WriteTimeout (#855) #998 (the test runs on io_uring and never reaches std). Two things to do: give the client read a deadline so a missing close frame fails the test in seconds instead of hanging the package, and find out why the close frame never arrives in this shape (CI passes it; the shape is the 8 MiB memlock one the cluster also uses). Same family as the earlier root-racehang inTestRouteDoesNotRunAfterAMiddlewareAnsweredNativeEngines927/io_uring/behind-etag. RULE 92: searched celeris by the test name: no tracker. Evidence: lanes-20261009/SSE855/logs/r4-ws-heap.log, r4-ws-heap-main.log. (from std engine: an SSE stream ends at the server's ReadTimeout/WriteTimeout (30 s in the bench adapter) while epoll, io_uring and adaptive keep it open; the docs say WriteTimeout: -1 is enough on every engine #855) -
test gap (found by the WIN937 / fix(drivers): the drivers and their nine stores compile and run on GOOS=windows through a portable net.Conn fallback (#937) #996 round-3 verify):
TestPortableLoopCloseOrderdoes not catch a revert of theLoop.Closeordering on the portable worker (conn.Close then fire(ErrLoopClosed) on the Loop.Close goroutine); onlyTestPgLoopCloseDuringStreamingDoesNotPaniccatches it and that test is built only on!linux && unix, so windows-latest and the Linux Unit job would pass the revert. Make the order test discriminate on every platform that compiles the portable worker. Nit in the same PR: inportableConn.shutdownConntheelse if err != nil { c.fire(err) }branch for a placeholder (conn == nil) is dead code (the placeholder has no onClose); delete it or fix the comment. (from fix(drivers): the drivers and their nine stores compile and run on GOOS=windows through a portable net.Conn fallback (#937) #996 review r3) (test(eventloop): the number-taking helper on every unix, a Loop.Close order test for the portable worker, and the CI list that requires them #1051) -
TestIouringH2CUpgradeAfterDirectBodyTail908and...TailAsync908(internal/engine/iouring/h2c_tail_908_linux_test.go): the K1 preconditiont.Fatalf("the 101 did not go out chained to a recv") runs before the assertion on the defect (h2Streams908), so a regression points at the linking, and an unlinked 101 plus standalone re-arm (a correct alternative) also fails them. Poll and recordRecvLinkedArms/RecvArmDeclinedwithout failing, runh2Streams908, then report the precondition withErrorf. (from PR epoll,iouring: one handling of an HTTP/1 verdict for the recv path and the transplant replay; a pipelined h2c upgrade behind a direct-body request #1015 review r2, minor) -
internal/adaptive
TestPromotionKeepsAListenerUpWhenTheIncomingEngineListensLate683flaked in two independent lanes without any adaptive change in them: theAdaptive engineCI job of redis: run the PubSub rig tests on the epoll and io_uring engines (v1.6.0 checklist item) #1017 (passed on rerun) and a full-package run in the ADTELEM lane (adaptive: log the per-bucket error delta with the engine switch line (v1.6.0 polish item) #1024: failed once, then passed 10 of 10 alone on both the fixed tree and 4abbac2). Cause not found; look at the 150 ms late-listener timing against thelistenWatchpoll and scheduler load. (orchestrator, 2026-10-13) (partial: adaptive: the #683 late-resume tests assert the measured switch and hold up under load #1096) -
internal/engine/epoll/adaptive_abort_844_linux_test.go(thepkg844 == "iouring"arm of its setup, about lines 270-285): when the forced switch to io_uring has not landed after 20 attempts it callst.Fatalf("the forced switch left %v active"), where its twininternal/engine/iouring/adaptive_abort_844_linux_test.gocallsSkipUnlessIOUringUnusable844, which skips where io_uring cannot be used here at all and fails otherwise. So on a host whose io_uring cannot be set up (a seccomp profile, memlock under the floor) the epoll copy reports a failure the iouring copy skips. Found by the sweep of v1.6.0 polish: adaptive engine #886 item 11 in PR v1.6.0 polish: adaptive tests: io_uring refusal seam, floor test on the probe's answer, early-return hold, standby-failure verdicts, io_uring stall bar #1027 (the controller-driven adaptive tests); read from the code, not run. RULE 92 search of the four repos (adaptive_abort_844, pkg844, SkipUnlessIOUringUnusable, forced switch) found no tracker. The directory is another lane's, so v1.6.0 polish: adaptive tests: io_uring refusal seam, floor test on the probe's answer, early-return hold, standby-failure verdicts, io_uring stall bar #1027 leaves it. -
CI never runs the io_uring base tier (
CELERIS_MAX_IOURING_TIER=base;grep -rn MAX_IOURING_TIER .github/finds nothing). PR iouring: the run loop retries a full CQ ring's EBUSY, the single-shot accept is non-blocking, the pool wait gates HTTP/1 and detached writes, the async hijack refusal comes first (v1.6.0 polish items) #1026 item 53 made the base tier's accepted sockets non-blocking (they were blocking), and the state that change makes reachable on it (a partial inline write of a detached stream leaving a remainder inwriteBuf:TestShutdownDetachedPartialWriteIsFlushed806) is covered only by local runs. Add a CI job, or a matrix leg of the io_uring job, that runs the internal/engine/iouring package, and the root package's Shutdown/Detach/SSE/Stream/WebSocket/Hijack/Drain tests, withCELERIS_MAX_IOURING_TIER=base(local runs of PR iouring: the run loop retries a full CQ ring's EBUSY, the single-shot accept is non-blocking, the pool wait gates HTTP/1 and detached writes, the async hijack refusal comes first (v1.6.0 polish items) #1026's head on the base tier: iouring package 797 PASS / 0 FAIL / 6 SKIP,-racewith no DATA RACE). RULE 92 search of the four repos by mechanism found no tracker. (from PR iouring: the run loop retries a full CQ ring's EBUSY, the single-shot accept is non-blocking, the pool wait gates HTTP/1 and detached writes, the async hijack refusal comes first (v1.6.0 polish items) #1026 review r1) -
tests (found by PR tests: start helpers retry ENOMEM and fail rather than skip; probe retry logs its rounds (checklist #888 items 3, 31, #890 item 6) #1047 review r1, minor, item 6's siblings):
keptProbeIOUring(middleware/metrics/retained_labels_linux_test.go,logger/retained_attrs_linux_test.go,recovery/retained_attrs_linux_test.go,otel/retained_attrs_linux_test.go,adapters/retained_strings_linux_test.go,test/server/adapt_request_linux_test.go) has the same probe retry asc714ProbeIOUringunderCELERIS_REQUIRE_IOURING_WORKERS=1, silently: a CI log cannot show that it fired. PR tests: start helpers retry ENOMEM and fail rather than skip; probe retry logs its rounds (checklist #888 items 3, 31, #890 item 6) #1047 logs one line per round in the c714 copies (ws, sse, otel, test/server detach) and in session'sc731ProbeIOUring; the sixkeptProbeIOUringcopies are untouched (give them atesting.TBand the sameLogf, or fold the copies into one helper per module). Item 6's second half, narrowing fix: clone the request values a detached stream keeps (websocket Conn.Query, SSE Last-Event-ID, Context.Detach, requestid and otel context values) (#714, #717, #718) #723's "No ENOMEM retry fired" sentence to the counted runs, is an edit to a merged PR's body that PR tests: start helpers retry ENOMEM and fail rather than skip; probe retry logs its rounds (checklist #888 items 3, 31, #890 item 6) #1047 does not make. RULE 92: searched goceleris/celeris, probatorium, docs and loadgen bygh search issues(keptProbeIOUring, c731ProbeIOUring, ProbeIOUring retry, CELERIS_REQUIRE_IOURING_WORKERS probe): only Follow-ups from #723: otel traceparent clone cost, SSE OnConnect and session c.Set views, stale #714/#718 scope, ENOMEM claim wording, deliberate-red record #740 and item 6 above. -
portable worker: an
UnregisterConn(fd)that runs whileRegisterConn(fd, ...)is still opening the net.Conn is lost (found by the correctness review of PR test(eventloop): the number-taking helper on every unix, a Loop.Close order test for the portable worker, and the CI list that requires them #1051, lane DRVEVTST, minor).register(driver/internal/eventloop/portable.go) puts a placeholder (conn == nil) inw.connswhileopen()runs;UnregisterConnfinds it, deletes it and returns nil;register's second section checks onlyw.closed, not that its placeholder is still in the table, so it installs the opened conn and starts a reader: the conn stays in the table,onClosenever fires and the net.Conn is never closed. Probe (portable worker, darwin,-race):register(4242, false, open)with anopenthat parks,UnregisterConn(4242)while parked: it returned nil; afterwards the conn is in the table, noonClosewithin 300 ms, the net.Conn is open (review-correctness-r1/logs/darwin-zz.login lane DRVEVTST). Only a caller that unregisters a number while its own RegisterConn of that number is opening reaches it; decide whetherregistermust re-checkw.conns[fd] == placeholderunderw.mu(and close the opened conn if not) or whether the contract excludes the race. RULE 92: searched goceleris/celeris, probatorium, docs and loadgen bygh search issues(portable UnregisterConn placeholder, UnregisterConn during RegisterConn lost, RegisterConn concurrently UnregisterConn same fd portable) and read v1.6.0 polish: CI and tests #890's body forplaceholderandportable: only the line-158 nit about the placeholder branch ofshutdownConn, which PR test(eventloop): the number-taking helper on every unix, a Loop.Close order test for the portable worker, and the CI list that requires them #1051 resolves by deleting the dead branch. -
TestRunSurvivesARingWaitEBUSY884/submit-and-wait-timeout(internal/engine/iouring/run_ebusy_884_linux_test.go, PR iouring: the run loop retries a full CQ ring's EBUSY, the single-shot accept is non-blocking, the pool wait gates HTTP/1 and detached writes, the async hijack refusal comes first (v1.6.0 polish items) #1026) failed once on CI with its own apparatus guard:the injected EBUSY never reached a submit-and-wait-timeout wait: the test would prove nothing(injected=0, worker seen=3168, clients ok=73117 errs=0), in the Coverage job of PR iouring: driver-descriptor close bound on the live path, shutdown wait without a sleep, the bound test without the shared semaphore #1057, run 39057835360, ubuntu azure 6.17, 1 worker (RLIMIT_MEMLOCK cap:requested=4 capped_to=1); the same job passed on the rerun and on PR iouring: Write on a closing driver conn returns ErrUnknownFD, as epoll does #1058's head. The test arms the seam under 32 keep-alive clients and waits at most 3 s for 40 injections in the timed mode, but the worker takes the timed wait only when it has nothing to submit or ready, which a single worker under a continuous load can avoid for the whole window; the other two sub-tests inject in the modes the load always reaches. Make the timed mode reachable on purpose (stop the load before arming, or drive the wait from the test), or retry the arm like the io_uring: closing a driver socket with SO_LINGER blocks the worker for the whole linger time (retire closes the engine's duplicate on the worker) #735 linger tests do. Not related to PR iouring: driver-descriptor close bound on the live path, shutdown wait without a sleep, the bound test without the shared semaphore #1057/iouring: Write on a closing driver conn returns ErrUnknownFD, as epoll does #1058 (the failing run's diff does not touch run loop code). RULE 92: searched goceleris bygh search issuesandgh search prs(TestRunSurvivesARingWaitEBUSY884, "injected EBUSY never reached", submit-and-wait-timeout EBUSY test flake) and read v1.6.0 polish: CI and tests #890's body forebusy: only the EBUSY product items (the run loop's waits, ringWaitContinues' ceiling), none on this test. (found by the builder of the IODRIVER lane, PR iouring: driver-descriptor close bound on the live path, shutdown wait without a sleep, the bound test without the shared semaphore #1057 CI) -
driver/postgresTestAllocBudgets/Query_1col_1row(alloc_guard_test.go,//go:build !race) measures 14 allocs/op against its budget of 4 ({name: "Query_1col_1row", budget: 4, ...}in the guards table). It never failed because the guard only logs unlessTESTING_STRICT_ALLOC_BUDGETS=1, which no CI step sets, and the file never ran in CI before ci: SKIP tallies fail on a joined SKIP line, non-race allocation guards, drain-order step with memlock raised, -tags http2legacy step, Redis with a password for the auth tests (v1.6.0 polish items) #1063's non-race step. Measured by that step: linux/arm64 container (golang:1.27.2), 20 of 20 runs with the variable set failed on this subtest and on nothing else; the GitHub amd64 runner logs the same 14 (run 39074958353, "celeris#890 allocation guards" step). Every other guard stays within its budget, several exactly at it (4/4, 26/26, 100/100, 1/1). Decide: raise the budget to what the driver costs or bring the allocations down (driver/postgresis outside the CI lane); then setTESTING_STRICT_ALLOC_BUDGETS=1on that step. RULE 92: searched goceleris/celeris, probatorium, loadgen and docs bygh search issuesforQuery_1col_1row,alloc budget postgres,TESTING_STRICT_ALLOC_BUDGETS,alloc_guardandallocation guard: no hit but this issue. (found by the builder of the CITALLY lane, PR ci: SKIP tallies fail on a joined SKIP line, non-race allocation guards, drain-order step with memlock raised, -tags http2legacy step, Redis with a password for the auth tests (v1.6.0 polish items) #1063) -
test/server/race761_off_test.go(//go:build !race, the one of the 14 files of item 51 that holds no guard: it setsraceOn761 = false) makes the large-response tests run their full-size shape, and no CI step runs that shape.lean761()(large_response_linux_test.go:raceOn761 || testing.CoverMode() != "") is true under-raceand under coverage, which are the only ways CI runstest/server, so the epoll/io_uring: an HTTP/1.1 response of 4 MiB or more sends only its headers; the body is dropped and a keep-alive client waits until its own timeout #761/epoll, io_uring: an HTTP/1.1 response body of 8 KiB or more is sent from a buffer the handler has already given back, so a client can get another request's body, another connection's included (c.JSON) #817 tests (4 call sites inlarge_response_linux_test.go, plush2_stream_context_836_linux_test.goandresponse_body_ownership_linux_test.go) never run their 16 MiB / 64 MiB bodies, andAsyncHandlers: !raceOn761(h2_stream_context_836_linux_test.go:424) is false in CI. ci: SKIP tallies fail on a joined SKIP line, non-race allocation guards, drain-order step with memlock raised, -tags http2legacy step, Redis with a password for the auth tests (v1.6.0 polish items) #1063's non-race step lists the guard packages and exemptstest/serveron purpose (its discovery check says so). Decide: add a non-race, non-coveragetest/serverstep for the large-response tests (-runby name,-v, tallied, no SKIP), or record that the lean shape is the CI contract. Not run: the full-size shape under load on a runner (its time and memory are unknown). RULE 92: searched goceleris/celeris, probatorium, loadgen and docs bygh search issuesforlean761,race761,large-response full size,raceOn761 coverage leanand761 817 non-race: only epoll/io_uring: an HTTP/1.1 response of 4 MiB or more sends only its headers; the body is dropped and a keep-alive client waits until its own timeout #761 and epoll, io_uring: an HTTP/1.1 response body of 8 KiB or more is sent from a buffer the handler has already given back, so a client can get another request's body, another connection's included (c.JSON) #817 themselves (closed), none on CI coverage of the full-size shape; read v1.6.0 polish: CI and tests #890's body fornon-race,without -raceand!race: items 51 and 56 (the guards and the sse tests) and 89, none on this file. (found by the review of PR ci: SKIP tallies fail on a joined SKIP line, non-race allocation guards, drain-order step with memlock raised, -tags http2legacy step, Redis with a password for the auth tests (v1.6.0 polish items) #1063, round 1) -
zizmor is the one CI tool pinned outside
.github/tools/go.modthat Dependabot does not track:ci.ymlrunspipx run --spec zizmor==1.30.0 zizmor ...(Lint job), and nopipentry independabot.ymlwatches it, so it stays on 1.30.0 until someone remembers; the Lint job's.github/scripts/tool-pins-check.sh(celeris#1069) does not cover it. Found by the review of celeris#1069. Fix: arequirements-ci.txt(or similar) withzizmor==1.30.0plus apipentry for that directory,pipx run --spec -r/pip install -rin the step, and a check-4 intool-pins-check.shthat the workflows name no==version of their own. Verify with the Dependabot CLI that the entry proposes a bump. Needs a decision first: whether a Python manifest in the repository is acceptable for one tool. Searched the issues of celeris, probatorium, loadgen and .github bygh search issuesforzizmor,pipx,package-ecosystem pipandhand-pinned tool dependabot: no hit. -
TestStreamedResponseIsDelivered/adaptive/async-route/close(test/server/large_response_linux_test.go:497) failed once in the Coverage job (go test -race -covermode=atomic, GitHub runner, kernel 6.17.0-1022-azure, 4 vCPU amd64) of PR epoll, iouring: a detached conn whose peer takes none of its queued bytes for WriteTimeout is closed (v1.6.0 polish items) #1077 at ce4706d (run 39125806725, job 117670563070; the same test passed in the Coverage job of the same PR's earlier heads):adaptive/async-route streamed 8 MiB close: received 2681685 of 8388608 body bytes (2681856 bytes in all), then EOF, 0.12 s into the subtest, i.e. the server closed a non-detachedConnection: closeStreamWriter response mid-flush. Not reproduced in 40 runs per tree ofgo test -race -count=40 -run TestStreamedResponseIsDelivered/adaptive ./test/serverin a golang:1.27.2 linux/arm64 container on 7b02cae and on the PR head (lanes-20261009/SSENONREAD/final2/streamed-flake-*.out). Triage: whether the mid-flush close is the closing-drain bound or a short-write path on adaptive over epoll with an async route. RULE 92: searched the issues of celeris and the markdown of probatorium, docs and loadgen for the test name and the message: no tracker. Log kept aslanes-20261009/SSENONREAD/final2/ci/coverage-ce4706d-run39125806725-job117670563070.log. (from the SSENONREAD lane) -
test/integration: the epoll arms of three tests skip when the engine's
Newfails, the same trap as item 59 (PR epoll: a failed New or bind in a test helper fails the test instead of skipping it (v1.6.0 polish item) #1089 fixed theinternal/engine/epollones):test/integration/pipeline_test.go:40(t.Skipf("epoll engine unavailable: %v", err)afterepoll.New), andtest/integration/slowloris_synthetic_test.go:74and:213(t.Skipf("%s engine unavailable...")aftertc.newEng(cfg);:74also has atc.minSkipskip that may be a host's).Newfails only on config validation, so on the epoll arms a skip hides a defect. Turn the epoll arms' skips intot.Fatalf(keep the io_uring arms' skips for v1.6.0 polish: middleware #888, and leave the adaptive, sse and session skips to their lanes). Measured on 6a6b07f by grep; not changed by any PR. RULE 92: searched the four goceleris repos, issues and PRs, by mechanism (logs/search-integration-skips-*.txtin the EPTSTDRV lane dir, scriptsearch-integration-skips.sh; positive control found v1.6.0 polish: CI and tests #890 and epoll: a failed New or bind in a test helper fails the test instead of skipping it (v1.6.0 polish item) #1089): only CI: two test groups have never run — the epoll sendfile e2e tests (Workers: 1 fails validation, the helper skips) and the #592 settled-route io_uring subtests (skip at 8 MiB) #709 and Follow-ups from #768: named CI interlocks for the #713 tests and epoll twins, and why the async standby's drain grew from ~1 s to ~11 s #797 (closed, the epoll sendfile e2e helper skip and the io_uring: a parked worker keeps a stale cachedNow; if the park outlasted ReadTimeout, conns it adopts or accepts on waking get a past timestamp and the first checkTimeouts closes them (adaptive: 6/6 promotes 31 s after a demote, 0/8 at 18 s; A/B not run) #713 interlocks), CI: with io_uring unavailable, 16 ./adaptive tests skip and the adaptive job passes even under CELERIS_REQUIRE_UPSWITCH=1 #684 (adaptive) and this issue; none names these files. (from PR epoll: a failed New or bind in a test helper fails the test instead of skipping it (v1.6.0 polish item) #1089 review round 1, lane EPTSTDRV) -
test/integration:
skipOrFailIfStandbyAborted886(test/integration/adaptive_standby_guard_886_linux_test.go:60-61) keeps its own inlineos.Getenv("CELERIS_REQUIRE_UPSWITCH") == "1"gate: the same mechanism asskipOrFailUpswitch662ininternal/adaptive, which PR adaptive tests: route every io_uring-availability skip through one gate and guard it (Refs #684) #1091 (celeris#684 item 1) reduces to one definition there. The two cannot share a function (separate packages, andinternal/adaptivetest files are not importable), so decide: leave as is (the guard of PR adaptive tests: route every io_uring-availability skip through one gate and guard it (Refs #684) #1091 covers./internal/adaptiveonly), or add atest/integrationguard likeTestAdaptiveSkipRouting684that fails on a rawt.Skip*intest/integration/adaptive_*files that bypasses this helper. Not checked: the othertest/integrationandtest/serverskips for an io_uring-availability cause outside the adaptive files (the item above on the epoll arms lists some). RULE 92: searched the issues of goceleris/celeris, probatorium, loadgen and docs bygh search issuesforCELERIS_REQUIRE_UPSWITCH,skipOrFailIfStandbyAborted886andduplicate upswitch gate(logs/search-upswitch-gate-dup.txtin the I684 lane dir): only CI: with io_uring unavailable, 16 ./adaptive tests skip and the adaptive job passes even under CELERIS_REQUIRE_UPSWITCH=1 #684 (the adaptive package), adaptive tests: TestIdleConnsFollowSwitchRevert{Sync,Async} panic on a nil e.secondary when the io_uring standby cannot start (ENOMEM); the engine correctly stays on epoll #804, CI: lift the adaptive job's quarantine of TestRampH1Sync and TestRampH1Async once #674 is on main (>= 6 green GitHub-hosted runs) #708 (closed) and the checklists v1.6.0 polish: adaptive engine #886 and v1.6.0 polish: CI and tests #890; none names this helper; read v1.6.0 polish: CI and tests #890's body forskipOrFailIfStandbyAborted886andREQUIRE_UPSWITCH: only the epoll, io_uring: a stopped engine hands its loop threads back to Go still pinned to one CPU (PinToCPU is never undone before UnlockOSThread) #905 interlock item. (from the review of PR adaptive tests: route every io_uring-availability skip through one gate and guard it (Refs #684) #1091, round 1, lane I684) -
CodSpeed main baseline is 8 days stale: of the 81 push runs of
codspeed.ymlon main (2026-09-27 to 10-20), 54 succeeded and 26 were cancelled; the last success was run 38305334372 (2026-10-12T20:36Z, sha f8fcdb2), every one of the 26 runs since was cancelled (superseded by the next push: groupcodspeed-refs/heads/main,cancel-in-progressfalse for pushes, or left queued for the macro runner), and the newest (6ab5c28, 10-20T10:28Z) is still queued. The cause is not diagnosed (the monthly budget of the first item above is one candidate). The README therefore leaves out the CodSpeed badge for now (Make the celeris, loadgen, probatorium and docs READMEs attractive for visitors: consistent live badges (Codecov, CodSpeed, Scorecard; drop Go Report Card and stale tier badges) and a stronger first screen #700 asks for it once main has a macro-runner baseline: it had one until 10-12, so add it once a main run completes after the cancellations stop and re-read it at merge time). Data:gh api repos/goceleris/celeris/actions/runs?branch=main&event=pushfiltered to.github/workflows/codspeed.yml. Searched celeris, loadgen, probatorium, docs for "codspeed": nothing tracks this (loadgen Config.MaxRequestBodySize — engine-level body size limit #96, fix: reduce buffer ring memory 8x + fix H2 stream leak on connection close #109 are about benchmark noise). Found by DOCSREAD, PR docs(readme): a first screen for visitors, performance claims with their conditions, and the v1.6.0 text (release-gated) #1092. -
s0Bind(internal/adaptive/flap_conns_per_ring_test.go:261, shared by the adaptive integration tests) waits 3 s fore.Addr()and then failsadaptive engine never bound; under CPU starvation that deadline was missed 1 time in 400 runs ofTestRevertKeepsAListenerUpWhenTheIncomingEngineListensLate683on main at--cpus 2with 2 busy loops (logs/ad-base-rev-c2-400.out) and 1 time in 400 on an intermediate tree of adaptive: the #683 late-resume tests assert the measured switch and hold up under load #1096 (logs/ad-fix2-c2-400.out); 0 in the final 800 runs of adaptive: the #683 late-resume tests assert the measured switch and hold up under load #1096's tree, and at--cpus 4never. Whether Listen was slow to bind or the poll was starved is not measured. Wait for the engine's listening condition (or a much longer deadline bounded by the test's own timeout) and log how long the bind took. Searched celeris, loadgen, probatorium, docs for "s0Bind" and "adaptive engine never bound": no tracker (celeris#684 only mentions that s0Bind's skip ignores CELERIS_REQUIRE_UPSWITCH). Found by A683FLAKE, PR adaptive: the #683 late-resume tests assert the measured switch and hold up under load #1096 (evidence under evidence/lanes-20261009/A683FLAKE/logs).
Activity
test(api): allowlist later v1.x deprecations and catch a notice that …
FumingPower3925 commentedon Oct 3, 2026
New item (lane F3, 2026-10-03), for this checklist rather than a bug issue: a test-harness robustness gap, no product defect.
engine/iouring's fd-lifetime engine tests start their io_uring engine without the ENOMEM retry the root package's helpers have (startServer761, ring_budget_linux_test.go since epoll/io_uring: Listen publishes a nil Addr() and logs "engine listening" when boundAddr fails on loop 0 #639). In the Coverage workflow's root-module step (8 MiB memlock, packages in parallel, one uid's lock count shared by every package that starts a ring) a start that lands while another package holds the one ring 8 MiB funds fails:TestHandoffHasNothingInFlight/async: iouring engine failed to start: ... io_uring_setup: cannot allocate memory (likely RLIMIT_MEMLOCK ...), thenengine did not stop within 5s(fd_lifetime_engine_test.go:363,:68), Coverage run 37085476164 on fix(h2): bound the response bytes a connection holds for the peer's window; over the budget, handlers run on the pool and send as the windows open (celeris#893) #906's head03b3cdb, which does not touchengine/iouring(the same job passed on fix(h2): clear the write queue's pending flag before the drain takes the shards, so a frame enqueued during the drain is not stranded (celeris#837) #900, fix(api): an error or panic before a taken StreamWriter sends anything is still answered, so an SSE OnConnect rejection reaches the client (celeris#835) #901, fix(cache, compress, etag, singleflight): pass a 206 Partial Content through instead of storing, encoding, tagging or sharing it like a 200 (celeris#832) #902). Givefd_lifetime_engine_test.go's start the same bounded ENOMEM retry, and count the retries in the log as Follow-ups from #723: otel traceparent clone cost, SSE OnConnect and session c.Set views, stale #714/#718 scope, ENOMEM claim wording, deliberate-red record #740 item 5 asks for the c714 probe. Every PR that adds an io_uring engine test to another package of that step makes the collision more likely.FumingPower3925 commentedon Oct 3, 2026
One item from the round-1 review of #906 (celeris#893, lane F3):
Add a per-request-body cell to
TestH2SilentClientCannotPinResponseBodies893(h2_outbound_budget_893_linux_test.go), bounded against std's growth rather than against 16 MiB. Every current cell serves one shared slice, so the test sees only the budget on copies. A route that builds its body per request (c.JSON, orc.Blobof freshly built bytes) is held by its waiting handler, one body per stream, as on net/http.The review measured 99 silent streams of a 512 KiB per-request body on a native engine:
evidence/lanes-20261003/F3/review-correctness-r1/logs/ctr/893-head-probes.log,logs/893-std-native-probes-head.log);review-perf-api-oracle-r1/logs/owned-body-probe.log).The cell would pin parity with std; the shared-body cells pin the budget.
FumingPower3925 commentedon Oct 3, 2026
New item (lane SW, 2026-10-03), for this checklist rather than a bug issue: a test that flakes on darwin, no product defect.
TestServerMixedAsync_ConcurrentNoCrosstalk(server_async_test.go:127on main 2ca5d72) opens 200 concurrent connections to a std-engine server. On darwin, wherekern.ipc.somaxconnis 128, the listen queue overflows and the client readsconnection reset by peer, whichgetBodyturns into at.Fatalf(server_async_test.go:142). Measured with the test alone, 40 runs per tree, alternating, prebuilt binaries, on a loaded laptop: main + fix(router, conn, sse): answer HEAD with the GET route and OPTIONS with the Allow list; a streamed HEAD sends no body (celeris#421, celeris#833) #841 (b73ff92) failed 1/40 and the Use-mounted swagger/pprof/debug answer 404 for their own paths: global middleware is skipped on unmatched routes unless a custom NotFound is set #852 branch (9924c6d) 2/40, every failure a reset, never a wrong body (evidence/lanes-20261003/SW/scripts/56-flake-mixedasync-isolated.sh, loglogs/56-flake-isolated.out). The routes it hits are matched routes, a path Use-mounted swagger/pprof/debug answer 404 for their own paths: global middleware is skipped on unmatched routes unless a custom NotFound is set #852 does not change. Linux CI has not shown it. Fix: bound the requests in flight (say 32), or retry a dial that is reset before the request is sent.FumingPower3925 commentedon Oct 3, 2026
A recurrence for the existing item "Replace sleep-as-synchronisation in #805's tests ... TestBackloggedPeerIsClosed's 300 ms sleep ... can fail falsely (from #818 7)", from lane F3:
TestBackloggedPeerIsClosed/adaptive/async-routefailed once in the Unit job:response 1: 2634119 of 3145728 bytes, equal=false, then EOF(large_response_linux_test.go:432; run 37093936847, job 111119890124, on fix(cache, compress, etag, singleflight): pass a 206 Partial Content through instead of storing, encoding, tagging or sharing it like a 200 (celeris#832) #902's merge refbedb2d3onto mainc88d843).evidence/lanes-20261003/F3/fix-r1/logs/ci/902-r1-unit-37093936847.log.FumingPower3925 commentedon Oct 3, 2026
A recurrence for the
engine/iouringfd-lifetime ENOMEM item (issuecomment-5964149125), from lane F3:TestHandoffHasNothingInFlight/async: iouring engine failed to start: ... io_uring_setup: cannot allocate memoryfailed #906's Coverage job again (run 37095434570, job 111124269137, headfc3474b, a head that does not touchengine/iouring). The one rerun passed. That makes 2 of the 3 Coverage runs #906 has had. Log:evidence/lanes-20261003/F3/fix-r1/logs/ci/906-r1-coverage-37095434570.log.FumingPower3925 commentedon Oct 3, 2026
One item from lane MW (PR #918, celeris#706):
test/drivercmp/*modules nor theceleris_closeprobebuild ofmiddleware/websocket, and both carry findings that are on main (2ca5d72) too: golangci-lint v2.13.2 reports 5 intest/drivercmp/redis(3 errcheck inintegrated_test.go:resp.Body.Close, twofmt.Fprintf; 2 revive var-naming,benchmarkPipeline_Celeris/_GoRedis) and 4 inserver_close_drain_linux_test.gounder--build-tags celeris_closeprobe(goimports at :22, misspellstratx3). Fix them, and add Lint-job steps for the drivercmp modules and the closeprobe tag so they stay clean.FumingPower3925 commentedon Oct 3, 2026
One item from the round-2 review of #901 (celeris#835, lane F3, correctness lens):
stream_writer_unused_error_835_linux_test.go(/committed835) cannot detect anything on the 8 native async h2c cells. As fix(api): an error or panic before a taken StreamWriter sends anything is still answered, so an SSE OnConnect rejection reaches the client (celeris#835) #901's body says for M2c, the injected500there arrives afterEND_STREAM, and Go's HTTP/2 client discards it, so those cells pass with the defect injected (evidence/lanes-20261003/F3/fix-r1/logs/901/engine-m2c-race.log: M2c 28 FAIL / 20 PASS, the native async h2c cells among the PASS). Read that route's frames with a rawhttp2.Framerand fail on any secondHEADERSafterEND_STREAM; then confirm M2c fails those cells too.FumingPower3925 commentedon Oct 3, 2026
One item seen while watching #906's CI (lane F3, round 2):
-racerun in the Unit job is approaching its timeout. ci.yml runsgo test -race -count=1 -timeout=300s $pkgs, and the root package took 166.9 s on main1b78612(job 111136512801), 181.4 s on fix(h2): bound the response bytes a connection holds for the peer's window; over the budget, handlers run on the pool and send as the windows open (celeris#893) #906'sfc3474b(job 111124269942), 209.4 s on0682241(job 111141008137) and 208.4 s on44801e5(job 111145195769). fix(h2): bound the response bytes a connection holds for the peer's window; over the budget, handlers run on the pool and send as the windows open (celeris#893) #906's two round-2 engine tests take about 9 s together (evidence/lanes-20261003/F3/fix-r2/logs/906-r2b/engine-head*-both*.log), so most of that spread is the runner's. That leaves about 90 s of headroom, and the job fails as a whole at 300 s. Either raise the root package's timeout, or split the long engine tests (large_response_linux_test.go,h2_outbound_budget_893_linux_test.go, ...) into their own named step with a tally, as other steps do.FumingPower3925 commentedon Oct 3, 2026
New item from lane MW (#920 round 2):
-racerun in no CI step.middleware/sse'sTestRingBufferAppendAllocs,TestSlowClientSendNonFullQueueAllocsandTestBrokerFanOutFormatsOnceskip whenraceEnabled, under coverage, or with-short("alloc counts unstable under -race / coverage / -short"). The CI steps that run the whole package use-race(the root module's Unit step) or-racewith coverage (the Coverage workflow), and the named-test steps (middleware/websocket: on epoll and io_uring, Conn.Query returns garbage once a frame has arrived, because the captured query strings alias the engine's read buffer (200/200 per engine) #714, Remaining #732 sites: recovery middleware slog views, otel SLICE/MAP attribute values, and #736 follow-ups (logger cost wording, Adapt MaterializeHeaders row, latency-vs-CPU claim, request_size_bytes test) #742) do not select them, so no CI step runs these gates today. The suite tallies list them as SKIP (evidence/lanes-20261003/MW/fix-r2/logs/a-suites1.log). fix(recovery, otel, singleflight, cache, sse, compress, api): copies for the request strings a log record, a span, a metric attribute set, a coalesced waiter or an SSE client keeps past the request (celeris#732, celeris#742) #920 changes what two of them count (the copy of the event's strings, celeris#732), and the lane ran them by hand without-race. Add a step that runs these tests without-race, with a tally that requires each to PASS, so a SKIP fails the step.middleware/websocket's tests useraceEnabledtoo; check them for the same skip (git grep -l raceEnabled -- '*_test.go': 5 files in sse, 4 in websocket, counting each package'srace_on/race_offpair).TestBackpressureInboundSequenceIntegritycounts its dial and handshake errors and discards them, then fails as "environment". Its 96 clients dial with a 3 s timeout and handshake under a 5 s deadline (inbound_sequence_linux_test.go:313andpause_cancel_linux_test.go:388on main56a6c1e); any failure is only counted (:327,:335) and ends the cell withenvironment: N dial/handshake failures(:472), with no error text or timing. On a loaded laptop (an interactive app at load average about 7,-race, the CI shape) it failed that way on main56a6c1ein 2 of 3 runs and on fix(websocket): Conn.Param, IP and RemoteAddr answer from the upgrade request on every engine; every test helper that starts a server fails at once with Start's error (celeris#721, celeris#706) #918's head in 1 of 3 (plus once in that head's websocket suite), in the epoll, io_uring and multishot_recv cells (evidence/lanes-20261003/MW/fix-r2/logs/c-flake-{main,head}{1,2,3}.log,fix-r2/logs/c-suites1.log). CI's websocket step runs it. Log the first few errors with their elapsed time, so a real accept or handshake defect can be told apart from host load, and consider a bounded retry of the dial. Searched the four repos: no issue covers it (io_uring resets one or two connections of 96 slightly more often than epoll (12/16 vs 6/16, p=0.037) — needs a cluster reproduction before it is believed #652 and WebSocket inbound oracle at its default size under -race: on x86 epoll one connection's client-to-server stream crawls at a 2 KiB server window for minutes, and its close misses the 240 s cap (3 of 5 processes) #783 are about this oracle's resets and slow closes).FumingPower3925 commentedon Oct 3, 2026
The
engine/std/listen_cancel_budget_test.goitem above (from #821 3, "el < budgethas a sub-microsecond theoretical margin") has now failed CI. It was on the Coverage job of #933 (head aeeccf7, run 37133330659, job 111232677229, about 2026-10-03T15:29Z,go test -race -covermode=atomic):The shortfall is 44.5 µs, so the margin is not sub-microsecond. At 28383e8 the test still creates
shutCtx(the deadline) before it takesstart(listen_cancel_budget_test.go:133-135). On a loaded hosted runner the gap between those two lines can be larger than the timer's own lateness. #933 changes onlydriver/internal/eventloop, so this is the test, not that PR. The fix is the item's: takestartbeforecontext.WithTimeout, or compare againstshutCtx's deadline. That makes it a flake fix rather than polish. The job is being re-run once. Log:evidence/lanes-20261003/D1/fix-r2/logs/ci-933-coverage-job111232677229.log.126 remaining items
engines: tests for the cross-engine small fixes, red on main (v1.6.0,…
engines: cross-engine small fixes (v1.6.0, Refs #885 #886 #889 #890)
engines: tests for the cross-engine small fixes, red on main (v1.6.0,…
engines: cross-engine small fixes (v1.6.0, Refs #885 #886 #889 #890)
ci: self-test rows that fail on the scripts of 3438221 (review r1 of #…
ci, test: the tally follows NAME/CONT and wants the probe's message, …
adaptive tests: review r1 minors and nits (v1.6.0, Refs #886, Refs #890)
engines: OnDetach rollback first, io_uring pause wake, zoned bind add…
Merge origin/main into l18-ci-tests (Refs #890)
ci: the base-tier job sets CELERIS_REQUIRE_ZONED_PEER=1, floors from …
ci: CodSpeed label gate and daily main, base-tier and full-size jobs,…
test(driver): probes for the review of the sentinel give-up, the wake…
driver/redis: sentinel give-up and Close own the close, Client.Close …
redis: dial waits out the io_uring number refusal, sentinel moves on …
tests: the driver/postgres Query_1col_1row allocation guard no longer…