Repository navigation
test: TestBackloggedPeerIsClosed failed once under -race in CI (epoll/adaptive async-route, 2 of 4 responses) #1068
Description
TestBackloggedPeerIsClosed (test/server/large_response_linux_test.go, from #761) failed once in CI and passed on the re-run of the same commit.
- Run: https://github.com/goceleris/celeris/actions/runs/39113915340 (attempt 1, job "Unit root shard b", PR epoll, iouring: deterministic tests of the H2 64 MiB cap, the pendingBytes resync and the refused-write close (v1.6.0 polish item) #1066 head 6353d16, race build).
- Failing subtest:
epoll/async-routeofTestBackloggedPeerIsClosed(4.13 s test). The log line just above it waslarge_response_linux_test.go:446: adaptive/async-route: 2 of 4 responses, then the close, so the result for an engine that is expected to deliver all 4 responses was 2. - Attempt 2 of the same run passed with no code change.
- epoll, iouring: deterministic tests of the H2 64 MiB cap, the pendingBytes resync and the refused-write close (v1.6.0 polish item) #1066 only adds tests plus a mechanical extraction of the H2 write pass in
run(); it does not touch the H1 path this test drives, so I think this is an existing timing assumption in the test, not a regression. Not yet confirmed: the test reads afterbacklogGrace761and counts whole responses before the close; under the race build on a shared runner the close may arrive earlier than the grace allows.
Needs: reproduce under -race -count=N on a loaded runner, then either widen the observable condition (wait for the queued bytes to drain instead of a grace) or fix the engine if the close really happens early.
Activity
FumingPower3925 commentedon Oct 25, 2026
Status: the mechanism is found and a fix is in review (draft #1106,
Refsonly). Correcting the issue body: the "2 of 4" line in the log is adaptive's passing line; the real failure was a truncated first response (the response of an async route that ends its connection, cut at the byte count the first flush took).Mechanism: in the epoll
serveAsyncthe dispatch goroutine storedasyncClosedBEFORE takingasyncInMuto clearasyncRun;drainDetachQueuecould look in that gap, take it for an outside close of a running handler and callcloseConnwhile the response's remainder was still queued, so the client read EOF at the bytes the socket buffers had taken. The same bug explains theTestStreamedResponseIsDelivered/*/async-route/closetruncation (8 CI occurrences on 5 runs, offsets 2681685 / 2634069 / 2634119 of 8 MiB) that is only mentioned in #890: tracked here, no separate issue. io_uring has no such gap (its closing path hands the bytes tocloseConn, which defers the close behind them).Evidence: failing-first test
TestAsyncRouteCloseSendsTheQueuedRemainder1068fails 20 of 20 with only the lock move reverted (at the CI byte counts) and passes 30 of 30 with the fix; natural rate on main with--cpus 2plus busy loops 2 of 1500, 0 of 6000 after the fix (Fisher p = 0.04, supporting evidence only; the deterministic test is the proof). Two Opus reviews (correctness and concurrency; performance and statistics) found no blocker; a similar gap inabortAsyncHandler(a handler that panics past recovery) is being addressed in the same PR.fix(epoll): the loop leaves the close of a still-running dispatch gor…
fix(epoll): the drain decides "running", "busy" and the owed close in…
test(epoll): the abort tests fail if the loop never looked while the …
epoll: an async route's closing response is sent whole; the EBUSY tim…
FumingPower3925 commentedon Oct 29, 2026
Fixed by #1106 (merged 2026-10-25), verified by failing-first reproduction in the review of the flake lane: the fix reverted fails the regression test 20 of 20, the merged tree passes 20 of 20 (#1068 epoll async closing response truncation: product bug; #1107 and #1108: test bugs). No recurrence in the CI runs since the merge. Refs #1106.