[FumingPower3925]

Refs #884

io_uring shutdown gate: the twin of epoll's #1116 (closeIdleH1, refusesRecv, CloseAfter), plus the ring-wait ceiling and the test-only items. Verify-first: commit 7117555 is red on main (assertions, not compile errors); the fix is 5e6b94c.

Numbering. Item numbers count every checkbox of #884 (ticked or not), the numbering the lane brief used. #1099/#1056 count unchecked boxes, which gives different numbers today. Each item is quoted by its opening words so there is no ambiguity.

Items

item (opening words)	state
73 "The HTTP/1 refusal gate of the io_uring shutdown (refuseRecv ..."	done (below)
72 "Worker.ringWaitContinues ... has no ceiling"	done
16 "Add a shutdown trial that holds the worker between a linked recv's SEND CQE and shutdown()"	trials (a) and (b) added; (c) and the keep-or-remove decision stay open
80 "io_uring: no test makes the DRAINING to SUSPENDED park's w.sweepRetract() ... discriminate"	test added, item not closed: proves the retraction is needed for a manufactured state; not that production reaches it, nor that it is dead behind the closeFDOwed gate
71 "engine exit tests (PR #1020 review r2, minor)"	io_uring half done
63 "The shutdown drain's case udAccept and cancelAccept ..."	comment only

73 (+ L3 follow-ups F1, F4)

Past the gate (250 ms into the pool wait or the send drain), closeIdleH1 (paced like the producer walk, at most 64 closes per pass) closes an HTTP/1 conn that is idle, holds nothing queued (user space, SEND/ZC in flight, kernel send queue via SIOCOUTQ), has no request in progress, and is a detached stream the gate ended (sync or Async route) or has sent no byte. gateReadsOn makes handleRecv read a request in progress on (headers in pieces, a body arriving, recvIntoBody, protocol-decision bytes) instead of dropping it; it is answered with Connection: close (H1State.CloseAfter, written only on h1State the worker owns or under a successful detachMu TryLock).
Decisions: idle close + read-on over SHUT_RD; the grace stays 250 ms into the wait (as epoll), not at the cancel; the lastActivity complaint is covered by the idle close (the stamp is not moved); a pool wait with a no-deadline ctx and a stuck HTTP/2 handler is still unbounded (as std), only the HTTP/1 clients are released.
Goes beyond epoll: for Async-promoted conns an idle conn is closed and a request in progress is read on (epoll's F2 leaves both to the end of the wait). Safety: asyncInMu (input waiting, or goroutine alive and out of its park loop), then detachMu.TryLock (free only between requests), then h1State is read under it; a failed TryLock means "in progress". The worker is the only feeder of asyncInBuf.
Known corner: a client that sends bytes after the gate on a conn the gate later closes gets an RST (close with unread bytes), as with any close(2) with unread data.

72

Ceiling 1 s of consecutive retried passes with an empty CQ ring, then Error log and w.shutdown(). "Consecutive" means: the loop pass count (tickCounter, once per pass) moved by at most one since the last retry (a larger jump means a wait succeeded between), and no more than 250 ms (ringWaitRunGap) passed since the last retry (a park or a slow callback ends the run; review r1 m1). A retry that finds a completion waiting resets the run (deliberate reading of "no reaped completion": a ring that keeps delivering completions is never given up on). With one worker (memlock-capped) the ceiling stops the only worker, as any other ring failure does.

16, 80, 71, 63

16: TestShutdownDrainEndsALinkedRecvAfterItsSendCQE884, TestShutdownDrainEndsAReArmedMultishotRecv884; red with the drain disabled (M1). 80: TestParkedWorkerRetractsResidueOfACloseWithNothingInFlight884. What is proven: the retraction is needed for the state the test manufactures in testHookCheckTimeouts (red with it removed; the other two arms stay green as documented). What is NOT proven: that production reaches that state (no traffic path found), nor that the retraction is dead behind the closeFDOwed gate. The item is not closed on either outcome. 71: TestLeaveGroupAfterExitClosesTheRingAndTheListener884, TestReportWorkerExitLeavesTheGroupBeforeTheFlag884 (red 5/5 when leaveGroupAfterExit moves after the flag), TestGoexitedWorkerLeavesTheGroupBeforeItIsMarkedExited884; not done: the epoll half, CI single-worker coverage, hotpath-table.txt (a lane artifact). SHUT_RD removal (M13) stays green on the new tests; the #848 lifecycle tests cover it. 63: known limit documented at the drain's case udAccept and in cancelAccept.

Review r1 (Opus, head 556a5f2) and what changed

• Major: gateReadsOn read cs.h1State of a promoted conn before checking asyncPromoted (data race with the dispatch goroutine's h2c switch, celeris#722). Fixed by the reorder; promotedInProgress also returns false for a conn whose writes ended (writeEnded, an atomic). The search for the same mistake found stopDetachedProducers (pre-existing, run by the gate pass) reading h1State before its TryLock on a promoted conn: fixed the same way. Regression tests TestGateReadsOnNeverReadsH1StateOfAGoroutinesConn884 and TestStopDetachedProducersNeverReadsH1StateOfAGoroutinesConn884 fail under -race before the fix (the tests-first commit) and pass after; putting the read back above the check (mutant M14) is red.
• m1: see item 72 above; TestRingWaitContinuesDoesNotCarryARunAcrossALongPass884 failed first.
• m3: the recvIntoBody term of gateCloseable cannot be told from HasPendingData on the wire: the worker sets recvIntoBody only when NextRecvBuf returned a buffer, which needs bodyNeeded > 0, which HasPendingData reports. It is an equivalent mutant at the server level; held by the unit table, and TestRecvIntoBodyImpliesPendingData884 pins the implication.
• n3: TestGoexitedWorkerLeavesTheGroupBeforeItIsMarkedExited884 is racy under its mutant (red 3 of 5); TestReportWorkerExitLeavesTheGroupBeforeTheFlag884 is the deterministic guard (5 of 5), noted in the test.
• Nits: SIOCOUTQ comment corrected (close(2) keeps the send queue; the hazard is an RST from unread peer bytes), the gate pass running before the loop reaps new completions is documented, and the cap-after-marking behaviour.

Findings, not fixed here (design extension on both engines, shared with epoll #1116)

Connection: close is not guaranteed on every response after Shutdown (net/http marks every response after Shutdown as close): (a) every response written between the cancel and the gate (the first 250 ms); (b) a request read on a promoted conn whose goroutine is busy at every look (input waiting, or inside a handler: CloseAfter cannot be written without the lock); (c) an undetected conn holding a protocol-decision prefix ("P", "PR"), which has no h1State to mark and is answered in the same handleRecv. A keep-alive client that then sends its next request gets a dropped request and a FIN or RST. Follow-up for both engines: make CloseAfter an atomic.Bool (or have handleH1Request consult an engine shutting-down flag) and set it on every live HTTP/1 conn at the cancel.
Open: item 16 trial (c), whether a kernel issues work after the descriptors close, and the keep-or-remove decision on the drain.

Evidence

• On main (f57a260 + the final test files), TestIOUringShutdownGateConns884 fails in all 5 arms (pool-wait/send-drain x sync/async-route, adaptive-iouring): 6 idle conns, 2 silent conns, 4 detached streams (no FIN), the POST/headers-in-pieces requests (and POST on an Async route) unanswered, the idle Async conn, and the drained download left open. The ceiling tests fail on main (stuck ring retried for 5 s / 6 s).
• Mutants (patches and logs kept by the lane): CloseAfter writes removed, promoted conns not closed, promoted read-on removed, read-on removed, send-drain close removed, pool-wait close removed, ended-stream close removed, undetected-conn close removed: each red. Kernel send-queue check removed: green on the server matrix, red on TestGatePassWaitsForTheKernelSendQueue884.
• Fixed tree: targeted tests -race -count=20 (unit/exit/ceiling tests, and the server matrix) green; full ./internal/engine/iouring/... -race at 8 MiB memlock green; test/server TestShutdown*/TestDrain* green; golangci-lint v2.14 clean.
• Hot path: the new handleRecv condition sits behind the existing w.refuseRecv && short-circuit; the pass runs only inside ctx.Err() != nil; ringWaitContinues runs only on a wait error or with the seam set. A/B of BenchmarkIouringRecvShapes between main and this branch (6 runs each, laptop VM) overlaps within noise (10.4 to 26.7 us vs 11.1 to 25.6 us per op).

For the reviewer

CloseAfter write sites and who owns h1State; the window between gateCloseable releasing its locks and closeConn; the blocking asyncInMu.Lock in the pass (the goroutine can hold it across the application's RouteAsync); ceiling false positives on 5.19 to 6.1 kernels; the SIOCOUTQ check counts unacknowledged bytes, so a stalled peer holds its conn to the end of the wait, as on epoll.

[coderabbitai]

Caution

Review failed

The pull request is closed.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

❌ Patch coverage is 94.68085% with 5 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/engine/iouring/shutdown_gate_884.go	95.52%	3 Missing ⚠️
internal/engine/iouring/worker.go	92.59%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!