[FumingPower3925]

Refs #885
Refs #884

What a user sees (release note)

Every HTTP/1 response written after Server.Shutdown has begun carries Connection: close and its connection is closed after it, on all engines (std, epoll, io_uring, adaptive), as net/http's does: a request that was already in its handler when Shutdown began (sync or Async route) and a request that arrives on a keep-alive connection in the first 250 ms of the wait. A request pipelined behind such a response is not served (one already in the socket when the engine's gate looks at the connection is read off and dropped; see below). A handler's own Connection header on such a response is replaced by the one close; this also applies to a response to a client's own Connection: close or an HTTP/1.0 request outside any shutdown (it used to carry both lines). The 417 of a refused Expect now says Connection: close. On the native engines an HTTP/1.0 client gets the header too; std (net/http) closes the connection without a header for an HTTP/1.0 client.

The gap

Engine.Shutdown on epoll and io_uring only recorded the budget; the engines' gate (#1116, #1123, #1138) closes idle conns and marks the conns it meets close-after (H1State.CloseAfter), but only from 250 ms into the wait, and CloseAfter is read when a request begins. So a handler already running when Shutdown began (a sync handler on the loop's thread, an Async-route handler on its goroutine) and a request that reached an idle keep-alive conn in the first 250 ms answered keep-alive and left the conn open.

Design

• Engine.shutdownBegun (an atomic.Bool, per engine) is raised first thing in Engine.Shutdown, and by a loop/worker that sees Listen's context cancelled (StartWithContext cancels the context Listen was derived from, which the loops see at the same time as the watcher that calls Shutdown). It is never lowered; an adaptive engine switch does not touch it (pause/resume, not Shutdown or a cancel; guard test below).
• conn.H1State.Shutdown *atomic.Bool points at it (set in initProtocol, next to CloseAfter). The H1 response adapter reads it where a final response head is built (status >= 200): closing(status). It then emits the one connection: close (the same bytes the client-requested close emits), clears the adapter's keepAlive, and handleH1Request turns that into req.KeepAlive = false, which is the existing path to errConnectionClose: the engine flushes and closes, and ProcessH1 drops the bytes already read behind the request (a pipelined request is not served; net/http does not either). No new close path.
• Read at head build, not at request start, so a handler in flight when Shutdown began is covered, with no cross-goroutine write to a per-conn field (the dispatch goroutine owns H1State on an Async route).
• A handler's own Connection header on a closing response is dropped for the one close (cold path: only a closing response scans; no allocation unless such a header exists). Also applies to a client Connection: close request outside shutdown (it used to emit both lines).
• 1xx heads (101 is a handshake) are never touched.

Interaction with #1138 (stated plainly)

I reused the close-after decision (req.KeepAlive -> errConnectionClose, plus the same header bytes) and did not add a second close path. But the flag is a second input to it, next to CloseAfter, not a replacement: per-conn CloseAfter is a plain bool "set and read on the goroutine that runs ProcessH1", read when a request begins. The loop cannot raise it at Shutdown start for a conn whose dispatch goroutine owns it (a data race), and it is read before a handler already in flight writes its response. The gate's CloseAfter writes are untouched; #1138's tests are unchanged and green. No design conflict found.

Late pipelined requests (found by the Opus review of this PR)

A client that pipelines its next request after the handler began (it reaches the kernel while the response is still flushing) left a request in the receive queue of a conn that this change now closes after its response. On epoll the gate's first pass (closeIdleH1, #1104's read-on) read it and served it, a second response behind the Connection: close one (on main only a client's own Connection: close did this). Skipping such a conn is not enough: close(2) with unread bytes sends an RST that destroys the response in flight (a 16 MiB body arrived at 14.5-15.3 MiB, then ECONNRESET), on epoll and, measured here, on io_uring. The first pass now reads a closing conn's receive queue and discards it unparsed (discardUnreadOfClosingConn, bounded to 64 reads of 4 KiB, not for HTTP/2, detached, fixed-file or dispatch-goroutine-owned conns); the conn closes itself once its response is out. The contract test has a late-pipelined arm (second request written after the handler entered, counting handler must stay at 0, whole 16 MiB body then EOF, not a reset) on every native arm where main delivers the whole body.

Adaptive

The sub-engines raise their flags in their Shutdown, which adaptive calls once it holds e.mu; a switch that holds it (a standby build waits up to 5 s for its bind) delays the flags, and the sub-engines cannot be reached without that lock. A response written in that wait is keep-alive; a Shutdown that gives the lock up cancels Listen's context, which every loop and worker turns into the flag. Documented in adaptive.Engine.Shutdown.

What is NOT changed (known limitations)

• Streaming: a streamed response whose head went out before Shutdown keeps its keep-alive head (cannot be taken back); its conn is nevertheless ended by the shutdown (measured: at once on epoll without a wait and on std, when the gate looks at it, about 255 ms into a wait, otherwise). A chunked (non-detached) stream whose head is written after the start says close and the conn closes after its last chunk. A stream is ended by the gate (stopDetachedProducers) as before.
• SSE / detached streams: untouched (ended by the engines). An SSE stream opened after Shutdown began carries connection: close in its head, like net/http's.
• WebSocket: the 101 is a raw write in middleware/websocket (does not pass the adapter), and a 101 through WriteHeader is exempt by status. Untouched.
• Hijacked conns: the application's. HTTP/2: GOAWAY, out of scope.
• Undecided-protocol conns (no byte read): not touched, stay keep-alive. I read "stay as they are" as: nothing is done to them while undecided; once their first bytes decide HTTP/1 their response is an HTTP/1 response after Shutdown and says close, as requirement 1 says every response. If the user meant their first response should stay keep-alive, that is a one-line status/decided guard, not done.
• Pipelined bytes: read from the socket, then dropped with the conn. A request that reaches a closing conn after its handler began (late pipelined) and is in the kernel when the gate's first pass looks at it is read off and discarded unparsed (epoll and io_uring; see "Late pipelined requests"). A request that reaches a conn after that, or a conn of a loop that shuts down at once, can still turn the close into an RST that cuts a large response, as it does on main and on net/http (the wire arm asserts the whole body only where main delivers it).
• expectation417Response is a raw write without a Connection header (the conn closes anyway); predates this change.
• epoll's shutdown of a loop with nothing to wait for ends an Async-route StreamWriter response that is still in its handler (detachClosed, phase 1), so such a response is lost; predates this change (noted in the wire test; follow-up candidate).
• A handler-set Connection: close outside shutdown still does not close the conn; predates.
• The io_uring send-drain window (as opposed to the pool wait) has no wire arm of its own; the flag is engine-wide.

Disagreements with the brief

1. net/http after Shutdown: measured (plain net/http, Go 1.27.3, and std in the wire test): an HTTP/1.1 response carries Connection: close, an HTTP/1.0 response carries none (net/http adds the header only for HTTP/1.1+) though the conn closes. The brief's "(response Connection: close)" for 1.0 is what native does (a superset); the std arm only asserts it closes and has no more than one Connection line.
2. "Undecided conns stay keep-alive": see above.
3. Commit 1 declares three inert seams (H1State.Shutdown, Engine.shutdownBegun on epoll and io_uring) so the tests compile on main and fail by assertion.

Evidence

Commit 1 on top of main (tests + the three inert declarations) vs commit 2:

tests	commit 1 (main behaviour)	commit 2
internal/conn TestShutdownFlag* (13 tests, 27 incl. subtests)	9 FAIL / 4 PASS (the 4 are controls: flag down changes nothing, head sent before Shutdown not rewritten, CloseAfter alone still closes, 101 untouched)	all PASS
test/server TestShutdownConnCloseContract884, 14 arms (the late-pipelined arm needs the whole 16 MiB body then EOF on the native pool-wait arms; the arm without the discard fails on epoll and, measured, on io_uring) (std x2, epoll, io_uring, adaptive-epoll, adaptive-io_uring; pool-wait/direct; sync/async-route; 11 request shapes each, in flight and in the window)	12 FAIL (every native arm) / 2 PASS (std: it already does)	14 PASS
internal/engine/epoll and iouring TestShutdownCallRaisesTheFlag884 (inline/async), TestListenCancelRaisesTheFlag884	FAIL x3 each	PASS
test/integration TestEngineShutdownCallAloneSaysClose884 (4 arms), TestAdaptiveSwitchKeepsKeepAlive884	4 FAIL / guard PASS	PASS

-race: -count=20 -cpu 2 on the contract test (test/server), the engine-level tests (epoll, io_uring), conn tests, integration tests: all pass. Whole root package, test/server, internal/conn, internal/engine/epoll, internal/engine/iouring, internal/adaptive under -race once: pass. golangci-lint v2.14 (linux): 0 issues. GOOS=windows/darwin/linux go vet ./... clean; mage apiCheck matches (no exported API change).

CI notes: test/integration is not run by the CI workflow (only its skip guard is), which is why the Engine.Shutdown-alone and Listen-cancel-alone checks also exist in the engine packages. The new test/server and engine tests contain no t.Skip.

Mutants (each must go red)

rule	mutant	red in
1 close on every response	flag ignored at head build	conn: 6 tests
	stream head / sendfile head ignore the flag	StreamHead..., Sendfile...; AnswersCloseOnEveryShape
	initProtocol does not wire the flag (epoll / io_uring); loop/worker not given the engine's flag (epoll / io_uring)	wire: all arms of that engine
	Engine.Shutdown does not raise it (epoll / io_uring)	TestShutdownCallRaisesTheFlag884, integration
	loop/worker does not raise it on a cancelled context (epoll / io_uring)	TestListenCancelRaisesTheFlag884
2 busy / window	flag read at request start only (not at head build)	RaisedInsideHandler (+ wire in-flight arms)
3 reuse #1138	CloseAfter removal	AndCloseAfterAreOneDecision884
4 header	dedupe off (WriteResponse / streamed head / sendfile head), dedupe case-sensitive, 1xx not exempt	HandlerConnectionHeader, StreamHeadConnectionHeader, Sendfile..., Interim..., HandshakeIsNotClosed
5 pipelined	close not propagated to req.KeepAlive (plain / streamed path)	Pipelined..., RequestBodyInTwoReads, AnswersCloseOnEveryShape
5b late pipelined (review)	epoll: closing conn's queue not discarded / read on and parsed; io_uring: not discarded	contract pool-wait/sync arms (reset after 14.5-15.3 MiB / /v/second served)
4b 417	no Connection: close	TestExpectationFailedSaysClose884
6 std	n/a: std arms PASS on commit 1 (the contract holds today)

Not killable deterministically: none left among those listed (the loop-side setter is killed by TestListenCancelRaisesTheFlag884; the server-level cancel arms alone would only see a microsecond race).

Benchmarks (darwin arm64, Apple M5; main b10b4b4 vs branch, interleaved, -count 12 each, benchstat)

BenchmarkProcessH1Respond884 (new, a handler writing a response through the adapter; ns/op, allocs/op identical on all arms: 64 B, 1 alloc):

goos: darwin
goarch: arm64
pkg: github.com/goceleris/celeris/internal/conn
cpu: Apple M5
                                        │     base     │               branch                │
                                        │    sec/op    │    sec/op     vs base               │
ProcessH1Respond884/fused-200-json-10     77.44n ±  9%   74.97n ±  9%       ~ (p=0.660 n=12)
ProcessH1Respond884/general-200-3hdr-10   98.08n ±  9%   92.93n ±  5%       ~ (p=0.078 n=12)
ProcessH1Respond884/head-200-10           83.76n ±  5%   78.30n ±  6%  -6.52% (p=0.024 n=12)
ProcessH1Respond884/204-10                74.96n ± 24%   73.94n ± 11%       ~ (p=0.977 n=12)
geomean                                   83.10n         79.69n        -4.10%

Existing: ProcessH1Backlogged890 55.1 -> 51.9 ns (-5.8%, p=0.003), H1StreamWrite884 write 35.95 -> 36.47 (), flush 1.905 -> 1.988 (), 0 allocs. Root ContextString ~, ContextBlob +7.5% (p=0.004), HandleStreamFull ~: ContextBlob and HandleStreamFull run on a test recorder and do not execute the changed code (they never reach internal/conn); read as code-layout noise of the root test binary; allocs identical (80 B/2, 288 B/6).

Cost on the hot path with the flag wired and down: one nil test and one atomic load of a word written once, per final response; no new allocation. (Not "unchanged": the last interleaved run, main 92eb215 vs this branch, read +3.05% geomean on ProcessH1Respond884 (fused-200 +4.9%, p=0.02; the other arms ~), the earlier run -4.1%; the machine's noise is of that size. Allocs identical.)

Draft; not for merge. Milestone v1.6.0.

CI note

The first two heads failed Coverage / Unit steps because the new tests assumed 2 ring workers: with the runner's 8 MiB memlock io_uring starts 1 worker, and a gated sync handler (or an inline "async" test handler) then holds the only ring so neither GOAWAY nor the flag could be seen. Fixed (soft barrier in the wire test; /gate is a real Async route in the engine tests) and reproduced/verified locally with --ulimit memlock=8388608 (workers=1). Head bc3df25 passed all 29 checks; the head after the review fixes is in the PR.

[coderabbitai]

Caution

Review failed

The pull request is closed.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

❌ Patch coverage is 89.85507% with 7 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/engine/epoll/poolwait_gate_885.go	28.57%	5 Missing ⚠️
internal/engine/iouring/shutdown_gate_884.go	85.71%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!