Skip to content

epoll: the write timeout of a parked response is a minimum rate on acknowledged bytes, no ceiling, inbound bytes are not progress (v1.6.0, Refs #885) - #1119

Merged
FumingPower3925 merged 15 commits into
mainfrom
l3-ep-writeclock
Oct 28, 2026
Merged

FumingPower3925 merged 15 commits into
mainfrom
l3-ep-writeclock

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Refs #885 (item 51)

Draft: lane L3 EP-TIMEOUTS of v1.6.0, the redesign of item 51 after Opus review r1 of celeris #1116 (head 88e0eb1) rejected its IdleTimeout ceiling (it cut legitimate downloads at IdleTimeout, sendfile included, and any inbound byte refreshed it). #1116 is now 885.49 and 885.42 only. The rule's numbers are a maintainer decision; the user has decided the policy below.

History of this branch: fd2a230 tests, 7c04aaa the rule (reviews r1, r2, r3 in the probatorium evidence), then, after Opus review r3 (APPROVE-WITH-FIXES): a merge of main (fe2d765, no conflicts: the two PRs' godoc hunks did not overlap), the commit pairs below, a second merge of main (fca2281, only driver/postgres files), two comment-only godoc commits (8a23790), then, after Opus review r4 (APPROVE-WITH-FIXES, no blocker): a third merge of main and one comment/godoc/body-only commit (F11, F15, F16, F17), no behaviour change. New commits only; no rebase, no force-push.

The rule

A response a keep-alive conn has parked on the socket (the kernel did not take it whole; HTTP/1 and HTTP/2 alike) is held another WriteTimeout window if the peer acknowledged at least parkedMinAcked = 64 KiB (tcp_info bytes_acked, one getsockopt per parked conn per window) in the last window, and closed if it did not.

  • Windows start at the first sweep that finds the response parked (that sweep takes the baseline), and move on at each sample that passes. The floor is 64 KiB per WriteTimeout: 64 KiB/s at 1 s, about 1 KiB/s at the 60 s default.
  • Nothing the peer sends is progress: the rule reads acknowledged bytes only.
  • No elapsed-time ceiling. WriteTimeout <= 0 is no write timeout at any rate (ReadTimeout and IdleTimeout slide with the flush clock exactly as on main).
  • Once the rule has a baseline for a parked response it is the only bound on the conn (r3 F1): sweepWriteClock returns "done" after the rule has had its say, so checkTimeouts skips the ReadTimeout/IdleTimeout scan for that conn. ReadTimeout and IdleTimeout apply again from the end of the response (endWrite) or if the baseline is lost. A conn with no parked response is reaped by them as on main.
  • A conn whose socket reports no acknowledged bytes (not TCP, a failed getsockopt, or a count of 0, r3 F4) has the flush clock alone, as before the rule, and ReadTimeout/IdleTimeout run from it as on main. The baseline is retried at most every quarter window.
  • No new connState field (TestConnStateStaysInItsSizeClass: 448 bytes). The baseline reuses closeAcked/closeAckedKnown/closeSampled; it is forgotten where the response's clock ends (endWriteSlow, the sweep's empty-queue case, markClosing). closeSampled is now also zeroed there for a live conn, so the next response on the same conn gets its baseline at its first sweep and not after a quarter of WriteTimeout.

F1: choice made and why

Review r3 measured that ReadTimeout and IdleTimeout, which slide only with flushes that took bytes and with passed windows, still cut a steady reader that takes at least a window between two EPOLLOUT edges whenever they are not above WriteTimeout (the defaults' shape): RT=WT closed at 806 ms, RT=WT/2 at 401 ms, IT=WT at 811 ms. I chose the reviewer's option (a): exempt a parked conn whose baseline is known from the idle scan. It is the smaller change (one early return in sweepWriteClock, no new sampling, no new state) and the safer one: option (b) (stamp writeSince/lastActivity on any acked advance, sampled every min(RT,IT,WT)/4) would put a getsockopt on a quarter of the shortest of three timeouts for every parked conn. Two consequences of (a) that needed a change each:

  1. The sweep that finds the queue empty (a drain by the dispatch goroutine's own flush, which the loop does not see) used to date the idle clock at writeSince, which for a rule-governed response is the stamp of the last passed window (up to WriteTimeout old); with ReadTimeout shorter than that the conn would be reaped at the sweep that found its response whole. It now dates the drain at the sweep (TestParkedResponseDrainedBehindTheLoopStartsTheIdleClockAtTheSweep885).
  2. A stale closeSampled from the previous response gated the next response's baseline for a quarter window, during which ReadTimeout/IdleTimeout still apply: zeroed at the drain (TestParkedSecondResponseRightAfterTheFirstIsRateGoverned885).

Behaviour change against v1.5.9 and main (also in the release note): with ReadTimeout or IdleTimeout below WriteTimeout, a client that stalls a parked response is no longer closed at min(ReadTimeout, IdleTimeout) but by the rule, between WriteTimeout and WriteTimeout + 2 sweeps after the response parked (ReadTimeout 5 s, WriteTimeout 60 s: about 60 s, was about 5 s). That is the user's decision (the rate rule governs parked responses).

Maximum lifetime of a parked response (WriteTimeout T, response S bytes, R = 64 KiB, sweep s of 25 to 100 ms, which holds with ReadHeaderTimeout set (the default): the 25 ms sweep timer is armed only then, and with ReadHeaderTimeout -1 a stalled response with T = 1 s went at 5 s, on main too; rows re-checked against the final code)

client behaviour main this PR
takes nothing (zero window), or a byte at a time (the kernel acknowledges a window update, not a byte) min(T, ReadTimeout, IdleTimeout) after the last flush between T and T + 2s after parking, whatever ReadTimeout/IdleTimeout (they no longer cut it earlier), where the socket reports acknowledged bytes; a socket whose bytes_acked reads 0 (a peer that has acknowledged nothing yet, or a stack that does not fill it) has no baseline and is still cut at min(T, ReadTimeout, IdleTimeout) as on main, until a sample sees acknowledged bytes (retried every T/4)
takes fewer than R per T, an EPOLLOUT edge in every window (shallow queue), or sends bytes for ever (the flush clock restarts at every edge or inbound flush) closed at the first window end (T to T + 2s)
takes at least R per T, reads steadily between two edges, sends nothing closed at min(T, ReadTimeout, IdleTimeout) (no edge) held, whatever ReadTimeout/IdleTimeout
takes exactly R per T as the row above S/R windows: S x T / R (4 MiB at T=60 s: 64 min; 16 MiB at T=1 s: 256 s)
takes R per T and sends bytes (a pipelined GET, a PING) a pipelined request: held for ever same as without sending: the bytes are not progress. A pipelined request body still pending is bounded by ReadTimeout only once the response has been sent (F11 below)
stops after a window it passed min(T, ReadTimeout, IdleTimeout) up to 2T + 2s after its last byte (TestParkedReaderThatStops...: within 2T + 250 ms at s = 20 ms)
WriteTimeout -1 (or 0) never closed by the write clock same; ReadTimeout/IdleTimeout slide with the flush clock as on main
download at a reasonable rate with IdleTimeout < download time completes completes
socket reports no acked bytes (not TCP, failed getsockopt, bytes_acked = 0) flush clock (T after the last flush that took bytes) the same flush clock; ReadTimeout/IdleTimeout from it
a malformed request read while the response is queued (closing drain) per-byte rule, no floor (#876) unchanged: per-byte, no floor (known limit, below)
HTTP/2 response waiting for the client's flow-control window (not parked: nothing queued on the socket) ReadTimeout/IdleTimeout from the client's last frame (#1134) unchanged; no write timeout of either kind on epoll until the per-stream deadlines land
io_uring write clock from the start of a send (#1136) unchanged

Tests

"main" = origin/main 3f516f1 + the PR's test files that build there; "previous head" = fe2d765 (main merged with 7c04aaa) + the same test files; "this head" = d9f819e (the engine and test code are identical at the PR head 8a23790: the later commits are merges of unrelated files and comments). docker golang:1.27.3, --cpus 2, memlock 8 MiB, seccomp unconfined, count=1, -race. Commit pairs: 40a3d5f (tests, RED on fe2d765) then 7349549 (fix F1); 7d38eb7 (seam, no behaviour change), 1cb89a8 (tests, RED on the seam commit) then 45499a3 (fix F4); 4a9a3c0 (F9/F10: test-only hardening, no fix pair: its RED column is main's engine); d9f819e godoc.

test main previous head this head
TestParkedReaderAboveTheFloorIsNotCutByReadOrIdleTimeout885 (rig, no edge; RT=WT, RT=WT/2, IT=WT, RT=IT=WT, defaults-shaped RT=WT IT=10WT; each at 1.6x and 4x the floor: 10 cells) 10/10 FAIL (reaped at 0.31 to 0.65 s) 10/10 FAIL 10/10 pass
TestParkedReaderThatStopsIsClosedWithinTwoWindowsWhateverReadTimeout885 (3 cells: stops after two windows at 4x; RT=WT/2, RT=IT=WT, both off) 3/3 FAIL 2/3 FAIL (the RT cells; both-off passes) 3/3 pass
TestParkedSecondResponseRightAfterTheFirstIsRateGoverned885 (second response parks right after the first, RT 150 ms < WT/4) FAIL (reaped 0.15 s after parking) FAIL (153 ms) pass
TestParkedResponseDrainedBehindTheLoopStartsTheIdleClockAtTheSweep885 (added with the fix, not test-first; on the previous head ReadTimeout reaps the conn before the drain is reached, so it can only be RED for the F1 reason) FAIL (no baseline) FAIL (F1 reason) pass
TestParkedConnWithNothingParkedIsStillReapedByReadTimeout885 (guard: drained then idle on TCP; AF_UNIX rig that reports no acked bytes) pass pass pass
TestParkedConnOfAStackThatReportsNoAckedBytesIsJudgedByTheFlushClock885 (fake stack reporting bytes_acked = 0, via the ackedProbe seam) n/a (seam) FAIL (reaped 0.7 s in on the seam commit) pass
TestParkedConnIsRateGovernedOnceTheStackReportsAckedBytes885, ...AndATakesNothingIsReaped885 (guards) n/a pass, pass pass, pass
TestParkedPeerAboveTheFloorWithoutAnEdgeIsNotReaped885 (rig, now 4x) FAIL (reaped 0.81 s) pass pass
TestEngineReaderAboveTheFloorIsNotClosedBetweenEdges885 (sync, async; now WT 2 s, client 128 KiB/s = 4x) FAIL, FAIL (closed at 2.08 / 2.13 s) pass, pass pass, pass
TestEngineSlowlorisBelowTheFloorIsClosed885 x2 FAIL, FAIL (held 8 s) pass, pass pass, pass (1.04 s)
TestEnginePeerInboundBytesDoNotRefreshTheWriteClock885 none / CRLF / GET x sync / async 4 pass, GET x2 FAIL (held 8 s) 6 pass 6 pass
TestEngineStuckPeersAreClosedWithinTwoWindows885 x3, TestParkedPeerBelowTheFloorIsReapedWhateverItsFlushes885, TestParkedConnWithNoWriteTimeoutIsNeverReapedByTheRate885, TestEngineNoWriteTimeoutHoldsAStuckPeer885, TestEngineDownloadAtAModerateRateCompletesUnderIdleTimeout885 x4 pass pass pass
test/server TestEpollStaticFileDownloadAtAModerateRateCompletes885 x2, TestNonReadingH2ClientIsStillCutByWriteTimeout889 (epoll, io_uring, adaptive), #1134's h2 timeout tests pass n/a pass

Flake checks on this head, -race -count=10, the 16 new and changed tests (160 top-level runs, 280 subtests): plain 748 s, 0 failures; under six busy loops in the same --cpus 2 container 850 s, 0 failures, 0 retries used. The reviewer's F9 reproduction (10/10 failures under six busy loops) no longer fails. The reviewer's own probe (TestL3R3Probe, r3 F1/F6) on this head: rig RT=WT, RT=WT/2, IT=WT, RT=2WT all held 4 s; engine ReadTimeout 1 s, 500 ms, -1, sync and async, all held 5 s (was gone at 0.54 s for 500 ms); BAD\r\n\r\n still held 8 s (the known limit).

Margins (the client's rate over the floor of 64 KiB per WriteTimeout)

test rate WriteTimeout floor margin note
TestParkedPeerAboveTheFloorWithoutAnEdgeIsNotReaped885 320 KiB/s 800 ms 80 KiB/s 4x (was 2x) catch-up pacing
TestEngineReaderAboveTheFloorIsNotClosedBetweenEdges885 128 KiB/s 2 s 32 KiB/s 4x (was nominally 2x, 1.6x achieved, 42 to 66 KiB per window under load) catch-up pacing; under six busy loops windows 1 and 2 are always exercised at about 250 KiB each, window 3 in most runs (the test's own loop is sometimes delayed near its end: last call at 5.4 s of 6.5 s in 3 of 10 runs; no empty reads, so it is not a supply stall)
the F1 table (10 cells) 1.6x and 4x 600 ms 109 KiB/s 1.6x and 4x catch-up pacing; a run in which the client itself took under 1.25x the floor in a window before an engine close is retried (up to 3), reported in the log
stops / second-response / unreported-stack tests 4x 600 to 800 ms 4x
TestEngineSlowlorisBelowTheFloorIsClosed885 40 KiB/s 1 s 64 KiB/s 0.63x (below the floor: load can only lower it)
TestEnginePeerInboundBytesDoNotRefreshTheWriteClock885 10 KiB/s 1 s 64 KiB/s 0.16x
TestParkedPeerBelowTheFloorIsReapedWhateverItsFlushes885 40 KiB/s 600 ms 109 KiB/s 0.37x
TestEngineDownloadAtAModerateRateCompletesUnderIdleTimeout885 2 MiB/s 60 s / -1 1 KiB/s about 2000x

F10: the engine tests' goneAfter885 now waits until ActiveConnections >= 1 before starting its clock (an accepted conn the engine has not counted reads as "gone") and the closed-within tests also assert a lower bound (about one window), so they cannot pass vacuously.

Verification

  • epoll package, -race -count=1, whole package: ok in 377 s on this head (347 s on the merge tree before this round, +30 s), against the root step's 540 s per-package limit; measured with other agents' containers sharing the host, so a CI runner needs headroom; the new tests add about 28 s of F1 cells, 8 s of F4 tests and a few seconds of F9 changes per count.
  • test/server -race -run '889|885|WriteTimeout|ReadTimeout|IdleTimeout|WriteRate': ok, 0 failures (h2 timeout tests from engines: ReadTimeout and IdleTimeout skip an HTTP/2 conn that has a stream open, proposal with the policy decision pending (v1.6.0, Refs #889) #1134 and the sendfile download included).
  • golangci-lint v2.14 (CI pin): 0 issues on ./internal/engine/epoll/... and the root package. GOOS=windows, GOOS=darwin go vet ./... clean; linux/arm64 vet of epoll and test/server clean. mage apiCheck: api/ matches (config.go changes are comments only).

Comparison with the other engines

  • std: WriteTimeout is one absolute deadline for the whole response: every download longer than T is cut, at any rate. epoll has never done that for a response in progress, and the docs say so.
  • io_uring: unchanged. A live conn with a SEND outstanding is closed when now - lastActivity > WriteTimeout, where lastActivity is restamped at the last read, the start of a send from idle (iouring: the write clock of a conn starts when its send starts, so a quiet h2 stream is not cut at a chunk (v1.6.0, Refs #889) #1136) and a completed send. A client that completes one SEND per T, or sends any byte, holds it for ever; there is no rate floor and no acknowledged-bytes signal for a live conn. The io_uring side would take the same rule in a follow-up (tcp_info sample in its sweep).

Known limit (not changed here)

A malformed request read while a response is parked is answered as a request error and makes the conn a closing conn (and so does any request read behind a backlog over maxPendingBytes, 4 MiB): its drain keeps the #876 per-byte rule (noteClosingAcked, no floor, documented in config.go), so a client that sends such a request and then acknowledges a few bytes per window is held by the closing drain, not by this rule. Measured on this head and on main: BAD\r\n\r\n 50 ms after the GET, client reading 4 KiB/400 ms (10 KiB/s) of a 3 MiB response, WriteTimeout 1 s: held 8 s (62 KiB read); a bare CRLF every 2 s is closed at about 1 s because the first window passes before the first CRLF (config.go no longer claims a bare CRLF cannot hold the conn). Applying the floor to the closing drain changes #876's tests and decision (a slow reader of a Connection: close response); left for the same maintainer decision.

Findings, not fixed here

  • Half-close after the GET (r3, main too, not this PR): a client that half-closes (SHUT_WR) right after its GET leaves the active-connection count within 20 to 27 ms having read only 4 KiB of a 3 MiB response (reviewer, main and the PR; reproduced on this head: gone at 20 ms, 4 KiB read, sync and async). Either the response is truncated on peer half-close or the closing conn leaves the active count at once; apparatus-ambiguous, not investigated. Worth its own issue: possible truncated response.
  • F11 (r4, minor, measured): a pipelined request body behind a rate-held response is not bounded by ReadTimeout until the response ends. The early return in sweepWriteClock skips the idle scan, and a pipelined request whose head is whole and whose body is pending has no header deadline (ReadHeaderTimeout still bounds a partial head: measured, closed at 1.87 s with ReadTimeout 1 s, the header check runs first). Reviewer's probe, a floor-reader at 6x of a 64 MiB response plus a POST with 10 of 1000 body bytes, ReadTimeout 1 s: main cut it at 1.34 s, this head held it for the whole 10 s run; with a 4 MiB response that drains at about 6 s, ReadTimeout applies again from the drain, gone at 6.94 s on both trees. The same early return suspends the async-handler overrun check of h2ServerOwes (read, not measured). Bounded by the accepted policy (the client must keep acknowledging the floor), adds no new resource class; comment and godoc now say so. A stricter variant (keep the idle scan while a request body is in progress) would reinstate F1 for a client that downloads and uploads on one conn; left as a follow-up if the maintainer wants it.
  • F12(4) (r4, nit, read; the PR's own commit 7c04aaa): endWriteSlow clears closeAckedKnown unconditionally, so a stalled detached stream (stall clock running, closeSince != 0) whose queue drains through a loop flush loses its baseline: detachedStalled then answers "no bound" until the next sample, which is a baseline, so such a peer can be reaped up to about a quarter of WriteTimeout earlier than the documented 3/4 to 5/4 of WriteTimeout after its last byte. Suggested fix: guard that line with cs.closeSince == 0 like the closeSampled line beside it. Not changed here (this round is comments, godoc and body only).
  • r3 F5b (a nit): each baseline/window sample allocates one unix.TCPInfo (about 0.25 KiB) on the loop thread; same as the existing noteClosingAcked, once per window per parked conn. Not fixed: the alternative, a raw SYS_GETSOCKOPT, is not defined on every linux GOARCH (386 goes through socketcall).
  • r3 F5: at a WriteTimeout below the sweep interval every sweep samples every parked conn; bounded by parked conns / max(WriteTimeout, sweep), the same order as the closing drain. Accepted.
  • io_uring has no rate rule (r3 F7): listed for a follow-up.
  • HTTP/2 parked on the socket is governed by the same rule in the code (the HTTP/2 queue drain calls noteWriteProgress and armEpollOut starts the clock) but no epoll unit test drives it; TestNonReadingH2ClientIsStillCutByWriteTimeout889 (test/server) passes on epoll and shows a non-reading h2 client is cut, without proving which clock cut it.
  • Abuse bound (informational, r3): at the default WriteTimeout 60 s a client acknowledging exactly 64 KiB per window holds a response of S bytes for S/64 KiB x 60 s (4 MiB: 64 min; 64 MiB: 17 h) and the unsent part of the response with it, at about 1.1 KB/s of attacker download per held conn. Strictly tighter than v1.5.9 and main for every behaviour except a client that acknowledges at least 64 KiB per window with no edge (intended), looser than std by design.

Release note

epoll (and adaptive while it runs on epoll): WriteTimeout is now a minimum rate for a response the kernel did not accept whole. Such a response is kept for as long as the client acknowledges at least 64 KiB per WriteTimeout (about 1 KiB/s at the 60 s default, 64 KiB/s at 1 s), with no elapsed-time ceiling, and the connection is closed at the end of the first WriteTimeout window in which the client acknowledges less (bytes the client sends do not count). A download at a reasonable rate therefore completes whatever ReadTimeout and IdleTimeout are.

Behaviour change: while such a response is queued, ReadTimeout and IdleTimeout do not apply to its connection. A client that stops reading a large response is now closed after about WriteTimeout (as net/http does), not at ReadTimeout/IdleTimeout as in v1.5: with ReadTimeout: 5s, WriteTimeout: 60s a stalled download is dropped after about 60 s instead of about 5 s. If you relied on a short ReadTimeout or IdleTimeout to drop stalled downloads, lower WriteTimeout instead. A request pipelined behind such a response is bounded by ReadHeaderTimeout for its headers and by ReadTimeout only once the response has been sent.

A slow but steady client can hold a connection and the unsent part of its response for size / 64 KiB x WriteTimeout (a 4 MiB response at 60 s: about 64 minutes), where net/http stops at WriteTimeout; size WriteTimeout with that in mind (a smaller one raises the floor: at 1 s it cuts clients slower than about 512 kbit/s). A client that sends a malformed request while its response is queued falls under the closing drain instead (one acknowledged byte per WriteTimeout). A response that waits for the client's HTTP/2 flow-control window is bounded by ReadTimeout/IdleTimeout from the client's last frame. Sockets that report no acknowledged byte count keep the previous behaviour. WriteTimeout: -1 disables the write timeout. The times above hold with ReadHeaderTimeout set (the default). io_uring and std are unchanged.

@FumingPower3925 FumingPower3925 added this to the v1.6.0 milestone Oct 26, 2026
@FumingPower3925 FumingPower3925 added area/engine Engine interface or implementation engine/epoll Epoll engine specifics labels Oct 26, 2026
@coderabbitai

coderabbitai Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…no ceiling, inbound bytes are not progress (Refs #885 item 51)

Tests only. Failing on main as assertions: a reader above the floor but
between two EPOLLOUT edges is closed at WriteTimeout, and a client taking
40 KiB/s with a shallow send queue is held for ever. The rest are guards
that pass on main and fail on the first design of this item (an
IdleTimeout ceiling, celeris PR 1116 head 88e0eb1): a download at 2 MiB/s
and a sendfile download complete under IdleTimeout, a CRLF every 2 s does
not hold a slow reader, WriteTimeout -1 holds a stuck peer.
…knowledged bytes (Refs #885 item 51)

A response parked on the socket is held another WriteTimeout window if
the peer acknowledged at least 64 KiB (tcp_info bytes_acked) in the
last one, and closed if it did not. Nothing the peer sends moves it,
there is no elapsed-time ceiling, WriteTimeout <= 0 is no write timeout.
A socket that reports no acknowledged bytes keeps the flush clock. One
getsockopt per parked conn per window; the sample record reuses the
closing drain's fields (no connState growth). config.go's WriteTimeout
paragraph follows; TestWriteClockRestartsWhenTheFlushTakesBytes885 waits
two windows for the reap of a peer that stopped.
@FumingPower3925

Copy link
Copy Markdown
Contributor Author

Sweep micro-benchmarks for this head (7c04aaa) against main 99e347a, with an A/A twin of main (docker --cpus 2, 10 rounds interleaved, median ns/op, 300 iterations per run):

benchmark main head head/main A/A (twin/main)
CheckTimeouts865/sync/256 853 848 0.99 0.96
CheckTimeouts865/sync/4096 18152 17857 0.98 1.00
CheckTimeouts865/async/256 1094 1110 1.02 1.02
CheckTimeouts865/async/4096 23749 23168 0.98 1.00
CheckTimeoutsParked885/sync/256 1154 1634 1.42 0.97
CheckTimeoutsParked885/sync/4096 24172 31564 1.31 1.00
CheckTimeoutsParked885/async/256 1838 2265 1.23 0.99
CheckTimeoutsParked885/async/4096 36604 46642 1.27 1.03

The common sweep (no conn parked) is within the A/A noise: the inline gate writeSince != 0 || dirty is untouched. The parked case (every conn parked, the worst case) costs about 1.7 ns more per conn (a call to parkedBelowRate: its early returns; in this benchmark the fake descriptor makes the getsockopt fail once per conn and then retry only every quarter window), 7 microseconds on 4096 parked conns per sweep, and a sweep runs at most every 25 ms: 0.03 percent of a core. A real parked conn adds one getsockopt per WriteTimeout. The RULE 74 cluster timing is left to the loop.

CI note: the first CI run of this PR (89951e4) failed TestParkedPeerAboveTheFloorWithoutAnEdgeIsNotReaped885 on both runners ("apparatus: the peer read nothing" at 3.4 s): the rig never flushed the engine's queue (no EPOLLOUT edge by design), so the peer ran out of what the kernel's send buffer held (a smaller buffer than on the dev machine). Fixed in 7c04aaa: the test keeps the peer supplied with flushWrites (which is not progress for the flush clock), 20/20 locally; still failing on main (reaped at 0.8 s).

@FumingPower3925

Copy link
Copy Markdown
Contributor Author

CI note, the Coverage failure of the run on head 7c04aaa: TestHijackedDescriptorIsNotReadAfterHandoff936/promoted-conn ("the handler never returned from Hijack", hijack_reread_936_linux_test.go:256, 5 s), the #1105 test. It is a flake that also fails on main under coverage; this PR's change cannot reach it (the test's hook blocks the loop thread, sweeps do not run, and sweepWriteClock is not on the Hijack path).

Measured in docker (--cpus 2, go test -race -cover -covermode=atomic -count=N -run TestHijackedDescriptorIsNotReadAfterHandoff936 ./internal/engine/epoll/, the flags of test-coverage.yml's root step):

tree count promoted-conn failures message
this PR, 7c04aaa 50 0
main 99e347a 50 0
this PR, 7c04aaa 1000 1 the handler never returned from Hijack
main 99e347a 1000 2 the handler never returned from Hijack

(promoting-request passed 1000 of 1000 on both.) So about 1 in 500 to 1 in 1000 on either tree, with the same message; under the whole-package run on a loaded runner the rate is higher. Not rerun in CI, as asked.

@codecov

codecov Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.18182% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/engine/epoll/write_rate_885.go 88.88% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

…en above the floor: the rate rule decides (Refs #885 item 51)
…the only bound: ReadTimeout and IdleTimeout do not cut a reader above the floor (Refs #885 item 51)
…d: the rate rule must not become a hard cutoff (Refs #885 item 51)
…wledges nothing: the flush clock judges it (Refs #885 item 51)
…x margins, catch-up pacing, the engine's count awaited before the clock (Refs #885 item 51)
…h the rate rule, field owners in conn.go (Refs #885 item 51)
…-held response, WriteTimeout 0 is the default, the sweep timer needs ReadHeaderTimeout (Refs #885 item 51)
@FumingPower3925
FumingPower3925 marked this pull request as ready for review October 28, 2026 02:32
@FumingPower3925
FumingPower3925 merged commit 46753f9 into main Oct 28, 2026
31 checks passed
@FumingPower3925
FumingPower3925 deleted the l3-ep-writeclock branch October 28, 2026 02:32
FumingPower3925 added a commit that referenced this pull request Oct 28, 2026
…the idle close sends GOAWAY(NO_ERROR); a burst of expiries is answered in one sweep (review r3; v1.6.0, Refs #889)

F1b/F1c (the regression): Sweep's CloseNow is a proposal. The engine closes the
conn only if it holds no output: the write buffer, the dirty list, EPOLLOUT, a
SEND in flight, and the kernel's send queue (TIOCOUTQ, the notion the pool-wait
gate uses before it closes a conn; an error counts as 'holds'): the stream is gone
from the manager once its last DATA is in the buffers and a close with the peer's
frames unread in the receive queue is a reset, which discards the queue. A conn
that holds output calls H2State.ExtendIdle: IdleTimeout runs again from there,
so D0 does not fight the rate rule of #1119 (it does nothing while a response is
parked) and a client that finishes taking its download is not closed on the
instant. The smaller safe change was chosen over a GOAWAY and closeWhenFlushed
for the held case: a close that waits for the peer to take the bytes would
re-implement the closing drain for a conn that is not closing. The close that
does go ahead sends GOAWAY(NO_ERROR) first (F6; best effort, TryLock), flushed
inline on epoll, staged before closeConn on io_uring.

F4: expired requests are answered in batches of 32 under a 2 ms budget until
none is due (they were 8 per sweep: the last of 100 was 0.3 s late on epoll,
1.27 s on io_uring). N1: reqArmed, which nothing read, is gone (one atomic add and
one CAS fewer per POST). N3: a client RST_STREAM on an expired stream ends its
memory. N4: ExpireRequest marks the stream closed on the conn writer as
sendRSTStreamAndMarkClosed does. The dangerous leak direction is pinned: with a
request deadline every stream whose request is open carries it, and the hint is
due by the earliest (TestEveryRequestOpenStreamCarriesADeadline889).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/engine Engine interface or implementation engine/epoll Epoll engine specifics

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant