Repository navigation
epoll: the write timeout of a parked response is a minimum rate on acknowledged bytes, no ceiling, inbound bytes are not progress (v1.6.0, Refs #885) - #1119
Conversation
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…no ceiling, inbound bytes are not progress (Refs #885 item 51) Tests only. Failing on main as assertions: a reader above the floor but between two EPOLLOUT edges is closed at WriteTimeout, and a client taking 40 KiB/s with a shallow send queue is held for ever. The rest are guards that pass on main and fail on the first design of this item (an IdleTimeout ceiling, celeris PR 1116 head 88e0eb1): a download at 2 MiB/s and a sendfile download complete under IdleTimeout, a CRLF every 2 s does not hold a slow reader, WriteTimeout -1 holds a stuck peer.
…knowledged bytes (Refs #885 item 51) A response parked on the socket is held another WriteTimeout window if the peer acknowledged at least 64 KiB (tcp_info bytes_acked) in the last one, and closed if it did not. Nothing the peer sends moves it, there is no elapsed-time ceiling, WriteTimeout <= 0 is no write timeout. A socket that reports no acknowledged bytes keeps the flush clock. One getsockopt per parked conn per window; the sample record reuses the closing drain's fields (no connState growth). config.go's WriteTimeout paragraph follows; TestWriteClockRestartsWhenTheFlushTakesBytes885 waits two windows for the reap of a peer that stopped.
89951e4 to
7c04aaa
Compare
|
Sweep micro-benchmarks for this head (7c04aaa) against main 99e347a, with an A/A twin of main (docker
The common sweep (no conn parked) is within the A/A noise: the inline gate CI note: the first CI run of this PR (89951e4) failed |
|
CI note, the Coverage failure of the run on head 7c04aaa: Measured in docker (
( |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…en above the floor: the rate rule decides (Refs #885 item 51)
…the only bound: ReadTimeout and IdleTimeout do not cut a reader above the floor (Refs #885 item 51)
…haviour change (Refs #885 item 51)
…d: the rate rule must not become a hard cutoff (Refs #885 item 51)
…wledges nothing: the flush clock judges it (Refs #885 item 51)
…x margins, catch-up pacing, the engine's count awaited before the clock (Refs #885 item 51)
…h the rate rule, field owners in conn.go (Refs #885 item 51)
…Timeout plus two sweeps (Refs #885 item 51)
…-held response, WriteTimeout 0 is the default, the sweep timer needs ReadHeaderTimeout (Refs #885 item 51)
…the idle close sends GOAWAY(NO_ERROR); a burst of expiries is answered in one sweep (review r3; v1.6.0, Refs #889) F1b/F1c (the regression): Sweep's CloseNow is a proposal. The engine closes the conn only if it holds no output: the write buffer, the dirty list, EPOLLOUT, a SEND in flight, and the kernel's send queue (TIOCOUTQ, the notion the pool-wait gate uses before it closes a conn; an error counts as 'holds'): the stream is gone from the manager once its last DATA is in the buffers and a close with the peer's frames unread in the receive queue is a reset, which discards the queue. A conn that holds output calls H2State.ExtendIdle: IdleTimeout runs again from there, so D0 does not fight the rate rule of #1119 (it does nothing while a response is parked) and a client that finishes taking its download is not closed on the instant. The smaller safe change was chosen over a GOAWAY and closeWhenFlushed for the held case: a close that waits for the peer to take the bytes would re-implement the closing drain for a conn that is not closing. The close that does go ahead sends GOAWAY(NO_ERROR) first (F6; best effort, TryLock), flushed inline on epoll, staged before closeConn on io_uring. F4: expired requests are answered in batches of 32 under a 2 ms budget until none is due (they were 8 per sweep: the last of 100 was 0.3 s late on epoll, 1.27 s on io_uring). N1: reqArmed, which nothing read, is gone (one atomic add and one CAS fewer per POST). N3: a client RST_STREAM on an expired stream ends its memory. N4: ExpireRequest marks the stream closed on the conn writer as sendRSTStreamAndMarkClosed does. The dangerous leak direction is pinned: with a request deadline every stream whose request is open carries it, and the hint is due by the earliest (TestEveryRequestOpenStreamCarriesADeadline889).
Refs #885 (item 51)
Draft: lane L3 EP-TIMEOUTS of v1.6.0, the redesign of item 51 after Opus review r1 of celeris #1116 (head 88e0eb1) rejected its IdleTimeout ceiling (it cut legitimate downloads at IdleTimeout, sendfile included, and any inbound byte refreshed it). #1116 is now 885.49 and 885.42 only. The rule's numbers are a maintainer decision; the user has decided the policy below.
History of this branch:
fd2a230tests,7c04aaathe rule (reviews r1, r2, r3 in the probatorium evidence), then, after Opus review r3 (APPROVE-WITH-FIXES): a merge of main (fe2d765, no conflicts: the two PRs' godoc hunks did not overlap), the commit pairs below, a second merge of main (fca2281, only driver/postgres files), two comment-only godoc commits (8a23790), then, after Opus review r4 (APPROVE-WITH-FIXES, no blocker): a third merge of main and one comment/godoc/body-only commit (F11, F15, F16, F17), no behaviour change. New commits only; no rebase, no force-push.The rule
A response a keep-alive conn has parked on the socket (the kernel did not take it whole; HTTP/1 and HTTP/2 alike) is held another
WriteTimeoutwindow if the peer acknowledged at leastparkedMinAcked= 64 KiB (tcp_infobytes_acked, onegetsockoptper parked conn per window) in the last window, and closed if it did not.WriteTimeout <= 0is no write timeout at any rate (ReadTimeout and IdleTimeout slide with the flush clock exactly as on main).sweepWriteClockreturns "done" after the rule has had its say, socheckTimeoutsskips the ReadTimeout/IdleTimeout scan for that conn. ReadTimeout and IdleTimeout apply again from the end of the response (endWrite) or if the baseline is lost. A conn with no parked response is reaped by them as on main.getsockopt, or a count of 0, r3 F4) has the flush clock alone, as before the rule, and ReadTimeout/IdleTimeout run from it as on main. The baseline is retried at most every quarter window.TestConnStateStaysInItsSizeClass: 448 bytes). The baseline reusescloseAcked/closeAckedKnown/closeSampled; it is forgotten where the response's clock ends (endWriteSlow, the sweep's empty-queue case,markClosing).closeSampledis now also zeroed there for a live conn, so the next response on the same conn gets its baseline at its first sweep and not after a quarter of WriteTimeout.F1: choice made and why
Review r3 measured that ReadTimeout and IdleTimeout, which slide only with flushes that took bytes and with passed windows, still cut a steady reader that takes at least a window between two EPOLLOUT edges whenever they are not above WriteTimeout (the defaults' shape): RT=WT closed at 806 ms, RT=WT/2 at 401 ms, IT=WT at 811 ms. I chose the reviewer's option (a): exempt a parked conn whose baseline is known from the idle scan. It is the smaller change (one early return in
sweepWriteClock, no new sampling, no new state) and the safer one: option (b) (stampwriteSince/lastActivityon any acked advance, sampled every min(RT,IT,WT)/4) would put agetsockopton a quarter of the shortest of three timeouts for every parked conn. Two consequences of (a) that needed a change each:writeSince, which for a rule-governed response is the stamp of the last passed window (up to WriteTimeout old); with ReadTimeout shorter than that the conn would be reaped at the sweep that found its response whole. It now dates the drain at the sweep (TestParkedResponseDrainedBehindTheLoopStartsTheIdleClockAtTheSweep885).closeSampledfrom the previous response gated the next response's baseline for a quarter window, during which ReadTimeout/IdleTimeout still apply: zeroed at the drain (TestParkedSecondResponseRightAfterTheFirstIsRateGoverned885).Behaviour change against v1.5.9 and main (also in the release note): with ReadTimeout or IdleTimeout below WriteTimeout, a client that stalls a parked response is no longer closed at min(ReadTimeout, IdleTimeout) but by the rule, between WriteTimeout and WriteTimeout + 2 sweeps after the response parked (ReadTimeout 5 s, WriteTimeout 60 s: about 60 s, was about 5 s). That is the user's decision (the rate rule governs parked responses).
Maximum lifetime of a parked response (WriteTimeout T, response S bytes, R = 64 KiB, sweep s of 25 to 100 ms, which holds with ReadHeaderTimeout set (the default): the 25 ms sweep timer is armed only then, and with ReadHeaderTimeout -1 a stalled response with T = 1 s went at 5 s, on main too; rows re-checked against the final code)
bytes_ackedreads 0 (a peer that has acknowledged nothing yet, or a stack that does not fill it) has no baseline and is still cut at min(T, ReadTimeout, IdleTimeout) as on main, until a sample sees acknowledged bytes (retried every T/4)TestParkedReaderThatStops...: within 2T + 250 ms at s = 20 ms)WriteTimeout-1 (or 0)Tests
"main" = origin/main 3f516f1 + the PR's test files that build there; "previous head" = fe2d765 (main merged with 7c04aaa) + the same test files; "this head" = d9f819e (the engine and test code are identical at the PR head 8a23790: the later commits are merges of unrelated files and comments). docker golang:1.27.3,
--cpus 2, memlock 8 MiB, seccomp unconfined,count=1,-race. Commit pairs:40a3d5f(tests, RED on fe2d765) then7349549(fix F1);7d38eb7(seam, no behaviour change),1cb89a8(tests, RED on the seam commit) then45499a3(fix F4);4a9a3c0(F9/F10: test-only hardening, no fix pair: its RED column is main's engine);d9f819egodoc.TestParkedReaderAboveTheFloorIsNotCutByReadOrIdleTimeout885(rig, no edge; RT=WT, RT=WT/2, IT=WT, RT=IT=WT, defaults-shaped RT=WT IT=10WT; each at 1.6x and 4x the floor: 10 cells)TestParkedReaderThatStopsIsClosedWithinTwoWindowsWhateverReadTimeout885(3 cells: stops after two windows at 4x; RT=WT/2, RT=IT=WT, both off)TestParkedSecondResponseRightAfterTheFirstIsRateGoverned885(second response parks right after the first, RT 150 ms < WT/4)TestParkedResponseDrainedBehindTheLoopStartsTheIdleClockAtTheSweep885(added with the fix, not test-first; on the previous head ReadTimeout reaps the conn before the drain is reached, so it can only be RED for the F1 reason)TestParkedConnWithNothingParkedIsStillReapedByReadTimeout885(guard: drained then idle on TCP; AF_UNIX rig that reports no acked bytes)TestParkedConnOfAStackThatReportsNoAckedBytesIsJudgedByTheFlushClock885(fake stack reporting bytes_acked = 0, via theackedProbeseam)TestParkedConnIsRateGovernedOnceTheStackReportsAckedBytes885,...AndATakesNothingIsReaped885(guards)TestParkedPeerAboveTheFloorWithoutAnEdgeIsNotReaped885(rig, now 4x)TestEngineReaderAboveTheFloorIsNotClosedBetweenEdges885(sync, async; now WT 2 s, client 128 KiB/s = 4x)TestEngineSlowlorisBelowTheFloorIsClosed885x2TestEnginePeerInboundBytesDoNotRefreshTheWriteClock885none / CRLF / GET x sync / asyncTestEngineStuckPeersAreClosedWithinTwoWindows885x3,TestParkedPeerBelowTheFloorIsReapedWhateverItsFlushes885,TestParkedConnWithNoWriteTimeoutIsNeverReapedByTheRate885,TestEngineNoWriteTimeoutHoldsAStuckPeer885,TestEngineDownloadAtAModerateRateCompletesUnderIdleTimeout885x4TestEpollStaticFileDownloadAtAModerateRateCompletes885x2,TestNonReadingH2ClientIsStillCutByWriteTimeout889(epoll, io_uring, adaptive), #1134's h2 timeout testsFlake checks on this head,
-race -count=10, the 16 new and changed tests (160 top-level runs, 280 subtests): plain 748 s, 0 failures; under six busy loops in the same--cpus 2container 850 s, 0 failures, 0 retries used. The reviewer's F9 reproduction (10/10 failures under six busy loops) no longer fails. The reviewer's own probe (TestL3R3Probe, r3 F1/F6) on this head: rig RT=WT, RT=WT/2, IT=WT, RT=2WT all held 4 s; engine ReadTimeout 1 s, 500 ms, -1, sync and async, all held 5 s (was gone at 0.54 s for 500 ms);BAD\r\n\r\nstill held 8 s (the known limit).Margins (the client's rate over the floor of 64 KiB per WriteTimeout)
TestParkedPeerAboveTheFloorWithoutAnEdgeIsNotReaped885TestEngineReaderAboveTheFloorIsNotClosedBetweenEdges885TestEngineSlowlorisBelowTheFloorIsClosed885TestEnginePeerInboundBytesDoNotRefreshTheWriteClock885TestParkedPeerBelowTheFloorIsReapedWhateverItsFlushes885TestEngineDownloadAtAModerateRateCompletesUnderIdleTimeout885F10: the engine tests'
goneAfter885now waits untilActiveConnections >= 1before starting its clock (an accepted conn the engine has not counted reads as "gone") and the closed-within tests also assert a lower bound (about one window), so they cannot pass vacuously.Verification
-race -count=1, whole package: ok in 377 s on this head (347 s on the merge tree before this round, +30 s), against the root step's 540 s per-package limit; measured with other agents' containers sharing the host, so a CI runner needs headroom; the new tests add about 28 s of F1 cells, 8 s of F4 tests and a few seconds of F9 changes per count.-race-run '889|885|WriteTimeout|ReadTimeout|IdleTimeout|WriteRate': ok, 0 failures (h2 timeout tests from engines: ReadTimeout and IdleTimeout skip an HTTP/2 conn that has a stream open, proposal with the policy decision pending (v1.6.0, Refs #889) #1134 and the sendfile download included)../internal/engine/epoll/...and the root package.GOOS=windows,GOOS=darwingo vet ./...clean;linux/arm64vet of epoll and test/server clean.mage apiCheck: api/ matches (config.go changes are comments only).Comparison with the other engines
WriteTimeoutis one absolute deadline for the whole response: every download longer than T is cut, at any rate. epoll has never done that for a response in progress, and the docs say so.now - lastActivity > WriteTimeout, wherelastActivityis restamped at the last read, the start of a send from idle (iouring: the write clock of a conn starts when its send starts, so a quiet h2 stream is not cut at a chunk (v1.6.0, Refs #889) #1136) and a completed send. A client that completes one SEND per T, or sends any byte, holds it for ever; there is no rate floor and no acknowledged-bytes signal for a live conn. The io_uring side would take the same rule in a follow-up (tcp_info sample in its sweep).Known limit (not changed here)
A malformed request read while a response is parked is answered as a request error and makes the conn a closing conn (and so does any request read behind a backlog over
maxPendingBytes, 4 MiB): its drain keeps the #876 per-byte rule (noteClosingAcked, no floor, documented in config.go), so a client that sends such a request and then acknowledges a few bytes per window is held by the closing drain, not by this rule. Measured on this head and on main:BAD\r\n\r\n50 ms after the GET, client reading 4 KiB/400 ms (10 KiB/s) of a 3 MiB response, WriteTimeout 1 s: held 8 s (62 KiB read); a bare CRLF every 2 s is closed at about 1 s because the first window passes before the first CRLF (config.go no longer claims a bare CRLF cannot hold the conn). Applying the floor to the closing drain changes #876's tests and decision (a slow reader of aConnection: closeresponse); left for the same maintainer decision.Findings, not fixed here
sweepWriteClockskips the idle scan, and a pipelined request whose head is whole and whose body is pending has no header deadline (ReadHeaderTimeout still bounds a partial head: measured, closed at 1.87 s with ReadTimeout 1 s, the header check runs first). Reviewer's probe, a floor-reader at 6x of a 64 MiB response plus a POST with 10 of 1000 body bytes, ReadTimeout 1 s: main cut it at 1.34 s, this head held it for the whole 10 s run; with a 4 MiB response that drains at about 6 s, ReadTimeout applies again from the drain, gone at 6.94 s on both trees. The same early return suspends the async-handler overrun check ofh2ServerOwes(read, not measured). Bounded by the accepted policy (the client must keep acknowledging the floor), adds no new resource class; comment and godoc now say so. A stricter variant (keep the idle scan while a request body is in progress) would reinstate F1 for a client that downloads and uploads on one conn; left as a follow-up if the maintainer wants it.endWriteSlowclearscloseAckedKnownunconditionally, so a stalled detached stream (stall clock running,closeSince != 0) whose queue drains through a loop flush loses its baseline:detachedStalledthen answers "no bound" until the next sample, which is a baseline, so such a peer can be reaped up to about a quarter of WriteTimeout earlier than the documented 3/4 to 5/4 of WriteTimeout after its last byte. Suggested fix: guard that line withcs.closeSince == 0like thecloseSampledline beside it. Not changed here (this round is comments, godoc and body only).unix.TCPInfo(about 0.25 KiB) on the loop thread; same as the existingnoteClosingAcked, once per window per parked conn. Not fixed: the alternative, a rawSYS_GETSOCKOPT, is not defined on every linux GOARCH (386 goes through socketcall).noteWriteProgressandarmEpollOutstarts the clock) but no epoll unit test drives it;TestNonReadingH2ClientIsStillCutByWriteTimeout889(test/server) passes on epoll and shows a non-reading h2 client is cut, without proving which clock cut it.Release note
epoll (and adaptive while it runs on epoll): WriteTimeout is now a minimum rate for a response the kernel did not accept whole. Such a response is kept for as long as the client acknowledges at least 64 KiB per
WriteTimeout(about 1 KiB/s at the 60 s default, 64 KiB/s at 1 s), with no elapsed-time ceiling, and the connection is closed at the end of the firstWriteTimeoutwindow in which the client acknowledges less (bytes the client sends do not count). A download at a reasonable rate therefore completes whateverReadTimeoutandIdleTimeoutare.Behaviour change: while such a response is queued,
ReadTimeoutandIdleTimeoutdo not apply to its connection. A client that stops reading a large response is now closed after aboutWriteTimeout(as net/http does), not atReadTimeout/IdleTimeoutas in v1.5: withReadTimeout: 5s, WriteTimeout: 60sa stalled download is dropped after about 60 s instead of about 5 s. If you relied on a shortReadTimeoutorIdleTimeoutto drop stalled downloads, lowerWriteTimeoutinstead. A request pipelined behind such a response is bounded byReadHeaderTimeoutfor its headers and byReadTimeoutonly once the response has been sent.A slow but steady client can hold a connection and the unsent part of its response for size / 64 KiB x
WriteTimeout(a 4 MiB response at 60 s: about 64 minutes), where net/http stops atWriteTimeout; sizeWriteTimeoutwith that in mind (a smaller one raises the floor: at 1 s it cuts clients slower than about 512 kbit/s). A client that sends a malformed request while its response is queued falls under the closing drain instead (one acknowledged byte perWriteTimeout). A response that waits for the client's HTTP/2 flow-control window is bounded byReadTimeout/IdleTimeoutfrom the client's last frame. Sockets that report no acknowledged byte count keep the previous behaviour.WriteTimeout: -1disables the write timeout. The times above hold withReadHeaderTimeoutset (the default). io_uring and std are unchanged.