Repository navigation
Conversation
…d of once per DATA frame, so an upload costs one WINDOW_UPDATE per 64 KiB (#630) A stream's receive window was credited frame by frame: a buffered body and a streamed body's padding as each DATA frame arrived, a streamed body as its handler read each chunk, and the upload behind a buffered answer parked on credit as each frame was dropped. Outside a read pass each read also made a flush of its own. A stream is now credited once what it has consumed and not yet been credited reaches a step, in one WINDOW_UPDATE for all of it. The step is a quarter of InitialWindowSize, at most 65,535 bytes and at least one: until the client has applied the server's SETTINGS it counts the default 65,535-byte window (RFC 9113 6.9.2), and a quarter of the default 256 KiB is one byte more than that. Consumed is what the stream was sent and not credited, from its ReceiveWindow, less what its body reader still holds unread, which a new count on the reader keeps; unread bytes are never credited, so the window still bounds what a handler holds. A request that has ended is owed nothing, as before, and a stream that is reset or ended is credited nothing afterwards. A read outside a pass flushes only when a step is due. The upload behind a parked buffered answer has no request left to count against, so the connection keeps a count per such stream, freed when the answer ends or the stream does. It counts what the handler left unread too: with steps, an upload whose window the handler had nearly filled would otherwise wait on credit that never comes.
…d uncredited, like any stream its handler is done with (#630) The count kept for that upload is gone: the branch in RetireStream that started it, CreditParkedUpload, _parkedUploads and its Remove in CompleteParkedAsync. So is the credit origin/main gave the same upload, one WINDOW_UPDATE per frame dropped: HandleData returns for any stream outside the stream table. A client that uploads its whole body before it reads, given a buffered answer bigger than its stream window by a handler that did not read the upload, now waits until its own timeout. A client that reads while it uploads still gets the whole answer. Tests: the two that had a client finish its upload behind a parked answer, and the one that read the count, tested only what is removed and are gone. Http2EarlyAnswerTests' parked test now checks that the upload is credited nothing and that a client crediting the answer gets it whole; with origin/main's per-frame credit put back it fails with "expected [0], got [65536]".
…T_STREAM(NO_ERROR) while its request is still arriving (#588) A buffered answer bigger than the client's stream window parks the rest on the client's credit, and its stream is retired with the rest still to go, so the retire does not reset it. An answer that ends at once is followed by RST_STREAM(NO_ERROR) once its END_STREAM is out, which asks the client to stop sending a request the server is done with and leaves it the answer (RFC 9113 8.1); a parked one was not. With that upload no longer credited, a client that reads while it uploads got the whole answer and then had its upload stall. ParkRest now reads whether the request is still arriving, before the retire takes the stream out of the stream table, and the parked rest resets the stream with NO_ERROR once its END_STREAM is staged, if the response is still live: not after the peer's reset, nor after one of the stack's own. No connection state. One imprecision: a request END_STREAM that arrives while the answer is parked is not seen, so the reset can follow a stream already closed both ways; RFC 9113 5.1 has the peer ignore a RST_STREAM in that state. Tests: - Unit: a buffered answer that waited on credit, its request still arriving, is followed by RST_STREAM(NO_ERROR) after its END_STREAM, as the stream's last frame. Without the reset: "the answer ended with its request still arriving, and no RST_STREAM asked the client to stop uploading". - Unit control: a request that had ended before its answer parked, with a body reader and without, gets no reset. Reading only whether the stream is in the table: "a request that had ended was reset". - Unit guard: no reset back once the peer has reset the stream, nor a second after the stack's own FLOW_CONTROL_ERROR, both while the answer's first flush is in flight. Without the liveness check: "stream 1 was reset after the peer had reset it". - E2E: .NET's HttpClient with a duplex content, which reads while it uploads, posting 64 MiB to a handler that answers a 256 KiB 413 unread: the answer arrives whole and the upload stops under 512 KiB. Without the reset: "the upload neither finished nor stopped, at 262144 bytes".
This was referenced Nov 15, 2026
# Conflicts: # tests/Ioxide.Tests.Unit/Http2EarlyAnswerTests.cs
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #630.
Part of #588.
What was wrong
The connection's receive window is topped up once it falls below a quarter, but each stream's window was credited frame by frame (RFC 9113 6.9, 6.9.1):
So an upload cost one WINDOW_UPDATE per DATA frame however small its frames were: 255 for a 4 MiB body in 16 KiB frames, 4,194 for the same body in 1,000-byte frames.
What changes
All in
src/protocols/ioxide.http2; no core change.ReceiveWindow, and from that the bytes still queued unread in its body reader are taken off. Those are never credited: the stream window is what bounds what a handler holds unread, as before.InitialWindowSize, but at most 65,535 bytes and at least one byte. See below for why it is capped.CreditBody's detached flush outside a pass happens only when a step was staged. A read short of a step writes nothing and flushes nothing.New state:
Http2BodyReader.Unread, the bytes its queue holds. Taking unread bytes off what is owed needs that sum at every read, and summing the queue each time goes quadratic with tiny frames. It is kept byPush, by each hand-over, and zeroed when the queue is dropped.Removed
RetireStream,CreditParkedUpload,_parkedUploadsand itsRemoveinCompleteParkedAsync. Both are gone:HandleDatareturns for any stream outside the stream table and credits nothing.In its place: RST_STREAM(NO_ERROR) after a parked answer's END_STREAM
ParkRestreads whether the request is still arriving, before the retire takes the stream out of the stream table, and the parked rest resets the stream with NO_ERROR once its END_STREAM is staged, if the response is still live: not after the peer's reset, nor after one of the stack's own. No connection state.Why it never stalls
A client is blocked on a stream only when its bytes in flight there, sent and not yet credited, fill its view of the stream window. On the server those bytes are either unread, queued in the body reader, or consumed and not yet credited. Whenever the handler has read everything queued, the unread part is zero, so the uncredited part alone is everything in flight. If the client is blocked, that is at least its window, and so at least the step as long as the step fits inside every window the client can be counting. The read that emptied the queue, or for a buffered body or padding the frame that arrived, runs the check and sends the credit. A handler that stops reading stops the upload, which is the backpressure this reader is for. This is about a request still being read; the upload behind an answer whose handler is done with it is under Removed.
The windows a client can count:
InitialWindowSizeonce it has applied them, which can happen before the server has their acknowledgement (6.5.3), so the server cannot tell which of the two a client counts until then.A quarter of
InitialWindowSizealways fits insideInitialWindowSize. But a quarter of the default 256 KiB is 65,536, one byte more than 65,535: a client that has not yet applied the SETTINGS would be blocked at 65,535 bytes in flight with 65,535 owed, one short of a step, and would wait until it applied them. Capping the step at 65,535 makes it fit both windows. The floor of one byte keeps a window below four bytes credited at all.The cap means a window above 256 KiB is credited every 65,535 bytes rather than every quarter. The server could raise the step to a quarter once the SETTINGS are acknowledged (6.5.3), at the cost of a flag it does not keep today; this PR does not.
What a caller would notice
InitialWindowSizedocs describe the steps.Tests
Each line gives what fails with its part reverted, the rest of the change in place.
Unit, new
Http2CreditStepTests, on the wire with the strict client and a client that keeps its own windows:CreditBodyflushing whether or not a step was staged, and at origin/main: "49152 bytes were read short of a step and 1 flushes went out behind them".MaxRequestBytesstill bounds it: a body of exactlyMaxRequestBytesis answered, one past it is reset with ENHANCE_YOUR_CALM, and DATA in flight after that reset is credited nothing. At origin/main: "64 DATA frames of a buffered body were credited by [16384, 16384, ...], not in whole steps". The bound passes there too.InitialWindowSize256 KiB (the default), 1 MiB, 65,536, 65,535, 16,384, 1,000, 100 and 3, with the client never applying the server's SETTINGS, applying them without acknowledging, acknowledging them before the upload, and applying them with one 65,535-byte window in flight and acknowledging after the next. Every upload completes. It passes at origin/main, which credits every frame. With the step uncapped: "InitialWindowSize 262144, SETTINGS Unseen: the upload stopped at 65535 of 787433 bytes, 65535 of them read, the client out of window", and the same at 1 MiB. With the step taken from the server's own window before the acknowledgement, a quarter of 65,535: "InitialWindowSize 1000, SETTINGS Applied: the upload stopped at 1000 of 197606 bytes", and the same for 100 and 3, in every state where the client applied the SETTINGS.The clean close, unit in
Http2EarlyAnswerTestsandHttp2EarlyAnswerGuardTests, and E2E inEarlyAnswerTests:HttpClientwith a duplex content, which reads while it uploads, posting 64 MiB to a handler that answers a 256 KiB 413 unread: the answer arrives whole and the upload stops under 512 KiB. Without the reset: "the upload neither finished nor stopped, at 262144 bytes".Removed with the special case: the two tests that had a client finish its upload behind a parked answer, and the one that read
_parkedUploads.Http2EarlyAnswerTests' "an upload still arriving while a buffered answer is parked on credit is credited on its stream", which tested origin/main's per-frame credit, now checks that the upload is credited nothing and that a client crediting the answer gets it whole; with origin/main's credit put back it fails with "expected [0], got [65536]".Existing tests that expected credit at once, each keeping what it proves:
Http2FlowControlTests"padding on a streamed body is credited" runs with a window whose step is the padded body's payload (4 x 1,457), and its padding-only frame now arrives once the handler has read the body and parked, so it is that arrival that credits. With the arrival check skipped for a streamed body's padding: "1457 flow-controlled bytes arrived and 0 were credited".Http2FlowControlTests"padding on the buffered path is credited in full" uses the same window, and "a DATA frame past MaxRequestBytes resets its stream, and only the frames taken are credited" a 64 KiB window, whose step is its one frame taken.Http2StreamedRequestTests"credit is returned on read, not on arrival" uses a 4,000-byte window, whose step of 1,000 is its two chunks, and "buffered dispatch still assembles the body" a 1,600-byte window, whose step of 400 is its first frame.Http2ReceiveWindowTests"the frame loop goes on reading while a flush is stalled" sends a step, four frames, instead of two.With the step taken from the server's own window before the acknowledgement, the first four of these fail as well ("expected [1000], got [0]", "expected [400], got [0]", "1457 flow-controlled bytes arrived and 0 were credited", "expected [1457], got [0]"). With the step uncapped, the existing "every byte comes back once, on the connection's top-up" fails: "stream 1 was sent 1610572675 bytes, read 1610572474 of them, and was credited 1090529176".
Playground/Http2/ManagedStreamedRequestno longer says each read hands back the credit for its chunk, and its docs pane is regenerated withscripts/gen-docs-panes.py.Suites
Pinned to the safe CPUs, on this head, one suite at a time; other agents were running their own suites on the box meanwhile.
Bench
h2c on an i9-14900K, origin/main (19aec5c) against this branch at 40c9b8f; the commits after it change only an answer parked on credit, which none of these runs takes. The two builds were interleaved, alternating which goes first: the server
taskset -c 2,3,6,7(two P-cores), h2loadtaskset -c 16-31(E-cores), 3 s of warm-up and 10 s measured, the server's CPU read from/proc/<pid>/statover the measured window. Uploads POST toPlayground/Http2/ManagedStreamedRequest, which reads its body as it arrives; the GET control isPlayground/Http2/ManagedBuffered. Other agents were running their own test suites on the box throughout, load average 1 to 6, and the GET control swung from 4.6M to 6.7M req/s between runs of one binary. No request failed in any run.Stream WINDOW_UPDATEs per upload, through
nghttp -v -d(16 KiB DATA frames):Throughput and server CPU per request, medians of the pairs:
-t8 -c16 -m4, 1 reactor, saturated, 5 pairs-t8 -c32 -m8, 1 reactor, saturated, 5 pairs-t4 -c16 -m4, 2 reactors, 4 pairs-t4 -c32 -m8, 2 reactors, 4 pairs-t8 -c64 -m32, 2 reactors, 4 pairsWhat it shows: the wire saving is 4 to 1 for 16 KiB frames, and a body that ends inside a step costs no WINDOW_UPDATE. Saturated 64 KiB uploads are about 1% slower on this branch, consistently: lower in 13 of 14 interleaved pairs, 9 of the 10 above and all 4 of a reviewer's re-run, which were lower by 0.7% to 1.8%. The cause is unknown. Otherwise the two builds are level: at a fixed request rate, which compares CPU at equal throughput, and in the saturated 1 MiB cell. The two-reactor upload rows ran at 1.5 to 1.9 of 2 cores, short of saturation; their 64 KiB row's -14% CPU did not reproduce at a fixed rate, and the 1 MiB row's +4.5% drifted with the load from round to round (pairs +5%, -2%, +11%, +2%).
Overlaps
Checked with
git merge-tree --write-treeagainst each branch's head, all based on 19aec5c.Http2BodyReader.cs,Http2Connection.Frames.cs,Http2Connection.Write.csand the E2EEarlyAnswerTests.cs. Two of its parts meet this PR:CompleteParkedAsync: it keeps a stream whose answer waits on credit in the stream table until the answer's END_STREAM, then resets it with NO_ERROR throughRetireStream. That does what theParkRestbool and the reset inCompleteParkedAsyncdo here, and it also sees a request END_STREAM that arrives meanwhile. With both, keep http2: an answer waiting on the client's credit has its unread upload credited, so a client that uploads everything first gets it (#588) #604's path and drop this PR's bool and reset.CreditConsumed, in steps:DropzeroesUnreadinDropQueuedand calls_owner.CreditBody(_streamId), andHandleDatacallsCreditConsumed(pending)where http2: an answer waiting on the client's credit has its unread upload credited, so a client that uploads everything first gets it (#588) #604 callsCreditStream(pending, credit).DuplexUploadandICountedUploadto the E2E tests; keep one of each.Http2EarlyAnswerTests.cs: both append tests at the end of its reset group; keep both.Http2Connection.Frames.cs, at the branch for a stream outside the table: keep its STREAM_CLOSED check first and the plain return after it. Its check applies only without a body reader, and an answer whose request is still arriving parks only with one.