Repository navigation
Conversation
…ETTINGS, so a connection's request bodies in flight are its streams times their window (#612) SETTINGS size stream windows only (RFC 9113 6.9.2), and the shim never raised the connection's, so it stayed at 65,535: a connection had at most 64 KiB of request body in flight across all its streams, and one handler that did not read held all of it. ih2_new now opens it with nghttp2_session_set_local_window_size right behind the SETTINGS, so the preface is SETTINGS and then WINDOW_UPDATE(0, 2,147,418,112), and nghttp2 tops it up once half of it is consumed. ih2_server_new takes the SETTINGS' MAX_CONCURRENT_STREAMS and INITIAL_WINDOW_SIZE, which Nghttp2Options now carries with ioxide.http2's defaults: 100 streams of 256 KiB, from 1000 of 1 MiB. The shim gains ih2_abi (ABI 1), and a connection refuses a library of another revision. RunCoreAsync no longer waits for a pass's flush before it reads again: with the window open, an upload behind a download the peer is not reading would overflow the transport's receive queue and close the connection. What later passes submit leaves with the drain in flight, and nghttp2 bounds what can queue there. A drain nobody awaits reports a fault through IsBroken and ends the loop's read, every drain fires the body wakes of the streams it closed, and at the end of input the loop waits for the drain in flight before the teardown. The SETTINGS flush is still waited for. Both libioxide_nghttp2.so are rebuilt with scripts/build-nghttp2-native.sh at its pinned ref. Tests: Nghttp2ReceiveWindowTests (Unit) for the preface, the SETTINGS, the stream windows, the connection's top-up after 1 GiB, and the frame loop behind a stalled flush; a Chaos test for an upload behind a stalled download over TCP. The Chaos tests that counted on the 65,535 connection window now hold uploads to their stream's, and the off-reactor E2E nghttp2 uploads run at 65,535 as their http2 twins do.
Owner
Author
|
Closed: opening nghttp2's connection window needs the frame loop to read on behind a flush, and that let a long streamed response hold back a short one, since nghttp2 sends non-incremental streams strictly in order. The 65,535-byte connection window is conformant and only slower for uploads, so it stays as it is. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #612.
What was wrong
RFC 9113 6.9.2:
SETTINGS_INITIAL_WINDOW_SIZEsizes stream windows only. The connection's receive window starts at 65,535 and grows only by WINDOW_UPDATE on stream 0, and a sender stays within both windows (6.9.1).The shim advertised 1000 streams of a 1 MiB window, but
ih2_newnever raised the connection's window. nghttp2 gave stream 0 back only what had arrived (buffered bodies, by its automatic WINDOW_UPDATE) or what handlers had read (streamed bodies, throughih2_consume). So a connection had at most 64 KiB of request body in flight across all its streams, which bounds uploads by round trips, and a handler that did not read held the whole of it: every other upload on the connection waited until that handler read or returned.Reproduced at main 19aec5c before any change, with a scratch program over the strict test transport and main's library (sha256
5db0c664...):The preface has no WINDOW_UPDATE on stream 0. 16 KiB that fits stream 3's own window still overran the connection's, a GOAWAY with FLOW_CONTROL_ERROR. And the frame loop read nothing while a flush was stalled, which is what made the window part unsafe on its own (below). The Chaos test that holds a handler pinned the stall at exactly 65,535.
What changes
What #593 did for ioxide.http2, here. All in
src/protocols/ioxide.nghttp2: nothing insrc/ioxidechanges.ih2_newopens the connection's window withnghttp2_session_set_local_window_size(session, NGHTTP2_FLAG_NONE, 0, NGHTTP2_MAX_WINDOW_SIZE)right afternghttp2_submit_settings. nghttp2 sends its queued SETTINGS first, so the preface is SETTINGS and then WINDOW_UPDATE(0, 2,147,418,112), in one write. nghttp2 tops the window up itself once half of it is consumed, in both modes: as DATA arrives, with its automatic WINDOW_UPDATE, and throughih2_consumewithout it.ih2_server_newtakesmax_concurrent_streamsandinitial_window_sizefor its SETTINGS.ih2_client_new, which nothing calls, keeps its 1000 streams of 1 MiB and gets the open window too.IH2_ABI 1with its history line, anih2_abi()export, andNghttp2.RequireAbi()in theNghttp2Connectionconstructor. Against main's library the fixed binding now throwsInvalidOperationException: ioxide.nghttp2: the native library reports ABI 0, this binding needs 1. A stale libioxide_nghttp2.so is being loaded - rebuild it with scripts/build-nghttp2-native.sh., where it would otherwise have read the two new arguments out of registers nobody set.Nghttp2Options.InitialWindowSize(256 KiB) andMaxConcurrentStreams(100),Http2Options' defaults, with docs saying what nghttp2 does with them. nghttp2 already enforced stream windows; nothing changes there.RunCoreAsyncawaited each pass's flush before it read again. With the window open, an upload behind a download the peer is not reading fills the transport's receive queue, and the delivery past its 64 entries closes the connection (TcpOptions.RecvQueueEntries). The loop now starts the pass's drain and reads on. What later passes submit stays in nghttp2's queue and leaves on that drain's next turn, through the_drainAgainit already had.ih2_readfails the session past 5000 frames besides DATA, or past 1000 queued SETTINGS and PING acknowledgements (NGHTTP2_DEFAULT_MAX_OUTBOUND_QUEUE_SIZE,NGHTTP2_DEFAULT_MAX_OBQ_FLOOD_ITEM). That stands where ioxide.http2 has its 1 MiB. DATA is bounded by the peer's windows, and a streamed writer by its 64 KiB unframed.FlushEgressAsync(detached: true)) reports a transport fault throughIsBrokeninstead of throwing. Any drain that ends with the connection broken cancels the loop's pending read, and the loop stops without feeding what that read returned. A drain that faults off the reactor ends on it, as http2's pump does.RunAsyncbegun off the reactor first reads from the reactor, where the TCP pipe completes that flush. tests/http2: "h2 RunAsync begun on the thread pool (http2)" fails now and then, once with FRAME_SIZE_ERROR from the server #614 suspects ioxide.http2 lost that by detaching its own.Nghttp2BodyReaderno longer says a handler that never reads holds the connection window down for every other stream. Nothing indocs/needed changing: the nghttp2 pages say every byte is credited once, which still holds.Behaviour a caller would notice
Nghttp2Options.InitialWindowSize,Nghttp2Options.MaxConcurrentStreams. Removed: none. The internalih2_server_newtakes two more arguments (ABI 1).Native libraries
runtimes/linux-x64andruntimes/linux-arm64are rebuilt withscripts/build-nghttp2-native.shat its pinned nghttp2 ref (500edc92, 1.70.90) and its defaultWORK:RID=linux-arm64in a throwawayubuntu:24.04container:gcc-13-aarch64-linux-gnu13.3.0-6ubuntu2~24.04.1cross1,binutils-aarch64-linux-gnu2.42-4ubuntu2.10, cmake 3.28.3-1build7, git 2.43.0.5db0c664b48293584f36f7e485b7671230303a3a757b98822fd68168912b46e97ece43419cb29eab09b6730a1e10e94b0f62423f74bb15caffc167ab15e1c5d168a99fa8cf516d5c063e8c6fe4abfa804a96518117001dff22ad15cd0ad89d45c241bb01f450577688e5d92b2b33a16cf5bfcdc84f56e08470b53b64dc069041ih2_new,ih2_server_newandih2_client_newgrow andih2_abiis new. Every other symbol keeps its size and moves with the code behind them.ih2_symbols plusih2_abi, and built without a warning. The shim also compiles clean with-Wall -Wextra.Tests
New,
Nghttp2ReceiveWindowTests(Unit), over the strict test transport:nghttp2 flow: the server's preface is its SETTINGS, then a WINDOW_UPDATE that opens the connection's window to 2^31-1, in the same writenghttp2: by default the server advertises 100 concurrent streams, each with a 256 KiB window, and Nghttp2Options sets bothnghttp2 flow: concurrent uploads are each held to their own stream window, and a byte past one ends the connection with FLOW_CONTROL_ERROR. Three handlers that do not read each have exactly one 256 KiB window in flight. Two are let go and take 512 KiB each. A byte past the third's window is nghttp2's GOAWAY(FLOW_CONTROL_ERROR), and that handler's read is cut off.nghttp2 flow: every byte comes back once, on nghttp2's top-up half way down the connection's window, whether it was read, dropped, reset or sent after its answer. With the stream windows at 2^31-1: bodies read, dropped unread when their handler answered, dropped by the peer's reset, and sent after the answer, some padded; then stream 1 in full frames up to 1,073,741,823 bytes. No top-up comes before the last byte, and then exactly one, for every byte charged. About a second of the suite.nghttp2 flow: the frame loop goes on reading while a flush is stalled. Behind a flush the peer does not take, a 32 KiB upload is read, and a PING is answered once the flush is out.nghttp2 flow: a stream nghttp2 resets behind a stalled flush ends its handler's read once that flush's drain sends the reset. A body past its content-length, read behind the stall, and nothing more from the peer.nghttp2 flow: a request read behind a stalled flush as the input ends is answered before the connection is torn downNew,
Nghttp2ChaosTests(Chaos), over TCP with the harness's 4 KiB receive buffers, the twin of #593's http2 test:nghttp2: a stream window of upload sent while the server's flush is stalled behind a download the client is not reading arrives wholeChanged, because they counted on the 65,535 connection window:
Nghttp2StreamedRequestTests(Chaos):a handler that has not read holds its own upload to one stream window, while another on the connection runs past it, was...holds its peer to one window...with the stall asserted at 65,535. It now stalls at the 262,144-byte stream window, and meanwhile a 4 MiB upload on stream 3 of the same connection completes.a peer RST_STREAM (...) mid-bodytests anda handler parked on its body when the connection goes awaytook the connection's credit as the sign that the handler had read 32 KiB. With the window open there is none to wait for, so they wait for the stream's, at a 65,535-byte stream window.uploads a handler never reads ...crossed the connection window with five 40 KiB uploads. Each now sends 100 KiB past a 65,535-byte stream window.a padded upload ...runs at a 65,535-byte stream window, so padding never credited would still stall it on its own.a body left unread when its stream is reset ...anda handler that fails mid-upload ...no longer name the connection's credit, which they can no longer see. The Unit top-up test counts it.Nghttp2ChaosTests:a streamed body arrives as separate DATA frameswaits for the server's SETTINGS ACK before its request. A chunk flushed while another drain is in flight leaves with that drain, and with the loop reading on, the drain of the pass that carried the client's SETTINGS can still be in flight when the request's pass runs: the four chunks then share one frame. It failed 2 of 100 runs alone that way (0 of 100 on main), and 0 of 400 with the wait.OffReactorH2FixTests(E2E): the two nghttp2 uploads run at a 65,535-byte stream window throughNghttp2Options, as their http2 twins already do throughHttp2Options.StrictClientgainsEndInput().Each part reverted with the rest in place, Unit and Chaos run whole:
nghttp2_session_set_local_window_size(x64 library rebuilt without it)the preface was [type 4 on 0], not SETTINGS and then a WINDOW_UPDATE on stream 0; the concurrent test,unread, the uploads had [65535, 0, 0] bytes in flight, not a whole 262144-byte window each; the top-up test,the connection ended with a GOAWAY 1098839 bytes in. Chaos, 2: the stalled download,the server did not open its connection window; the held handler,the server took 65535 bytes before its handler read any, not one 262144-byte stream window.Nghttp2Options' defaults back to 1000 and 1 MiBthe SETTINGS advertised 1000 streams of 1048576 bytes, not 100 of 262144, and the concurrent test,[524288, 524288, 524288]in flight.Setuppassing 100 and 256 KiB in place of the optionsthe SETTINGS advertised 100 streams of 262144 bytes, not 7 of 100000, and the top-up test, whose 2^31-1 stream window is lost:the connection ended with a GOAWAY 1098839 bytes in.behind the stalled flush the handler read 0 of the 32768 bytes sent after it. Chaos:the download stopped at 33554432 of 33554432 bytes, and the upload's answer never came, the receive queue having overflowed.the connection was torn down with an answer still queued behind its flushCancelPendingReadat the end of a drain that found the connection brokennghttp2: a pass's flush that reports the peer closed ends the connection, without waiting for its read:the connection is still waiting on a readFireBodyWakesat the end of a drainexpected [was cut off], got [still reading]nghttp2 streamed response: an endless response and a short one on one connection both get through: 100 of 100 runs alone fail withthe short response never ended, and the feed sent 12730174 bytes. 0 of 400 with the await, 0 of 100 on main.InvalidOperationException: ioxide.nghttp2: the native library reports ABI 0, this binding needs 1. A stale libioxide_nghttp2.so is being loaded - rebuild it with scripts/build-nghttp2-native.sh.The new tests use the new options, so they do not build against main; the scratch program above is main's side.
Suites
With the fix, pinned to the safe CPUs, on the commit pushed here:
CI passed on both runners, the arm64 one with this PR's arm64 library: Unit 270, E2E 723 with 7 skipped, Http 104, Pg 26, Redis 6 with 1 pending, Tls 269, File 13, Chaos 106, none failed.
Only Unit, Chaos and E2E reference ioxide.nghttp2. The two Chaos cases whose outcome depends on timing,
a streamed body arrives as separate DATA framesandan endless response and a short one, were also run alone 100 times each on the final code, and the new stalled-download test 40 times: no failures.h2load against
Playground/Http2/Nghttp2Buffered, 2 reactors on CPUs 14-15 and h2load on 16-23, 10 s after 2 s of warm-up, main and this branch alternated twice (the box had other sessions on it):-c 64 -m 16-c 16 -m 8GETs are within noise. Uploads are about 25% faster even over loopback, where a round trip costs almost nothing, and the client receives a third of the bytes per request it did, the difference being WINDOW_UPDATEs.
Overlaps
git merge-tree --write-treeagainst every open PR branch:Nghttp2Connection.cs,Nghttp2BodyReader.cs,Interop/Nghttp2.cs, the shim, UnitProgram.cs), and bothlibioxide_nghttp2.soconflict, as two rebuilds of one library must. A trial merge, its x64 library rebuilt from the merged shim, passed Unit (283, both PRs' tests) and Chaos (106). Whichever merges later merges main and rebuilds both libraries from the merged shim with the script, x64 on the host and arm64 inubuntu:24.04, rather than taking either side's. nghttp2: a request says whether a body follows its HEADERS, so a host need not guess from content-length (#581) #594 changes the shim's callback table, a struct crossing the boundary: if this PR merges first, nghttp2: a request says whether a body follows its HEADERS, so a host need not guess from content-length (#581) #594 bumpsIH2_ABIandNghttp2.Abito 2 with a history line; if nghttp2: a request says whether a body follows its HEADERS, so a host need not guess from content-length (#581) #594 merges first, its library has noih2_abiand reports 0, so 1 covers both.Http2StrictClient.cs, and http2: DATA after a request's END_STREAM resets its stream with STREAM_CLOSED instead of reading as more of its body (#589) #599 (fix-589) registers a test in UnitProgram.cs: both merge cleanly./unreaduploads inNghttp2StreamedRequestTestswith a reset rather than an answer. nghttp2, nghttp3, http3: a synchronous buffered handler that answers null takes down the whole connection, and on h3 an awaited one is never answered #616 changes the buffered dispatch in the two loop files this PR changes. tests/http2: "h2 RunAsync begun on the thread pool (http2)" fails now and then, once with FRAME_SIZE_ERROR from the server #614 points to this loop's awaited SETTINGS flush as why the nghttp2 twin of its failing test passes; it stays awaited.For the reviewer
ih2_readreturnsNGHTTP2_ERR_FLOODED) where ioxide.http2 pauses reading past 1 MiB. A peer that reads its responses never gets near them: per stream it can have the server queue one HEADERS, one WINDOW_UPDATE at a time and one RST_STREAM, besides its acknowledgements.nghttp2_session_get_outbound_queue_sizewould give a managed pause instead, if that is preferred.ih2_is_deadis checked after a pass, and over TCP a cancelled read does not wake a parked one. A peer that then goes silent is held untilReadTimeoutMs, like any silent peer.SendTimeoutMs.IsBrokencheck after a read has no test of its own: what it stops, a pass fed to a connection already known broken, shows nothing a test can see beyond handlers run for no one.