Repository navigation
Conversation
…500 on its own stream instead of ending the connection or going unanswered (#616) A buffered handler that answers null made the stack throw a NullReferenceException at its first read of the response, before anything was submitted. Only nghttp2's parked path caught it. nghttp2: answered synchronously, as nearly every handler is, it left DispatchBuffered's try/finally, and the read loop took it for a broken peer. It freed the session without a GOAWAY, losing every other stream on the connection and what the pass had submitted, and told nothing to stderr. nghttp3 and http3: answered synchronously or with a completed task, it left the dispatch and the run loop's try/finally, so the task RunAsync returned faulted. The teardown sent no CONNECTION_CLOSE, since no error code had been recorded, so the client heard nothing and every request on the connection waited out its idle timeout. Answered after an await, it faulted the task that finishes the request, which nothing awaits, outside its catch: that stream got neither a response nor a reset. Each stack now hands every buffered answer, synchronous, completed or awaited, to one helper that tests it for null first. A null answer gets what a handler that threw gets: the fault is reported with a NullReferenceException (OnHandlerFault on nghttp3, else stderr; stderr on nghttp2 and http3) and the stream gets a 500. nghttp2's parked path keeps its catch, and its body is the helper both paths share. The test comes before the submit, rather than a catch around it, because a catch stops the JIT inlining the submit's P/Invoke. Nothing else thrown while a response is handed over is handled differently.
# Conflicts: # src/protocols/ioxide.http3/Http3Connection.cs # tests/Ioxide.Tests.E2E/Protocols/Http2BufferedFaultTests.cs
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #616.
A buffered handler that answers
nullmade each of these stacks throw aNullReferenceExceptionat its first read of the response (PackHeaderson nghttp2 and nghttp3,resp.HeadIsValidin ioxide.http3'sSubmit), before anything was submitted. Only nghttp2's parked path caught it, inCompleteAsync. Everywhere else:RunAsync(Func<Nghttp2Request, Nghttp2Response>), and eitherValueTaskoverload when the handler's task is already complete, as nearly every one is. The exception leftDispatchBuffered'stry/finally, andRunCoreAsync'scatchtook it for a broken peer: it marked the connection failed and freed the session. Requests later in the same read were never dispatched, parked handlers' answers had nowhere to go, nothing the pass had submitted was drained, and the TCP connection closed with no GOAWAY, which RFC 9113 6.8 says should come before a close. stderr heard nothing. .NET'sHttpClientsaw "The response ended prematurely while waiting for the next frame from the server. (ResponseEnded)" on every request in flight and opened a new connection for the next one.DispatchBufferedSyncandDispatchBufferedAsync's fast path on nghttp3,DispatchReadyandDispatchReadyAsync's fast path on http3) and thenRunCoreAsync'stry/finally, so the taskRunAsyncreturned faulted and the reactor logged "quic handler faulted". The teardown sent no CONNECTION_CLOSE, since nghttp3 sends one only once an error code is recorded and http3 only after a fatal protocol error. So every request on the connection, the null one included, got neither a response nor a reset, and the client could only wait out its idle timeout (RFC 9000 10.1). nghttp3'sOnHandlerFaultheard nothing.CompleteBufferedAsyncsubmits outside itstry, so the exception faulted that task, which nothing awaits. The stream got no final response, after which RFC 9114 4.1 has the server close it, and no reset either: the client waited on it until its own timeout. The connection served its other requests.The fault concerns one response, and HTTP answers it with a 500 (RFC 9110 15.6.1); even a protocol error that affects only one stream is a stream error, not a connection error (RFC 9113 5.4.2, RFC 9114 8).
The change. As #603 does for ioxide.http2: a null answer gets a 500 on its own stream and is reported like a handler that throws, and nothing else thrown while a response is handed over is handled differently.
src/protocols/ioxide.nghttp2/Connection/Nghttp2Connection.RunBuffered.cs:DispatchBuffered's synchronous path submits throughSubmitAnswer, which tests the answer for null first. A null answer goes toAnswerFaultwith aNullReferenceException: stderr hears "[ioxide.nghttp2] request handler faulted: ...", and the stream gets a 500 unless the connection is broken.AnswerFaultis the body ofCompleteAsync'scatch, which now calls it, and isNoInliningso the cold path stays out of the dispatch's tier-1 code.src/protocols/ioxide.nghttp3/Connection/Nghttp3Connection.RunBuffered.cs: the three places a buffered answer is queued,DispatchBufferedSync,DispatchBufferedAsync's completed fast path andCompleteBufferedAsync, callQueueAnswer, which tests for null.AnswerNull, out of line, tellsOnHandlerFault(or stderr without one) with aNullReferenceExceptionand queues a 500.CompleteBufferedAsync'sstreamIdlocal only fed the call this replaces, so it goes.src/protocols/ioxide.http3/Http3Connection.cs: the same three places,DispatchReady,DispatchReadyAsync's fast path andCompleteBufferedAsync, callSubmitAnswer;AnswerNullwrites the stderr line every other fault writes and submits a 500.Whatever follows an answer at those places follows the 500 too: the request is retired or recycled, and on nghttp3 an upload still arriving is asked to stop with STOP_SENDING(H3_NO_ERROR). Nothing changes in
src/ioxide, in a native shim or in the public API, and no state is added.What a caller notices.
NullReferenceException, the exception ioxide.http2 reports for it with http2: a synchronous buffered handler that answers null gets a 500 on its own stream instead of ending the connection (#592) #603: to nghttp3'sOnHandlerFault, or to stderr, which is all nghttp2 and http3 have. It is created, not thrown, so it carries no stack trace.RunAsyncreturns no longer faults with that exception; it ends when the connection does.What to weigh.
NullReferenceException; here each stack tests the answer before it hands it over. The rule is the same, only a null answer and before anything is submitted, but on the native stacks acatchcosts every response: on x64 the JIT does not inline a P/Invoke inside atrythat has acatch. A first version of this fix caught as http2: a synchronous buffered handler that answers null gets a 500 on its own stream instead of ending the connection (#592) #603 does, and under h2load the tier-1DispatchBuffered(nghttp2) andDispatchBufferedSync(nghttp3) calledih2_submit_responseandih3_submit_responsethrough their IL stubs, where main inlines them. In a microbenchmark of a blittable P/Invoke (libclabs), a call inside atrywith a filteredcatchtook 4.80 ns, against 1.58 ns inside atry/finally. On ioxide.http3, which has no P/Invoke there, thecatchwas inlined intoDispatchReadyand grew its tier-1 code from 1192 to 1404 bytes. With the test, read withDOTNET_JitDisasmSummaryunder the same load:Nghttp2Connection.DispatchBufferedgoes from 1974 to 2019 bytes,Nghttp3Connection.DispatchBufferedSyncfrom 1923 to 1960,Http3Connection.DispatchReadyfrom 1187 to 1218. Both submits stay inlined P/Invokes, and each null path is one call (AnswerFault,AnswerNull). If you would rather all four stacks read alike, thecatchis a few lines in place of each helper, at the cost above.NullReferenceExceptionis thrown and caught, here it is created. Both report the same line.Playground/Http2/Nghttp2Buffered,Playground/Http3/Nghttp3BufferedandPlayground/Http3/ManagedBuffered, all three on the synchronous buffered overload, each with 2 reactors on CPUs 2-3 and h2load-t6 -c64 -m32(h2c, or h3 through an HTTP/3 build of h2load) on six other CPUs, in 10 s runs ordered fix, main, main, fix: 4 rounds on nghttp2, 3 on each h3 stack. Other jobs shared the box (load 1 to 6) and a few runs were plainly disturbed, so these are medians of CPU per request: nghttp2 1.407 us against main's 1.398 (+0.6%), nghttp3 3.988 against 3.992 (-0.1%), http3 1.986 against 1.998 (-0.7%). Each is inside the spread of main's own runs (1.374 to 1.478, 3.919 to 4.738, 1.985 to 2.204). A few ns a response is below what this box can resolve, so the case against thecatchrests on the code it compiles to, above.Tests.
Nghttp2NullAnswerTests, overStrictClient:nghttp2: a synchronous handler that answers null is answered 500 on its own stream, the fault reaches stderr, and the next stream in the same read is served. GET 1 and GET 3 arrive in one read and stream 1 answers null. Stream 1 gets exactly one frame, HEADERS with END_STREAM whose block is:status 500alone; stream 3 gets "ok"; stderr hears the fault; the connection is still up.Http2BufferedFaultTests, the nghttp2 twins of http2: a synchronous buffered handler that answers null gets a 500 on its own stream instead of ending the connection (#592) #603's two, over .NET'sHttpClientand h2c:h2 fault (nghttp2): a synchronous handler that answers null answers 500, and the connection serves on: /null gets 500 and /ok 200 "ok", on one connection.h2 fault (nghttp2): a request in flight beside a handler that answers null without parking is answered: /slow parks on aRingTimer, then /null answers with a completed null. /null gets 500 and /slow 200 "slow", on one connection.H3NullAnswerTests, new:H3ErrorCodeTests.RecordingQuic), for each stack and each buffered shape (the synchronous overload; theValueTaskoverload, completed and awaited; the body-reader overload, completed and awaited), 10 tests: stream 0 answers null and stream 4 "ok", both in one pass. Stream 0 must get a bare 500 with FIN and stream 4 its 200, the run loop must still be running and end cleanly once the transport closes, with no CONNECTION_CLOSE, and the fault must reach nghttp3'sOnHandlerFaultas aNullReferenceExceptionfor stream 0, or http3's stderr once.Against main's code all 17 new tests fail. The nghttp2 ones with "the connection ended on a null answer" (Unit) and "expected [500: ], got [reset: The response ended prematurely while waiting for the next frame from the server. (ResponseEnded)]" (both E2E); the h3 ones, synchronous or completed, with "the run loop ended on a null answer: Object reference not set to an instance of an object.", and awaited with "expected [500 []], got [nothing]"; and the four over QUIC with "the request whose handler answered null got no answer, not a bare 500".
Seen failing, one part of the fix at a time, on the final code:
SubmitResponse: the Unit test ("the connection ended on a null answer") and both nghttp2 null tests in E2E ("expected [500: ], got [reset: The response ended prematurely while waiting for the next frame from the server. (ResponseEnded)]").AnswerFaultwithout its 500: the Unit test ("stream 1 was sent [], not one HEADERS that ends it"), both null tests in E2E, and the two existing nghttp2 tests of a handler that throws ("The request was canceled due to the configured HttpClient.Timeout of 10 seconds elapsing."). So the helper is covered from both paths.AnswerFaultwithout its stderr line: the Unit test ("stderr had: []").QueueResponsein turn.DispatchBufferedSync: the synchronous recorded test ("the run loop ended on a null answer: Object reference not set to an instance of an object.") and the synchronous test over QUIC ("the request whose handler answered null got no answer, not a bare 500"). The fast path: the two completed-task recorded tests, with the same message.CompleteBufferedAsync: the two awaited recorded tests ("expected [500 []], got [nothing]") and the awaited test over QUIC.AnswerNullwithout the hook call: the five recorded nghttp3 tests ("expected [stream 0: NullReferenceException], got []"). Without the 500: those five ("expected [500 []], got [nothing]") and both nghttp3 tests over QUIC.Submitin turn: the same tests as on nghttp3, with the same messages.AnswerNullwithout its stderr line: the five recorded http3 tests ("stderr had: []"). Without the 500: those five and both http3 tests over QUIC.Suites. Unit 264/0, E2E 746/0, Chaos 105/0 and Http 104/0, nothing skipped, on a pinned Release build of the final commit; that is main's 263 and 730 plus the new tests. Http is in because its cross-stack tests serve through nghttp3. The build adds no warnings: the solution's seven are main's (CA2014, CA2022, CA1416). Tls, File, Pg and Redis weren't run, since none of them reaches these three stacks.
Overlaps.
src/protocols/ioxide.http3/Http3Connection.cs: it addsStopReadingIfAnsweredEarly(rs);right after the twoSubmitcalls this PR turns intoSubmitAnswer, inDispatchReadyAsync's fast path and inCompleteBufferedAsync. Keep both,SubmitAnswer(...)thenStopReadingIfAnsweredEarly(rs);, so a null answer's 500 to an upload still arriving is followed by STOP_SENDING(H3_NO_ERROR), as http3: a response that ends before its request asks the client to stop sending the rest (#587) #601 has it for a handler that throws. Checked: with that resolution the merged tree builds, and this PR's h3 tests and http3: a response that ends before its request asks the client to stop sending the rest (#587) #601's (Http3EarlyAnswerTests,EarlyAnswerTests, plusH3StopSendingTestsandH3BufferedFaultTests) pass, 60/0, as does a scratch check that a null answer, completed or awaited, to an upload still arriving is "send+fin, stop 0x100" on http3: a response that ends before its request asks the client to stop sending the rest (#587) #601's recording transport.tests/Ioxide.Tests.E2E/Protocols/Http2BufferedFaultTests.csonly, since its source change is in ioxide.http2. Both PRs add the same two null tests after the loop over the stacks, http2: a synchronous buffered handler that answers null gets a 500 on its own stream instead of ending the connection (#592) #603 for ioxide.http2 and this one for nghttp2, and qualify the summary andStart's comment with their stack; theAnswer,IsNullandWholeConnectionedits are the same on both sides. Keep all four tests, or fold the pairs into the loop since both stacks then answer null alike, and drop the stack from the two comments. Checked with the pairs folded into the loop: the merged tree builds, andHttp2BufferedFaultTests(8 tests), this PR's Unit test and http2: a synchronous buffered handler that answers null gets a 500 on its own stream instead of ending the connection (#592) #603'sHttp2EarlyAnswerTestspass, 23/0.git merge-tree); the test registrations in the twoProgram.csfiles sit apart. nghttp2: the connection's receive window stays at 65,535, so a connection has at most 64 KiB of request body in flight across its streams #612 and nghttp2: a response that ends before its request does lets the client upload the rest, which is credited and dropped #613 citeNghttp2Connection.RunBuffered.cstoo, atRunCoreAsyncandRefuseOverflowed, which this PR doesn't change; their branches weren't pushed when this was checked.