Skip to content

nghttp2: a response that ends before its request asks the client to stop sending the rest (#613) - #635

Merged
MDA2AV merged 2 commits into
mainfrom
fix-613
Nov 15, 2026
Merged

MDA2AV merged 2 commits into
mainfrom
fix-613

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

Fixes #613.

What was wrong

  • A handler that takes an Nghttp2BodyReader is dispatched at its request's HEADERS, so it can answer before the body has arrived: a 401, a 413, a redirect.
  • RFC 9113 8.1 lets a server that has sent a complete response, with the request still arriving, ask the client to stop with RST_STREAM(NO_ERROR) after its END_STREAM. The client keeps the response.
  • ioxide.nghttp2 ended such a response with END_STREAM alone, and nghttp2 sends no reset of its own: after our END_STREAM the stream is half-closed (local) and keeps receiving. So the client sent its whole body, and every DATA frame was credited on arrival and dropped. ioxide.http2 has sent this reset since http2, nghttp3: a response that ends before its request asks the client to stop sending the rest (#571) #582.

Reproduced at main (19aec5c) with a scratch program: .NET's HttpClient over h2c, through a proxy that counts frames, posts 64 MiB from a non-duplex StreamContent to a body-reader handler that answers 413 without reading.

Answer main this PR
streamed, 32 KiB all 67108864 bytes uploaded. END_STREAM, no RST_STREAM, and stream 1 credited +66584576 65535 bytes uploaded. RST_STREAM(NO_ERROR) right after END_STREAM, the stream's last frame. Only stream 0 credited (+65535)
buffered, 32 KiB all 67108864 bytes 65535 bytes, the same frames
buffered, no body (END_STREAM on HEADERS) all 67108864 bytes 65535 bytes, RST_STREAM right after the HEADERS

The controls are the same on both: a body-reader handler that reads an 8 MiB upload to its end, and a handler without a body reader given 4 MiB, get the whole upload and no reset.

The fix

All in src/protocols/ioxide.nghttp2/native/ioxide_nghttp2_shim.c, plus the two rebuilt libraries. No managed code and nothing in src/ioxide changes.

  • on_frame_send. After a HEADERS or DATA frame carrying END_STREAM has gone out, if nghttp2_session_get_stream_remote_close() is 0 (the request is still arriving), it calls nghttp2_submit_rst_stream(..., NGHTTP2_NO_ERROR).
  • Why from that callback. nghttp2 sends a queued RST_STREAM ahead of DATA still queued, so a reset submitted when the handler returns would cut its body off. mem_send runs on_frame_send before it hands the END_STREAM frame back, so the reset leaves in the same drain, right behind it. An answer parked on the client's credit is reset only once its last DATA goes.
  • Where it is registered. Only on server sessions with manual flow control, which is a RunAsync whose handler takes a body reader:
    • Only those handlers are dispatched before their request ends. The others are dispatched at END_STREAM, so the condition could never hold there.
    • Only under manual flow does on_stream_close report a NO_ERROR close after our END_STREAM. That is how the request leaves _pending (CallbackStreamError); its handler has returned by then.
    • The buffered-body mode, the common one, makes no extra call per frame.
  • DATA in flight. After the reset, nghttp2 ignores DATA and trailers still arriving on the stream. It counts that DATA against the connection window and, under manual flow, consumes it itself. So it is credited on the connection only, and never draws a WINDOW_UPDATE on the closed stream (RFC 9113 5.1).

Native libraries

runtimes/linux-x64 and runtimes/linux-arm64 are rebuilt with scripts/build-nghttp2-native.sh, unchanged, at its pinned nghttp2 ref (500edc92, 1.70.90) and its default WORK:

  • x64 on the host: GCC 13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04.1), cmake 3.28.3.
  • arm64 with RID=linux-arm64 in a throwaway ubuntu:24.04 container: gcc-13-aarch64-linux-gnu 13.3.0-6ubuntu2~24.04.1cross1, binutils-aarch64-linux-gnu 2.42-4ubuntu2.10, cmake 3.28.3, DOCKER_CONFIG pointed at a scratch directory.
  • Each toolchain first rebuilt main's shim byte for byte, so each library differs from main's by this change alone. Two builds of this shim are identical on each.
sha256 main this PR
linux-x64 5db0c664b48293584f36f7e485b7671230303a3a757b98822fd68168912b46e9 5c79c0a7366c03d17a4e238789107f65615a8f378344888fe7b12e93d3a28d45
linux-arm64 68a99fa8cf516d5c063e8c6fe4abfa804a96518117001dff22ad15cd0ad89d45 d7dbe1957953dc4172429103f8cb7afed0dcb3708311b5364065cdb09b6284f1
  • In both, the shim's symbols differ from main's only by the new local on_frame_send (0x57 bytes on x64, 0x60 on arm64) and a longer ih2_new.
  • Both still need nothing past libc (and, on arm64, the loader), have a GLIBC 2.17 floor, import the same symbols and export the same 15 ih2_ symbols. Both built without a warning, and the shim compiles clean under -Wall -Wextra with either compiler.

Behaviour a caller would notice

  • A handler given an Nghttp2BodyReader that answers before its request's END_STREAM now has its stream reset with RST_STREAM(NO_ERROR) right after the answer's END_STREAM. The client gets the whole answer and stops uploading. DATA and trailers it already had in flight are ignored. The request leaves the connection at that reset, not at the client's END_STREAM.
  • No handler loses body it could still have read. nghttp2 sends our END_STREAM only after the handler has returned: a streamed writer is completed by the connection when its handler returns, and a buffered answer is the handler's return value.
  • Unchanged: handlers without a body reader, and requests whose body ended (by its last DATA, an empty DATA or trailers) before the answer.
  • API: nothing added or removed.

Tests

New: Nghttp2EarlyAnswerTests (Unit), on the wire over the strict test transport. The response's :status is read with ioxide.http2's HPACK decoder.

  • Reset, each followed by exactly one RST_STREAM(NO_ERROR) after its END_STREAM, as the stream's last frame, with the answer whole:
    • a buffered answer to an upload still arriving is followed by RST_STREAM(NO_ERROR) (END_STREAM on DATA)
    • a bodyless answer, END_STREAM on its HEADERS, to an upload still arriving is followed by RST_STREAM(NO_ERROR)
    • a streamed answer to an upload still arriving is followed by RST_STREAM(NO_ERROR)
    • the 500 of a handler that faulted with its upload still arriving is followed by RST_STREAM(NO_ERROR) (parked, so written by its own drain)
    • with a flush in flight, the reset still follows END_STREAM, once
  • Controls, not reset: a request ended by its last DATA, by trailers, by an empty DATA, and a GET ended by its HEADERS.
  • Guards:
    • a buffered answer parked on credit is reset only after its END_STREAM, and arrives whole. The upload goes on, and is credited, while the answer waits.
    • a buffered / streamed answer to an upload the peer reset gets nothing back (RFC 9113 5.4.2)
    • a handler that fails after its headers, its upload still arriving, is reset once, with INTERNAL_ERROR
    • DATA in flight after the reset comes back on the connection, never on its closed stream. A connection window's worth sent after the reset draws no frame on stream 1. Then stream 3's body fits only if that DATA was credited on the connection.

New: Nghttp2EarlyAnswerTests (E2E), with .NET's HttpClient over h2c posting 64 MiB from a StreamContent to a handler that answers 413 unread:

Changed, Chaos:

  • H2cClient.Upload reports HEADERS followed by RST_STREAM(NO_ERROR) as a new UploadEnd.Stopped, where it used to return Reset with code 0. No caller expected that.
  • nghttp2 streamed request (...): uploads a handler never reads give their connection credit back sends its unread uploads to handlers that answer without reading. Those uploads can now be stopped at their answer, so the test accepts Stopped on its two unread paths. /read must still be answered whole.

Reverting the fix.

What is reverted What fails
the whole fix (main's library, the new tests in place) Unit: 270 passed, 7 failed. The five reset tests and the two guards that need the reset (parked on credit, DATA in flight) fail with the answer ended with its request still arriving, and no RST_STREAM asked the client to stop uploading. The controls and the other guards pass. E2E: the three stop tests fail with the client uploaded 67108864 of 67108864 bytes to a handler that read none of them: nothing asked it to stop, and the control passes. That run used a scratch runner that compiles just this file, since the E2E suite takes no filter
the END_STREAM condition (a reset after any HEADERS or DATA) Unit: 271 passed, 6 failed. the answer never ended for the buffered, streamed, flush-in-flight and in-flight-DATA tests, expected [16384], got [0] for the answer parked on credit, and stream 1 was reset with [0x0], where the stack's own reset is one 0x2 for the handler failing after its headers
the remote-close condition (a reset after every END_STREAM) nothing, see below

With the fix, main's version of the chaos test fails on both flavors:

  • /unread: upload 5 of 5, 160 KiB already sent, ended Reset (error 0x0) after 36793 of 40960 bytes
  • /unread: upload 3 of 5, 80 KiB already sent, ended Reset (error 0x0) after 16384 of 40960 bytes

The uploads before those ended before their answer, which is why the new version accepts either end. The new version passes against main's library and with the fix.

Suites

With the fix, pinned to the safe CPUs:

  • Unit: 277 passed, 0 failed, 0 skipped (the 263 on main and the 14 new)
  • E2E: 734 passed, 0 failed, 0 skipped (the 730 on main and the 4 new)
  • Chaos: 105 passed, 0 failed, 0 skipped

Overlaps

git merge-tree --write-tree of this branch with #594 to #604 and the open PRs from #622 to #633:

For the reviewer

  • The remote-close condition changes nothing on the wire. nghttp2_submit_rst_stream queues its reset with continue_without_stream = 0. A stream whose request already ended closes inside session_after_frame_sent1, right after our END_STREAM. So nghttp2 drops the queued reset in session_prep_frame before it leaves, and with the condition removed every test still passes.
    • It stays because it is RFC 9113 8.1's condition: without it, the reset would depend on that nghttp2 internal not to go out after every answer.
    • It also saves allocating and queuing an RST_STREAM item for every answer to a request that has ended, which under manual flow is nearly all of them.
  • Bodyless answers. For an END_STREAM on HEADERS, nghttp2 sets the stream back to OPENED after the callback that queued the reset, which had marked it CLOSING. The reset still leaves in the same ih2_write loop, before anything is read, so nothing can arrive in between. The bodyless tests cover that path.
  • CallbackStreamError now also runs for our own NO_ERROR reset, after the answer is out. It aborts the reader, which credits once whatever the handler left unread, as its retire would have, and marks the completed writer gone (OnPeerReset). Its "the peer gave up" comment was already loose: it also hears nghttp2's own resets and the INTERNAL_ERROR ones.
  • No docs change. The request-streamed nghttp2 pane says every byte nghttp2 delivers is credited once (read, left unread by a handler that returned, or on a reset stream), and that still holds. DATA after the reset is never delivered: nghttp2 credits it itself, as it does padding.
  • ih2_client_new shares ih2_new, but the callback is registered only for a server. Nothing managed calls the client.

…top sending the rest (#613)

A handler that takes an Nghttp2BodyReader runs at its request's HEADERS and can answer before the
body has arrived: a 401, a 413, a redirect. Its response then ended with END_STREAM alone, and
nghttp2 sends no reset of its own, so the stream stayed half-closed (local) and every DATA frame the
client went on sending was credited and dropped: a 64 MiB HttpClient POST answered with a 413 its
handler never read still uploaded all 64 MiB. RFC 9113 8.1 lets a server that has sent a complete
response ask the client to stop with RST_STREAM(NO_ERROR) after its END_STREAM; ioxide.http2 has
done so since #582.

The shim now does it. nghttp2 sends a queued RST_STREAM ahead of DATA still queued, so the reset is
submitted from an on_frame_send callback once a HEADERS or DATA frame carrying the response's
END_STREAM has gone out, and only while the request is still arriving
(nghttp2_session_get_stream_remote_close is 0). It leaves in the same drain, right behind the
END_STREAM. The callback is registered on server sessions with manual flow control only: those are
the sessions whose handlers are dispatched before their request ends, and the only ones whose
on_stream_close reports a NO_ERROR close after the response's END_STREAM, which is how the managed
side lets go of the request. DATA the client had in flight is then ignored by nghttp2 and credited
on the connection only, never on the closed stream.

Both libioxide_nghttp2.so are rebuilt with scripts/build-nghttp2-native.sh at its pinned nghttp2
(500edc92, 1.70.90) and default WORK: linux-x64 on the host with GCC 13.3.0, linux-arm64 in
ubuntu:24.04 with gcc-13-aarch64-linux-gnu 13.3.0. Each toolchain first rebuilt main's shim byte for
byte, and two builds of this one are identical. Each library differs from main's by the new callback
and the grown ih2_new; NEEDED, the GLIBC 2.17 floor and the 15 ih2_ exports are unchanged.

Tests:
- Unit Nghttp2EarlyAnswerTests, on the wire: a buffered answer, a bodyless one (END_STREAM on its
  HEADERS), a streamed one, the 500 of a handler that faulted, and an answer given behind a flush in
  flight are each followed by one RST_STREAM(NO_ERROR) after their END_STREAM, as the stream's last
  frame. Controls: a request ended by its last DATA, by trailers, by an empty DATA, or by its
  HEADERS is not reset. Guards: an answer parked on credit is reset only after its END_STREAM and
  arrives whole; an upload the peer reset gets nothing back, buffered or streamed; a handler that
  fails after its headers is reset once, with INTERNAL_ERROR; and DATA in flight after the reset
  draws no frame on its stream and is credited on the connection.
- E2E Nghttp2EarlyAnswerTests, with .NET's HttpClient over h2c posting 64 MiB from a StreamContent
  to a handler that answers 413 unread, streamed, buffered and bodyless: the answer arrives whole and
  the upload stops, at 64 KiB in a scratch run. Control: an upload read to its end is sent whole and
  answered.
- Chaos: H2cClient's Upload reports an answer followed by RST_STREAM(NO_ERROR) as Stopped, where it
  returned Reset with code 0. The unread-uploads test accepts that end on its two unread paths.

Against main's library, the five reset tests and the two guards that need the reset fail with "the
answer ended with its request still arriving, and no RST_STREAM asked the client to stop
uploading", and the three E2E stop tests with "the client uploaded 67108864 of 67108864 bytes",
while the controls and the other guards pass. With the END_STREAM condition dropped, the reset
follows the response's HEADERS and cuts its body off: six tests fail. With the fix, the chaos test
as it was fails: an unread upload ends Reset (error 0x0).
# Conflicts:
#	src/protocols/ioxide.nghttp2/runtimes/linux-arm64/native/libioxide_nghttp2.so
#	src/protocols/ioxide.nghttp2/runtimes/linux-x64/native/libioxide_nghttp2.so
@MDA2AV

MDA2AV commented Nov 15, 2026

Copy link
Copy Markdown
Owner Author

Merged main in after #594, which changed the same shim: libioxide_nghttp2.so is rebuilt from the merged ioxide_nghttp2_shim.c (both changes) with the build script's default WORK. linux-x64 sha256 29e065659ecd1991..., the same from the host and from the cross-build container; linux-arm64 bada4432386781bb..., from the container. The same setup first rebuilt #594's committed x64 and arm64 binaries byte for byte. All suites passed on the merged tree.

@MDA2AV
MDA2AV merged commit bea806d into main Nov 15, 2026
2 checks passed
@MDA2AV
MDA2AV deleted the fix-613 branch November 15, 2026 17:17
@MDA2AV MDA2AV mentioned this pull request Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

nghttp2: a response that ends before its request does lets the client upload the rest, which is credited and dropped

1 participant