Repository navigation
Closed
Conversation
…dropped, rather than spinning forever on a queue nothing drains (#607) Off the reactor, EnqueueReturnQ and EnqueueReturnQIncremental spin until their queue takes the buffer, and only the loop drains those queues. After the stop every return from another thread kept its entry, so once the 16384 entries of the shared ring's queue, or the 65536 of the incremental one, were taken, the next return spun forever and held its thread: a handler that held buffers across the stop and gave them back from the thread pool never finished. A return that finds its queue full now gives up once the reactor's posts are closed, at the end of Teardown, when nothing will drain the queue and no ring is left to take the buffer. The test sits inside the spin, so a return that finds room runs the same code as before. StoppedReturnTests (Unit): a return that finds the queue full waits while the reactor runs, and leaves when the stop lands, in both modes. StopParkedTests (E2E): handlers holding more buffers than the queue takes, 17000 on the shared ring and 3 x 22000 slices incremental, give them all back from the thread pool after the stop.
This was referenced Nov 14, 2026
Owner
Author
|
Closed in favour of #634: a stop that drains, then frees everything in one place, instead of a fix per after-stop path. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #607.
Touches core: src/ioxide/Reactor/Loop/Reactor.Drainers.cs and src/ioxide/Reactor/Loop/Reactor.Loop.Incremental.cs, because the return queues, their spin and the flag the stop sets are the reactor's own: every recv buffer a handler gives back from another thread goes through
EnqueueReturnQorEnqueueReturnQIncremental, and no satellite module sees either queue.What was wrong
Off the reactor,
EnqueueReturnQspins until_returnQtakes the buffer id, andEnqueueReturnQIncrementaldoes the same on_returnQInc. Only the loop drains those queues. After the stop every return from another thread kept its entry, and once all 16384 (shared ring) or 65536 (incremental) were taken, the next return spun forever and held its thread, typically a thread-pool worker.Reproduced at main 19aec5c with a scratch program: a real reactor whose handlers hold recv buffers across the stop and give them back from the thread pool afterwards.
RecvSlots32768 (the most the reactor accepts; at the default 4096 this queue cannot fill, since each id is out at most once), one connection holding 17000 buffers: 16384 of the 17000 returns went through, and the next was still spinning 5 s later.What changes
A return that finds its queue full gives up once
_postsClosedis set. The end of Teardown sets it, and from then on nothing drains the queue and no ring is left to take the buffer. The test sits inside the spin:EnqueueReturnQ(x64, Dynamic PGO) the two versions are identical up to the full-queue branch, which gains one byte compare and a jump (257 to 268 bytes).Returns after the stop that still find room take an entry, as before. Nothing reads them, and the queue is a managed array that goes with the reactor.
Behaviour a caller would notice
Only after the stop: a buffer return from another thread that finds the queue full now returns, where it never returned. While the reactor runs nothing changes, and a full queue still makes a return wait for the loop's drain.
Tests
Each fails when its half of the fix is reverted, and passes with it. Reverting the shared half fails exactly the two shared-ring tests; reverting the incremental half fails exactly the two incremental ones.
StoppedReturnTests: "reactor: a buffer return that finds its queue full waits while the reactor runs, and leaves when it stops (shared ring / incremental)". The real returns, on a reactor that never runs: the test fills the queue, starts one more return, checks it is still waiting 200 ms later, then sets the flag the end of Teardown sets and requires that return to finish within 3 s. Reverted:a return waiting on the full queue still spins after the stop, where nothing will drain it. It also catches the two near misses the E2E test cannot see: dropping a return on any full queue fails witha return that found the queue full while the reactor runs was dropped instead of waiting for room, and testing the flag only before the spin fails with the first message.StopParkedTests: "stop: handlers holding more buffers than the return queue takes give them all back from another thread after the stop (shared ring / incremental)". The repro as a test, with one-byte buffers so each byte sent is a buffer, or a slice, of its own. Reverted:16384 of the 17000 buffers went back after the stop, and the next return still spins on a queue nothing drains(shared ring) and65536 of the 66000 buffers went back after the stop, and the next return still spins on a queue nothing drains(incremental). The incremental one skips below kernel 6.12, as the other incremental stop test does.Suites
Built and run pinned to the safe CPUs, one at a time:
CI passed on x64 and arm64, the four new tests among them on both. The first arm64 attempt failed one E2E test, "h2 off-reactor (nghttp2): many streams flushing from the thread pool at once all resume on the reactor", right after
[conn] recv queue overflow; closing connection.: the client's frames filled the connection's 64-entry receive queue before the frame loop read them, which in the shared receive mode closes the connection. That happens while the reactor runs, where this change does nothing, and the rerun of that job passed.Overlaps
git merge-tree --write-treeof this branch with each of fix-581 to fix-592 (nghttp2: a request says whether a body follows its HEADERS, so a host need not guess from content-length (#581) #594 to http2: an answer waiting on the client's credit has its unread upload credited, so a client that uploads everything first gets it (#588) #604) is clean. nghttp2: a request says whether a body follows its HEADERS, so a host need not guess from content-length (#581) #594, http2: an answer waiting on the client's credit has its unread upload credited, so a client that uploads everything first gets it (#588) #604 and http2: DATA after a request's END_STREAM resets its stream with STREAM_CLOSED instead of reading as more of its body (#589) #599 also register Unit tests in tests/Ioxide.Tests.Unit/Program.cs, at other lines.EnqueueRecycle, a different method.For the reviewer
_postsClosed, the one point after which nothing drains the queue, which the stopped posts and ops already use._stopRequestedcomes earlier, while the loop may still be on its last pass.