Skip to content

httpclient: a request waiting out a pool's reconnect backoff ends when its reactor stops, rather than waiting forever (#584) - #596

Closed
MDA2AV wants to merge 1 commit into
mainfrom
fix-584
Closed

MDA2AV wants to merge 1 commit into
mainfrom
fix-584

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

Fixes #584.

What was wrong

After a failed connect, HttpClientPool holds new connects back for 200 to 400 ms. A request that found no connection inside that window queued with nothing in flight: only the pool's sweep, a reactor ticker, would open the next connection or fail the request at its acquire deadline. A stopped reactor runs no ticker. Its teardown completes only what is on the ring (it cancels the parked client ops) and the switches still in its post queue. So once the reactor stopped, that request waited forever, its own acquire deadline included. A request queued behind a connect or a response in flight already ended at the stop, with ObjectDisposedException ("The reactor has stopped."), because the stop cancels that op and its failure wakes the waiters. RingHttpClient's per-origin pools share the acquire path and had the same gap.

What changes

src/clients/ioxide.httpclient/Http1/HttpClientPool.cs only.

  • A request about to queue while the gate is shut and nothing is opening starts the retry itself: an open that first waits out the rest of the backoff on the ring (IRingHost.SubmitTimeout on one RingOpSource the pool holds), then connects. The request now waits behind something the stop completes.
  • The retry counts in _opening while it waits, so only one runs at a time, and the sweep's refill stands aside as it does for any open in flight. No new counter or flag. One SQE per failed connect; no syscall, no Task.Delay, no thread-pool hop.
  • When the stop cancels the wait (any result but -ETIME), the retry connects nothing, logs nothing and leaves the backoff alone. It gives back its _live count and wakes the waiters, which then end as they do behind a cancelled connect: each one's yield runs from the stopped reactor's queue on the thread pool and throws ObjectDisposedException. A pool disposed during the wait connects nothing either.
  • The retry starts after the deadline check, on a pass that is going to wait, so a request's last pass does not leave one behind.

What a caller would notice

  • A request waiting on a pool that is backing off now ends when its reactor stops, with ObjectDisposedException ("The reactor has stopped."), as a request queued behind a connect already did.
  • A retry behind the backoff fires when the backoff ends, not on the first 250 ms tick after it. Against an origin whose every open fails, a request waiting 2 s saw 7 to 8 opens with this change and 5 to 6 on main: the same backoff, no storm.
  • ConnectionCount counts the retry while it waits, so a pool backing off with a request queued reports 1 where main reported 0.

Tests

New tests/Ioxide.Tests.Http/HttpClientBackoffTests.cs:

  • httpclient h1: a request waiting out the reconnect backoff ends when its reactor stops: a warm pool of one to a dead port. The request is made on the reactor; 50 ms on, inside the backoff, the test thread stops the reactor and joins it. The request must end with ObjectDisposedException.
  • ring client: a request waiting out the reconnect backoff ends when its reactor stops: the same through RingHttpClient's on-demand pool.
  • httpclient h1: a request waiting on an origin whose every open fails retries at the backoff's pace: an origin that accepts each connection and closes it before the TLS handshake, so every open fails after a connect the origin can count. The request must fail on its 2 s acquire timeout after 3 to 12 opens.

Each part reverted on its own:

  • Without the retry from the acquire path, and on origin/main: both stop tests fail with expected [ObjectDisposedException: The reactor has stopped. Object name: 'Reactor'.], got [still waiting 2 s after its reactor stopped].
  • With the retry connecting at once rather than waiting the backoff out: the stop tests pass, and the pace test fails with 22495 opens in the request's 2 s, where its backoff allows 6 to 11 (25849 on a second run).
  • With a cancelled wait going on to connect: both stop tests fail with got [still waiting 2 s after its reactor stopped]. That connect is submitted from inside the teardown's cancel loop and takes a slot the loop has already passed, so nothing completes it.

The existing dead origin fails inside the acquire budget, no connect storm still passes.

Suites

Built and run pinned to CPUs 0-3, 6-7 and 14-31:

  • Unit: 263 passed, 0 failed, 0 skipped
  • E2E: 730 passed, 0 failed, 0 skipped
  • Http: 107 passed, 0 failed, 0 skipped (104 on main, plus the three above)

For the reviewer

  • No core change. The issue suggested failing the waiters from the reactor's teardown. That needs a new public Reactor hook: teardown walks no registered services (it frees only its TLS services, through an internal list), and httpclient sees no core internals. Keeping a request that waits out the backoff behind something on the ring needs none.
  • A retry outlives a request that gives up while it waits, at its acquire deadline. For an on-demand pool that is one connect nobody asked for, at most one per backoff; main does the same with an open started on a request's last pass.
  • Not covered here, and there on main as well:
    • A connection handed to a waiter in the reactor's last pass reaches it through a yield the stop fails. That connection is never closed, and the waiters behind it, with nothing else in flight, are never woken. With 16 requests looping on a pool of one to a live origin and the stop at a random moment, 13 of 20 trials left 15 of the 16 waiting (6 of 10 on main). Ending them takes either the teardown hook above or a cold path that touches the pool from the thread pool after the stop, so it is left for its own change.
    • At the stop nothing closes a warm pool's idle connections or frees RingHttpClient's TLS context, since only Dispose does that and a stopped reactor drops it. That needs the teardown hook too.

…n its reactor stops, rather than waiting forever (#584)

After a failed connect the pool holds new connects back for 200 to 400 ms, and a request that found
no connection inside that window queued with nothing in flight. Only the pool's sweep, a reactor
ticker, could then open the next connection or fail the request at its acquire deadline, and a
stopped reactor runs no ticker, so the request waited forever. RingHttpClient's per-origin pools
share the acquire path, and had the same gap.

Such a request now starts the retry itself: an open that waits out the rest of the backoff on the
ring, one IRingHost.SubmitTimeout on a RingOpSource the pool holds, and then connects. It counts in
_opening while it waits, so one runs at a time and the sweep's refill stands aside. The stop cancels
the wait, and the retry then connects nothing and reports no failure, but wakes the waiters, which
end with ObjectDisposedException as they already did behind a cancelled connect. A retry behind the
backoff now fires when the backoff ends rather than on the next 250 ms tick, and ConnectionCount
counts it while it waits.

Tests, in Http's HttpClientBackoffTests: a request queued behind the backoff ends with
ObjectDisposedException when its reactor stops, through HttpClientPool and through RingHttpClient;
on main both are still waiting 2 s after the stop. A request waiting 2 s on an origin whose every
open fails sees 3 to 12 opens; with the retry connecting at once it saw 22495.
@MDA2AV

MDA2AV commented Nov 15, 2026

Copy link
Copy Markdown
Owner Author

Superseded by #636: the pools' stop hooks end a request waiting out the reconnect backoff, and both stop tests from here are there. The retry that waits the backoff out on the ring is not needed for the stop. #584 is closed by #636.

@MDA2AV MDA2AV closed this Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

httpclient: a request waiting behind HttpClientPool's reconnect backoff is never ended once its reactor stops

1 participant