Skip to content

tests: the non-reading SSE tests' producers drop on a full queue before WriteTimeout, as the small-events ones do (v1.6.0, Refs #888 #1104) - #1164

Merged
FumingPower3925 merged 5 commits into
mainfrom
flake-sse-nonreading-queue-close
Oct 28, 2026
Merged

FumingPower3925 merged 5 commits into
mainfrom
flake-sse-nonreading-queue-close

Conversation

@FumingPower3925

Copy link
Copy Markdown
Contributor

Flakes lane. middleware/sse TestNonReadingClientEndsStream888/adaptive/async/steady failed in the Coverage job of #1150 (head 323fe30, run 40737498877, job 122989896907): OnDisconnect ran 816.6ms after connect, before WriteTimeout 1s: the stream was ended by something other than the stall bound (epoll was the active engine of the adaptive server). Tests only; the product did what it is configured to do. Refs #888, Refs #884, Refs #1104, Refs #1150

Verdict: the test's producer, not the engine

The 1 s lower bound is right (the stall bound, "no progress for WriteTimeout", must not end a stream earlier; it ends the healthy runs at 1.06 to 1.3 s after connect, never before 1.0 s in about 500 runs here). What ended this stream at 0.82 s was not the stall bound: it was OnSlowClient answering ClientPolicyClose because the 4-slot send queue was full. The "steady" producer sends one 1 KB event every 5 ms starting 700 ms after the handler began; a drain goroutine that is not scheduled for about 20 ms (a runner starved under -race and -covermode=atomic) leaves the queue full, and the producer's own policy closes the stream (700 ms settle + ~80 ms into the trickle, 0.82 s after the dial, with the handler starting a few tens of ms after it). The test already knew this failure: dropEarly ("CI saw cuts at 0.5 s, when the socket fills") makes a full queue drop an event for the first WriteTimeout, so the lower bound tests only the stall bound; only the two small-events producers had it set.

Evidence

  • Natural reproduction: 1 early close in 240 runs of epoll/{async,sync}/steady (docker golang:1.27.3, -race -covermode=atomic, two test processes on --cpus 1; 0 in about 200 runs at --cpus 2, 0.5 and 0.3 and GOMAXPROCS 1 and 4). The one: OnDisconnect ran 787.9ms after connect, before WriteTimeout 1s ... (OnSlowClient answered ClientPolicyClose 777ms after the handler began (a full queue)), i.e. 77 ms into the trickle. The same shape as CI's 817 ms.
  • Deterministic reproduction (commit 2, red on main): a producer steady-flood, which sends 50 events back to back where the trickle starts (what a stalled drain goroutine does to the pace, made certain): fails 20 of 20 on main at 0.72 s with the full-queue message.
  • After (commit 3): steady-flood and the h2c twin pass 50 of 50 (-count=10), the 240-run shape above passes 240 of 240, and one whole pass of TestNonReadingClientEndsStream888, TestSlowReadingClientKeepsStream888, TestNonReadingH2CClientEndsStream884 is green (epoll and std cells; the io_uring cells cannot run in this docker).

Commits

  1. tests: TestNonReadingClientEndsStream888 says whether a full queue ended the stream before WriteTimeout (diagnostic only: the failure message now names the cause).
  2. tests: ... gets a producer whose queue overflows before WriteTimeout (red on main).
  3. tests: the non-reading SSE producers drop on a full queue for the first WriteTimeout, so the lower bound tests the stall bound only: dropEarly on steady, steady-flood and sparse, and on both producers of TestNonReadingH2CClientEndsStream884 (same lower bound, same 5 ms trickle). After WriteTimeout the policy is Close as before, so a stream still ends by the stall bound or by a full queue at or after it; the lower bound is then true by construction.

Not verified / left alone

  • Whether the drain goroutine's stall on that runner was CPU starvation or the engine holding the connection's write lock (the engine's burst flush) is not shown; the product's policy answer is the same either way (the test chose Close).
  • TestSlowReadingClientKeepsStream888 (the control) has the same exposure the other way round: a full queue closes a stream that must live, and its producer (4 KB every 10 ms) has no dropEarly. Not changed here.
  • The adaptive cell cannot run in this docker (no io_uring): the epoll cells of the same producers were used.

Release note

None (tests only).

…ded the stream before WriteTimeout (diagnostic; Refs #888, Refs #1104)
…overflows before WriteTimeout (red on main: the stream ends at 0.8 s; Refs #888, Refs #1104)
…st WriteTimeout, so the lower bound tests the stall bound only (Refs #888, Refs #884, Refs #1104)
@FumingPower3925 FumingPower3925 added this to the v1.6.0 milestone Oct 28, 2026
@coderabbitai

coderabbitai Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Oct 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@FumingPower3925
FumingPower3925 marked this pull request as ready for review October 28, 2026 23:31
@FumingPower3925
FumingPower3925 merged commit 0a69809 into main Oct 28, 2026
30 checks passed
@FumingPower3925
FumingPower3925 deleted the flake-sse-nonreading-queue-close branch October 28, 2026 23:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant