Skip to content

tests: the adaptive engine's fixed bind polls wait on Listen (the direct-engine polls are a follow-up) - #1149

Merged
FumingPower3925 merged 4 commits into
mainfrom
flakes-e-bind-polls
Oct 28, 2026
Merged

FumingPower3925 merged 4 commits into
mainfrom
flakes-e-bind-polls

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Flakes lane, package E: fixed 3 to 15 s waits for a REAL adaptive engine to publish its address become the Listen-aware wait of #1128, in 10 test files of internal/adaptive, internal/engine/epoll and internal/engine/iouring. A test-hygiene change, no engine change. It does NOT convert every fixed bind poll in the repository (the rest is listed below as a follow-up). Not reproduced as a failure (see Evidence). Refs #1104

What changes

#1128 added awaitBind890 / listenAndAwaitBind890 in internal/adaptive because a 3 s poll was missed once in 400 runs at --cpus 2 with two busy loops (celeris#890 item 98): a poll that may have been starved, or a Listen that was slow, with no way to tell which. The new wait polls every 10 ms for up to 30 s (bindLimit890, a guard against a hang), stops at once when Listen returns before binding and fails with Listen's own error, and logs how long the bind took. Listen gives up by itself after 20 s (listenBindDeadline), so the 30 s is never the bound that matters.

Converted (a real adaptive engine, polled with a fixed limit that did not follow Listen)

site was
internal/adaptive/async_abort_791_linux_test.go (abortAdaptive791; also called by the child-process arm of idle_follow_switch_standby_804) 3 s
internal/adaptive/driver_provider_test.go (newBoundAdaptive, 5 tests) 3 s
internal/adaptive/detect_bound_974_linux_test.go 10 s
internal/adaptive/prebound_listener_test.go (preBoundAdaptive, 2 tests: TestAdaptiveStartsWithPreBoundListener, TestAdaptiveSwitchesWithPreBoundListener; TestAdaptiveRejectsNonTCPListener only calls New) 10 s
internal/adaptive/switch_idle_conns_linux_test.go (3 tests) 5 s
internal/adaptive/pinned_threads_905_linux_test.go 10 s
internal/adaptive/epoll_halves_e9_linux_test.go (4 tests) 15 s
internal/engine/epoll/adaptive_abort_844_linux_test.go (3 tests) 5 s
internal/engine/iouring/adaptive_abort_844_linux_test.go (the twin of the epoll test, 3 tests) 5 s
internal/engine/iouring/adaptive_switch_848_linux_test.go 5 s

The internal/adaptive ones use awaitBoundOrFail890 (new, next to the others): listenAndAwaitBind890 for a test that starts Listen itself and keeps its cancel and done channel to assert on the stop. The Listen goroutine of those tests closes an ended channel after it sends on done. The teardowns of switch_idle_conns, epoll_halves_e9 and detect_bound_974 wait on ended now, not on done: on a failure the helper drains done to report Listen's error, and a teardown waiting on it printed a false "did not stop" after 10 to 15 s per test. The two engine packages cannot import internal/adaptive's test helper (their tests are in other packages), so each has a small copy, awaitAdaptiveBound844 (30 s, returns at once with Listen's error and puts the value back on done for the teardown).

NOT converted (stated, not claimed)

  • internal/adaptive/shutdown_during_bind_wait_linux_test.go:107 (5 s): a MOCK primary, and its Listen runs with context.Background() (no cancel for the helper). Low risk.
  • internal/adaptive/h2_dial_race_test.go:113 and pauseaccept_h2_dial_race_test.go:73: 60 s polls, not Listen-aware (a failing Listen costs 60 s before "did not bind"); they do not have the 3 to 10 s weakness.
  • Already Listen-aware, with a longer fixed limit, left as they are: internal/adaptive/metrics_listen_race_linux_test.go:66 (15 s), port_hold_616_linux_test.go:176 (25 s), iouring_floor_linux_test.go:184 (25 s, listenFloor682, which turns an early Listen return into a skip or failure with the engine's error), test/integration/adaptive_hybrid_test.go:270 (20 s).
  • Not bind waits: driver_provider_test.go:327 (a 3 s loop of ForceSwitch calls) and prebound_listener_test.go:179 (the 30 s wait for a switch).
  • A follow-up, not done here: about 35 fixed polls of a DIRECT engine (Addr() == nil loops) in internal/engine/epoll (for example adopt_suspended_loop:117 10 s, async_handler_stall:727, async_largebody:85/151, backpressure:101, closing_e2e_876:54, closing_h2_restamp_876:123/308, closing_slow_reader_876:57, dispatch_bench_771:52, drainread_flood_885:85 5 s, driver_read_helpers_930:164, hijack_flush_936:102, hijack_inline_async_loop:107, hijack_offthread:332, keepalive_writeprogress_885:99, linger_arrivals_placement:148, nic_irq_rxfree_1036:110, pause_accept_idle:175, pause_accept_linger:858, pause_accept_queued:129, poolwait_gate_unread_885:408, recv_shapes_bench_884:47, sendfile_e2e:103 3 s, shutdown_counters_e2e_874:56, transplant_accounting:189 3 s, transplant_handoff_failure:274, driver_test:58-68), the same in internal/engine/iouring and internal/engine/std (std/h2c_goaway_878:130 has no deadline at all). They are a mechanical sweep with internal/engine/epoll's existing awaitBound.
  • Listen-aware, but with a short fixed limit (3 or 5 s, the "slow Listen" half of the weakness only): test/conformance/helper_test.go:95, test/spec/helpers_test.go:118, test/integration/driver_integration_test.go:363, drivers_shared_loop_test.go:73, per_handler_async_test.go:69, test/drivercmp/{redis,postgres}/integrated*_test.go, epoll/async_h2_notice_engcfg_linux_test.go:65, resume_wake_886:111, iouring/pause_wake_886:46, and the Server-level waitServerStarted/waitForReady callers.
  • internal/engine/iouring/errorclass_linux_test.go:180 (5 s) turns a bind timeout into t.Skip("engine never bound"), which can hide a real failure: it deserves a look of its own.

Evidence

I could not make the old waits fail. The bind takes 10 to 21 ms here (the helper logs it: 21 runs, bound 10ms/11ms/21ms after Listen started), and neither six busy loops nor eight busy loops with the test at nice -n 19 at --cpus 2 (-count=6 of the first five converted tests, on main) produced a "never bound". The 1-in-400 of #890 needs a worse stall than I can produce under the --cpus 2 rule. So there is no red/green number for the flake itself, and the PR does not claim one. What it does show (golang:1.27.3, --cpus 2):

result
mutant: bindLimit890 = 1ns 17 top-level tests fail at once with adaptive engine never bound: no address after 1ns (every converted internal/adaptive test: 791 x1 here, driver_provider x5, 974, prebound x2, switch_idle_conns x3, pinned_threads_905, epoll_halves_e9 x4), and no "did not stop" line: the teardowns no longer add 10 to 15 s
mutants: the epoll / iouring awaitAdaptiveBound844 limit set to 1 ns *844: 3 of 3 fail in each package; TestAdaptiveSwitchIsNotALoopExit848: fails
converted internal/adaptive tests, -race -count=10, 8 MiB memlock 10 of 10 pass each (372 s)
TestAdaptiveSwitchesWithPreBoundListener, -race -count=10 skips on its own standby-build check at 8 MiB, on main too; with --ulimit memlock=-1: 10 of 10 pass
epoll and iouring *844 and *848, -race -count=10 10 of 10 pass each
whole ./internal/adaptive, -race -count=1 ok (168 s)
whole ./internal/engine/epoll, -race -count=1 ok (268 s)
golangci-lint (CI pin v2.14) on the three packages 0 issues

Release note

None (tests only).

…ait (Refs #1104)

791, driver_provider, 974 and prebound waited 3 to 10 s for the address with a
poll that cannot tell a starved poller from a slow Listen; the epoll 844 test
waited 5 s. They use awaitBind890 (30 s guard, stops at once with Listen's own
error when it returns unbound) through awaitBoundOrFail890, a helper for tests
that start Listen themselves; the epoll test does the same inline. The 25 s wait
of listenFloor682, which already follows Listen, stays.
@FumingPower3925 FumingPower3925 added this to the v1.6.0 milestone Oct 28, 2026
@coderabbitai

coderabbitai Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Oct 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…or Listen, as the epoll one does (Refs #1104)

internal/engine/iouring's adaptive_abort_844 is the same test as the epoll one
and polled the adaptive engine for 5 s; adaptive_switch_848 did the same. Both
wait up to 30 s and stop at once with Listen's own error when it returns
unbound (awaitAdaptiveBound844, a copy of the epoll file's: another package's
test cannot import internal/adaptive's awaitBind890). The epoll copy of the
inline loop becomes the same helper.
…s_e9 wait for Listen; 974's cleanup waits on ended (Refs #1104)

Three more tests that drive a real adaptive engine used 5, 10 and 15 s polls
that cannot tell a starved poller from a slow Listen; they use
awaitBoundOrFail890. The teardowns of switch_idle_conns, epoll_halves_e9 and
detect_bound_974 waited on the done channel, which the helper drains on a
failure, so a failing run printed a false 'did not stop' after 10 to 15 s per
test; they wait on ended, which is closed after done is sent on.
@FumingPower3925 FumingPower3925 changed the title tests: the adaptive tests' remaining fixed bind polls wait on Listen tests: the adaptive engine's fixed bind polls wait on Listen (the direct-engine polls are a follow-up) Oct 28, 2026
@FumingPower3925
FumingPower3925 marked this pull request as ready for review October 28, 2026 01:19
@FumingPower3925
FumingPower3925 merged commit 63c2df4 into main Oct 28, 2026
30 checks passed
@FumingPower3925
FumingPower3925 deleted the flakes-e-bind-polls branch October 28, 2026 01:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant