Repository navigation
tests: the adaptive engine's fixed bind polls wait on Listen (the direct-engine polls are a follow-up) - #1149
Merged
Merged
Conversation
…ait (Refs #1104) 791, driver_provider, 974 and prebound waited 3 to 10 s for the address with a poll that cannot tell a starved poller from a slow Listen; the epoll 844 test waited 5 s. They use awaitBind890 (30 s guard, stops at once with Listen's own error when it returns unbound) through awaitBoundOrFail890, a helper for tests that start Listen themselves; the epoll test does the same inline. The 25 s wait of listenFloor682, which already follows Listen, stays.
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…or Listen, as the epoll one does (Refs #1104) internal/engine/iouring's adaptive_abort_844 is the same test as the epoll one and polled the adaptive engine for 5 s; adaptive_switch_848 did the same. Both wait up to 30 s and stop at once with Listen's own error when it returns unbound (awaitAdaptiveBound844, a copy of the epoll file's: another package's test cannot import internal/adaptive's awaitBind890). The epoll copy of the inline loop becomes the same helper.
…s_e9 wait for Listen; 974's cleanup waits on ended (Refs #1104) Three more tests that drive a real adaptive engine used 5, 10 and 15 s polls that cannot tell a starved poller from a slow Listen; they use awaitBoundOrFail890. The teardowns of switch_idle_conns, epoll_halves_e9 and detect_bound_974 waited on the done channel, which the helper drains on a failure, so a failing run printed a false 'did not stop' after 10 to 15 s per test; they wait on ended, which is closed after done is sent on.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Flakes lane, package E: fixed 3 to 15 s waits for a REAL adaptive engine to publish its address become the Listen-aware wait of #1128, in 10 test files of
internal/adaptive,internal/engine/epollandinternal/engine/iouring. A test-hygiene change, no engine change. It does NOT convert every fixed bind poll in the repository (the rest is listed below as a follow-up). Not reproduced as a failure (see Evidence). Refs #1104What changes
#1128 added
awaitBind890/listenAndAwaitBind890ininternal/adaptivebecause a 3 s poll was missed once in 400 runs at--cpus 2with two busy loops (celeris#890 item 98): a poll that may have been starved, or a Listen that was slow, with no way to tell which. The new wait polls every 10 ms for up to 30 s (bindLimit890, a guard against a hang), stops at once when Listen returns before binding and fails with Listen's own error, and logs how long the bind took. Listen gives up by itself after 20 s (listenBindDeadline), so the 30 s is never the bound that matters.Converted (a real adaptive engine, polled with a fixed limit that did not follow Listen)
internal/adaptive/async_abort_791_linux_test.go(abortAdaptive791; also called by the child-process arm ofidle_follow_switch_standby_804)internal/adaptive/driver_provider_test.go(newBoundAdaptive, 5 tests)internal/adaptive/detect_bound_974_linux_test.gointernal/adaptive/prebound_listener_test.go(preBoundAdaptive, 2 tests:TestAdaptiveStartsWithPreBoundListener,TestAdaptiveSwitchesWithPreBoundListener;TestAdaptiveRejectsNonTCPListeneronly callsNew)internal/adaptive/switch_idle_conns_linux_test.go(3 tests)internal/adaptive/pinned_threads_905_linux_test.gointernal/adaptive/epoll_halves_e9_linux_test.go(4 tests)internal/engine/epoll/adaptive_abort_844_linux_test.go(3 tests)internal/engine/iouring/adaptive_abort_844_linux_test.go(the twin of the epoll test, 3 tests)internal/engine/iouring/adaptive_switch_848_linux_test.goThe
internal/adaptiveones useawaitBoundOrFail890(new, next to the others):listenAndAwaitBind890for a test that starts Listen itself and keeps its cancel and done channel to assert on the stop. The Listen goroutine of those tests closes anendedchannel after it sends ondone. The teardowns ofswitch_idle_conns,epoll_halves_e9anddetect_bound_974wait onendednow, not ondone: on a failure the helper drainsdoneto report Listen's error, and a teardown waiting on it printed a false "did not stop" after 10 to 15 s per test. The two engine packages cannot importinternal/adaptive's test helper (their tests are in other packages), so each has a small copy,awaitAdaptiveBound844(30 s, returns at once with Listen's error and puts the value back ondonefor the teardown).NOT converted (stated, not claimed)
internal/adaptive/shutdown_during_bind_wait_linux_test.go:107(5 s): a MOCK primary, and its Listen runs withcontext.Background()(no cancel for the helper). Low risk.internal/adaptive/h2_dial_race_test.go:113andpauseaccept_h2_dial_race_test.go:73: 60 s polls, not Listen-aware (a failing Listen costs 60 s before "did not bind"); they do not have the 3 to 10 s weakness.internal/adaptive/metrics_listen_race_linux_test.go:66(15 s),port_hold_616_linux_test.go:176(25 s),iouring_floor_linux_test.go:184(25 s,listenFloor682, which turns an early Listen return into a skip or failure with the engine's error),test/integration/adaptive_hybrid_test.go:270(20 s).driver_provider_test.go:327(a 3 s loop ofForceSwitchcalls) andprebound_listener_test.go:179(the 30 s wait for a switch).Addr() == nilloops) ininternal/engine/epoll(for exampleadopt_suspended_loop:117 10 s,async_handler_stall:727,async_largebody:85/151,backpressure:101,closing_e2e_876:54,closing_h2_restamp_876:123/308,closing_slow_reader_876:57,dispatch_bench_771:52,drainread_flood_885:85 5 s,driver_read_helpers_930:164,hijack_flush_936:102,hijack_inline_async_loop:107,hijack_offthread:332,keepalive_writeprogress_885:99,linger_arrivals_placement:148,nic_irq_rxfree_1036:110,pause_accept_idle:175,pause_accept_linger:858,pause_accept_queued:129,poolwait_gate_unread_885:408,recv_shapes_bench_884:47,sendfile_e2e:103 3 s,shutdown_counters_e2e_874:56,transplant_accounting:189 3 s,transplant_handoff_failure:274,driver_test:58-68), the same ininternal/engine/iouringandinternal/engine/std(std/h2c_goaway_878:130 has no deadline at all). They are a mechanical sweep withinternal/engine/epoll's existingawaitBound.test/conformance/helper_test.go:95,test/spec/helpers_test.go:118,test/integration/driver_integration_test.go:363,drivers_shared_loop_test.go:73,per_handler_async_test.go:69,test/drivercmp/{redis,postgres}/integrated*_test.go,epoll/async_h2_notice_engcfg_linux_test.go:65,resume_wake_886:111,iouring/pause_wake_886:46, and the Server-levelwaitServerStarted/waitForReadycallers.internal/engine/iouring/errorclass_linux_test.go:180(5 s) turns a bind timeout intot.Skip("engine never bound"), which can hide a real failure: it deserves a look of its own.Evidence
I could not make the old waits fail. The bind takes 10 to 21 ms here (the helper logs it: 21 runs,
bound 10ms/11ms/21ms after Listen started), and neither six busy loops nor eight busy loops with the test atnice -n 19at--cpus 2(-count=6of the first five converted tests, on main) produced a "never bound". The 1-in-400 of #890 needs a worse stall than I can produce under the--cpus 2rule. So there is no red/green number for the flake itself, and the PR does not claim one. What it does show (golang:1.27.3,--cpus 2):bindLimit890 = 1nsadaptive engine never bound: no address after 1ns(every convertedinternal/adaptivetest: 791 x1 here, driver_provider x5, 974, prebound x2, switch_idle_conns x3, pinned_threads_905, epoll_halves_e9 x4), and no "did not stop" line: the teardowns no longer add 10 to 15 sawaitAdaptiveBound844limit set to 1 ns*844: 3 of 3 fail in each package;TestAdaptiveSwitchIsNotALoopExit848: failsinternal/adaptivetests,-race -count=10, 8 MiB memlockTestAdaptiveSwitchesWithPreBoundListener,-race -count=10--ulimit memlock=-1: 10 of 10 pass*844and*848,-race -count=10./internal/adaptive,-race -count=1./internal/engine/epoll,-race -count=1golangci-lint(CI pin v2.14) on the three packagesRelease note
None (tests only).