[FumingPower3925]

Lane L17 ADAPTIVE of v1.6.0, the bookkeeping and fault-count logic of the adaptive engine. Refs #886, Refs #890.

Method: every item was checked on main (a5fb8f1) first; all eight were still open. Item ordinals count every checkbox of the issue body (ticked or not), and each row quotes the item's opening words because #1099 counts only unchecked boxes and gives other numbers. Commit 1 is a test fix, commit 2 the failing-on-main tests, commit 3 the behaviour change, commits 4 to 7 test hygiene and a comment.

item (opening words)	premise	verdict	commit
886.29 "(user decision, behaviour change) the io_uring error-rate revert has no floor on faults or on the divisor"	TRUE (3 tests red on main)	fixed: decision A of 2026-10-25, a minimum of 3 faults per interval; max(requests, connection-seconds) kept	2, 3
886.30 "adaptive on an arm64 big.LITTLE host now sees 8 workers, not GOMAXPROCS" part (1), ioUringViable	TRUE (3 tests red on main)	fixed: compares the memlock ceiling with the length of the platform CPU plan	2, 3
886.30 part (2), the conns-per-worker thresholds	TRUE as stated (cpw = connections / 8 on msr1)	NOT changed: calibration needs cluster data, see Findings	4 (arithmetic pinned by a test)
886.26 "adaptive Metrics leaf lock slotsMu ... no test catches dropping slotsMu from the PRIMARY slot-publish block"	TRUE (the mutant survives main's tests)	test added, both blocks	4
886.27 "adaptive tests (PR #1027 review r2, minor/nits)"	TRUE for nits 1 to 6; nit 7 is an observation	nits 1 to 6 fixed, nit 7 not changed	6
886.31 "adaptive: Metrics.Workers is the sum of both sub-engines' loops"	TRUE for the test half	unequal-sum test added; the Warn de-duplication is platform code, not touched	4
890.98 "s0Bind ... waits 3 s for e.Addr()"	TRUE (3 s poll at flap_conns_per_ring_test.go:264, no duration logged)	fixed in s0Bind and newBoundAdaptiveH	5
886.20 "internal/adaptive/porthold.go (about lines 153-155) keeps its own zoneName copy"	TRUE; decided B on 2026-10-12 (keep, add a comment)	comment only	7

The four failing tests: a deterministic test assumption, not load, not an engine bug

TestReverseTransplant, TestReverseTransplantAsync, TestBidirectionalFlap, TestBidirectionalFlapAsync fail on unmodified main when the process has 2 CPUs. golang:1.27.3, -race, each run alone with -count=10 (pass counts):

config	main, each test alone	this PR, each test alone
--cpus 2	0/10, 0/10, 0/10, 0/10	10/10 x 4
--cpus 4	10/10 x 4	10/10 x 4
--cpus 2 -e GOMAXPROCS=4	10/10 x 4	not run
whole ./internal/adaptive package, -race, -count=1	--cpus 2: exactly these 4 fail, 175 others pass; --cpus 4: ok	--cpus 2: ok (195 PASS lines, 0 FAIL, 0 SKIP)

The third row separates load from worker count: same CPU quota as the failing run, GOMAXPROCS 4, and all 40 runs pass. The cause is the worker count. The tests drive every switch with ForceSwitch and put 64 keep-alive conns on the engine, but the controller's own tick loop runs too (cooldown = 0 in the helper). With 2 workers (GOMAXPROCS follows the cgroup quota) 64 conns are 32 conns/worker, over the 24 upThreshold; two ticks later the controller promoted the conns back to io_uring (the log shows engine switch recommended from=epoll to=io_uring reason="sustained high load" conns_per_worker=32 and oscillation detected), so the tests read the wrong engine. At 4 workers it is 16. It is a hard function of the worker count, so it fails every time at 2 and never at 3 or more; a hosted runner has 4 CPUs, which is why CI is green. Not an engine bug: the controller did what its policy says for 32 conns/worker; production has a 30 s cooldown.

Fix (commit 1, test only): newBoundAdaptiveH sets ctrl.connSwitchEnabled = false before Listen. It is not FreezeSwitching() (the s0Bind idiom) because a freeze also suppresses the error-rate revert, and connSwitchEnabled = false keeps it; every caller of the helper (adopt_carry_async_543, idle_follow_switch_standby_804, error_accounting_live, the two scenarios) switches by forceSwitchTo only.

886.29: the floor

faultRate returns rate 0 when the interval has fewer than minRevertFaults (3) faults. The divisor is still returned, ErrorDelta still holds the bucket growth, ErrorRate stays the plain ratio at 3 faults and up. The value 3 is the maintainer's ("for example 3", DECISIONS-RECOMMENDED), not a measurement. Per interval and not cumulative (decision text). max(requests, connection-seconds) is kept as is; a pure connection-seconds divisor or a minimum divisor were not chosen (option A only).

Failing on main (commit 2 tree 8374462, --cpus 2): TestFaultRateFloor886 5 subtests (one fault idle, one fault in one and in nineteen requests, two faults in four requests, two faults against 200 connection-seconds), TestOneStrayAcceptErrorDoesNotRevertAnIdleIOUring886 (one and two accept errors revert), TestFaultsBelowTheFloorDoNotAddUpAcrossIntervals886 (one sampler and one controller over four consecutive intervals: 2, 2, 1 faults do not revert, then 3 do). At commit 3 all pass. Existing TestPeerGoneIsSubtractedNotTheWholeErrorCount856 expected 0.01 for one real fault and now expects 0 (updated in commit 3, with a new case at the floor).

Mutant (floor arm deleted): TestFaultRateFloor886, TestFaultsBelowTheFloorDoNotAddUpAcrossIntervals886, TestOneStrayAcceptErrorDoesNotRevertAnIdleIOUring886 and TestPeerGoneIsSubtractedNotTheWholeErrorCount856 go red.

886.30 part 1: ioUringViable

wantWorkers was cfg.Resources.Resolve().Workers; it is now startedLoops(...), the length of platform.PlanWorkerCPUsFor(requested, explicit, ""), the plan the sub-engines start from (iouring.planWithinMemlock caps the plan, not the request). On a big.LITTLE host with 12 requested and 8 planned, a ceiling of 8 to 11 no longer keeps the engine off io_uring. A host of alike CPUs is unchanged (plan length = request). The plan is made without the listen address, so the rx-port rule, which can only lower the count, is not applied: the count is an upper bound, never lower than what the engines start.

Commit 2 carries the seam planLoopCPUs (a var over the platform planner, like maxWorkersForMemlock) with nothing reading it yet, so the tests can stand in a big.LITTLE host and compile; they fail on it (TestIOUringViableCountsTheLoopsTheEnginesStart886, ...PlansAUserSetWorkerCountAsExplicit886, TestChooseStartEngineUsesThePlannedLoops886: the WorkloadHint=HighConcurrency start stays on epoll). Commit 3 reads it. Mutant (Resolve().Workers restored): the same three go red.

886.30 part 2: not changed, and what data is missing

The thresholds (upThreshold 24, downThreshold 12, highWatermark 48) are already fields of the controller, so they are parameterised; I did not add a scale factor because no data says which way it should go. What the existing evidence (probatorium cluster-reads-20261025/p2-read/tables.tsv, msr1, h1 async, 8 loops after #1049, 12 before) brackets:

cell	conns	cpw at 8 loops	median rps (this tree / v1.5.11)
ad128 (adaptive)	128	16	485864 / 466081
io256 (io_uring)	256	32	521894 / 489919
ep512 (epoll)	512	64	437743 / 466935
io512 (io_uring)	512	64	506668 / 521699

Only one point has both engines at the same load: at 512 conns io_uring is ahead on both trees (+15.7% at 8 loops, +11.7% at 12 loops). There is no epoll cell below 512 and no io_uring cell at 128, and the adaptive cell at 128 sits at cpw 16 (10.7 at 12 loops), under the 24 threshold either way, so it never switched. The crossover the 24 comes from is not located on this host. Missing run: an msr1 sweep of epoll and io_uring, 8 pinned loops, at about 96, 128, 192, 256 and 384 conns (cpw 12 to 48), several repetitions, to see whether the crossover sits at a conns-per-loop or at a connection count. That is the P2/F5a cluster run; the cluster was not touched. Until then: on a host with 8 loops the thresholds act at 2/3 of the connection count they have at 12.

TestControllerDividesByTheActiveEnginesOwnLoops886 pins the arithmetic (192 conns on 8 loops is 24.0, divided by the active engine's loops, not the sum).

886.26, 886.31, 890.98, 886.27, 886.20

These pass on main by nature (a test that adds coverage, a helper, comments), so they cannot be failing-first; the evidence is a mutant that turns the named test red.

• 886.26: TestPerformSwitchPublishesTheBuiltStandbyUnderSlotsMu886 holds slotsMu itself, runs a switch that builds and publishes the lazy standby, and requires performSwitch to park on a mutex (stack read) before it commits; a publish without the lock commits instead. It needs neither -race nor an overlap. Both blocks: the io_uring standby of an epoll start and the epoll standby of an io_uring start (the one no test caught). Mutants: lock dropped from the primary block fails the primary subtest only; dropped from the secondary block fails the secondary subtest only.
• 886.31: TestMetricsWorkersSumsUnequalSubEngines886 (8 + 10 = 18, lazy shape adds none). The other half (the NIC-interrupt Warn is logged by the active plan and again by the lazy standby's plan, with nothing de-duplicating it) is in internal/platform/the engines, outside this lane's files, behaviour-neutral: not touched.
• 890.98: awaitBind890 replaces the 3 s poll in s0Bind and newBoundAdaptiveH: it waits until the address is published, Listen returns (its own error is reported) or 30 s, and logs how long the bind took. It does not tell whether the 1-in-400 miss was a slow Listen or a starved poll; the next occurrence will say. TestAwaitBindOutwaitsTheOldThreeSeconds890 and four more drive it with a fake address source; mutant (limit back to 3 s) fails the first. newRampAdaptive (ramp_transplant_test.go, 5 callers) had the same 3 s poll with the same message and goes through the helper too (review round 1). Left as they are: async_abort_791_linux_test.go:161 and driver_provider_test.go:57 (3 s bind polls; the item names the helpers), and the longer ones, detect_bound_974 (10 s), prebound_listener_test.go:57 (10 s), iouring_floor (25 s) and, in another package, internal/engine/iouring/adaptive_abort_844_linux_test.go:266 (5 s). (driver_provider_test.go:327 is a ForceSwitch loop, not a bind poll.)
• 886.27 (root package and internal/adaptive): the assertSettledRetimed592 doc describes the stack read, not the replaced latency bar; both /ping windows probe through one pingWatched (goroutine and timer per probe, the reference window passes no stall hook); the early-return test doc lists its three ways; fullStack589 grows its buffer until the dump fits; parkedOnMutex589 reads RWMutex frames and [sync.Mutex.Lock / [sync.RWMutex.*Lock wait reasons as well as a sync.(*Mutex).Lock frame; listenBindDeadline is an atomic. Mutants: classifier restored fails TestClassifyLoopStackReadsEveryMutexWait589; fixed 1 MiB buffer restored fails TestFullStackIsNotCutAtItsFirstBuffer589 (20000 parked goroutines, over 1 MiB, run in 0.27 s under -race). Nit 7 ("most fixed-tree arms record 0 captures") is an observation about the observable (it bites only when a probe goes unanswered; the rig logs captures=0 on the fixed tree) and is not changed. The rig itself, TestAdaptiveSettledRouteRetime592 as the CI step runs it (--cpus 2, -race, memlock unlimited, CELERIS_REQUIRE_IOURING_WORKERS=1): 2 of 2 runs pass, all six arms, no SKIP.
• 886.20: the comment at the bindHold call and on zoneName says bindiag lives at internal/engine/internal/bindiag and cannot be imported from internal/adaptive, that the zone rule is the same, and that a change has to be made in both. engines: OnDetach rollback first, io_uring pause wake, zoned bind address, one-dump NIC table, MaxEvents by allowed nodes, upgrade-carry count, backlog hook (v1.6.0, Refs #885 #886 #889 #890) #1127 adds bindiag.ZoneID only; the zone-name rule is untouched there.

Decision owed: the floor at low traffic

Found by the Opus review of this PR; the rule is unchanged here and is for the maintainer to confirm. The 3-fault floor matches the decision's words (U-14, "a minimum fault count (for example >= 3) per interval"), but it has a consequence the decision text did not weigh: it is a count per interval, so low traffic that fails in full never reaches it. Measured with a deterministic mock test over 600 one-second intervals on the opt-in io_uring start (connSwitchEnabled false, where the revert is the only switch the controller can make):

load (every interval)	rule before this PR	this PR
1 rps, 100% failing, 1 conn	reverts at interval 1	never in 600 s
2 rps, 100% failing, 2 conns	reverts at interval 1	never
4 rps, 50% failing	reverts	never
10 rps, 20% failing	reverts	never
33 rps, 6% failing	reverts	never
10 rps, 30% failing (3 faults/s)	reverts	reverts at interval 1

In general: any fault ratio p over 0.05 at fewer than 3/p requests a second (at most 60 rps) never reverts. A low-traffic server whose io_uring fails every response keeps failing for good; before the floor it went back to epoll within a second.

Interval length: the floor is a count, not a rate, so the same steady load behaves differently by interval length. 2 faults/s at 2 rps never reverts on 1 s intervals, but reverts on the first sample after a switch (one 30 s interval: 60 faults over 60 requests).

The alternative is a windowed floor (3 faults accumulated over the last N samples, N about 10, reset on a switch, with the rate taken over the same window; or a floor scaled by elapsed time). It keeps the stray-accept-error case (TestOneStrayAcceptErrorDoesNotRevertAnIdleIOUring886, TestFaultRateFloor886) and closes the hole, and it inverts TestFaultsBelowTheFloorDoNotAddUpAcrossIntervals886 (2 + 2 faults would then revert).

TestPinFloorLetsALowTrafficFailingServerStayOnIOUring886 and TestPinFloorIsACountSoALongIntervalReverts886 pin today's behaviour with the table above (mocks only, 0.00 s); they say in their comment that they document this consequence and must be flipped if a windowed floor is chosen.

Review round 1 (Opus, APPROVE-WITH-FIXES)

New commits on top, plus a merge of main (789d88d), no rebase:

• pin tests for the floor at low traffic (above);
• minors and nits: the startedLoops direction corrected; the leftover bind-poll list corrected and newRampAdaptive converted to listenAndAwaitBind890; the 886.12/886.28 follow-ups recorded; the upper bound of TestFullStackIsNotCutAtItsFirstBuffer589 is now read on both sides of the dump (before and after: max(before, after) + 50); TestControllerDividesByTheActiveEnginesOwnLoops886 is driven through ctrl.evaluate so the engine the controller hands to the sampler is what is checked; two comments reworded (the floor removes the cases of one and two faults; the constant pin test says it pins a constant); startedLoops notes the thread-affinity point.
• Not taken: nit n5 (the red-first split stays, as the reviewer accepted it), n6 (the porthold comment is true as written), listenGapHook being a plain var (not in the item).
• Not fixed, written up: the oscillation lock vs the safety revert (see Findings).

Local results

After review round 1 and the merge of main (789d88d), --cpus 2: the new and changed adaptive tests -race -count=3 ok (66 PASS lines, 0 FAIL); the root classifier, bar and full-stack tests -race -count=3 ok (15 PASS); whole ./internal/adaptive -race -count=1 ok in 333 s (197 PASS, 0 FAIL, 0 SKIP); go vet (GOOS=linux) clean; golangci-lint (GOOS=linux) no finding in a touched file. The figures below are those of the first head.

golang:1.27.3, --cpus 2 unless stated, seccomp unconfined, memlock unlimited. Whole ./internal/adaptive at HEAD, -race -count=1: ok in 332 s (195 PASS, 0 FAIL, 0 SKIP). New root tests under -race: pass. Every commit go vets clean for linux (checked one by one). golangci-lint (GOOS=linux): no finding in a touched file; 3 errcheck findings in untouched files (async_abort_791_linux_test.go:288, detect_bound_974_linux_test.go:163, driver_provider_test.go:63), present on main. No exported API changed, so mage apiCheck is not needed. No new test skips, so the CI race-step allow-list is unchanged; no workflow file is touched.

Release notes

• adaptive: the safety revert from io_uring to epoll needs at least 3 faults in one sampling interval (about one second); a single stray accept error no longer reverts a server started on io_uring (where the revert is one-way). The other side of the same rule: a server whose traffic is so low that it never has 3 faults in a second never reverts, even when every request fails (for example under about 3 requests a second with every one failing, or 10 requests a second with 20% failing). Before this change it fell back to epoll within a second. On the one 30 s interval that follows a switch the same traffic does revert, because the floor is a count.
• adaptive: on a big.LITTLE host, io_uring is judged viable against the number of loops the engines start rather than the requested count, so a low RLIMIT_MEMLOCK ceiling between the two no longer keeps the engine off io_uring.

Findings, not fixed here

• 886.30 part 2: needs the msr1 conns sweep described above (cluster, not touched).
• 886.31: the NIC-interrupt Warn is not de-duplicated between the active plan and the lazy standby's plan (platform/engine code, behaviour-neutral); the two plans may differ if a vector moved in between.
• Max-divisor dilution is kept by decision: N idle connections still make a real rate N times less likely to cross 0.05. 886.12 (confirm max()) was waiting for this decision: U-14 approved 886.29 "as recommended", and option A reads "+ confirm max()", so the confirmation is on record.
• 886.28 (docs repo, engines.md) is unblocked. Sentence to add to its rewording: "An interval with fewer than 3 faults reads an error rate of 0, so the io_uring-to-epoll safety revert needs at least 3 faults in one sampling interval; a server below about 3 faults per second never reverts, even when every request fails (see the decision owed in adaptive: error-rate revert floor of 3 faults, ioUringViable counts the started loops, slotsMu/Workers/bind-wait tests, the four 2-CPU test failures (v1.6.0) #1128)."
• Oscillation lock and cooldown suppress the io_uring error-rate safety revert (controller.go:154, :162 return before :184): pre-existing on main, 30-minute simulation at production cooldown with io_uring failing 50% whenever active: 15 switches, io_uring active 1042 of 1800 s, 892 s of them under the lock. It is the production form of the four 2-CPU test failures. Not fixed here; issue draft (not filed): /Users/fuming/.claude/projects/-Users-fuming-Documents-github-celeris-probatorium/evidence/lanes-20261025/L17ADAPTIVE/m2-issue-draft.md.
• The remaining 3 s bind polls (listed under 890.98).
• Three errcheck findings in untouched adaptive tests (listed above).

For the reviewer to attack

• The floor is per interval, and intervals differ in length: see "Decision owed" below. The reviewer measured the low-traffic hole; the sentence in round 0 of this body that a steady 2 faults/s "never reverts (decision text)" was my reading, not the decision's words.
• startedLoops skips the rx-port rule, so it is an upper bound on the loops the engines start. An over-count makes ioUringViable say "not viable" in the band between the loops left after the rx-port rule and the planned loops: it narrows viability, which is the safe direction (never "viable" for fewer loops than they start). It also plans on the affinity of the thread that calls New, the sub-engines on the one that calls Listen; they differ only if an application narrows one thread's mask (noted in a comment).
• TestPerformSwitchPublishesTheBuiltStandbyUnderSlotsMu886 reads a goroutine stack for a sync.Mutex park inside performSwitch; it relies on the test holding only slotsMu. If performSwitch ever takes another uncontended-but-held lock before the publish, the park check becomes ambiguous.
• connSwitchEnabled = false in newBoundAdaptiveH: any future caller that wants the controller's load policy must not use this helper.
• planLoopCPUs in commit 2 is dead code on purpose; confirm the red-first split is worth it.
• The 3 is a documented decision, not a measurement.

[coderabbitai]

Caution

Review failed

The pull request is closed.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!