adaptive: the oscillation lock and the cooldown suppress the io_uring error-rate safety revert #1145

Found while reviewing PR #1128 (lane L17). Pre-existing on main (a5fb8f1); not introduced by #1128 and not fixed there. Refs #886.

Summary

controller.evaluate returns at the oscillation-lock check and at the cooldown check before it reaches the io_uring error-rate safety revert. While either holds, a faulting io_uring stays the active engine. The lock is 5 minutes, and it is set precisely by the sequence an erroring io_uring produces (promote, error revert, promote again), so the controller holds the faulty engine for the longest stretches in the case the revert exists for.

Where (main a5fb8f1, internal/adaptive/controller.go)

• :154 if c.state.locked && now.Before(c.state.lockUntil) { return false }
• :162 if ... now.Sub(c.state.lastSwitch) < c.cooldown { return false }
• :182-189 the always-active snap.ErrorRate > c.errorRevertRate revert, reached only after both.
• recordSwitch (:301-328): 3 switches within 5 minutes set locked and lockUntil = now + 5 min.

The comment at :175-182 already argues that the safety revert must not wait behind the standby-build backoff ("a backoff over it would leave an erroring engine serving for as long as standbyBuildBackoffMax"); the same argument holds for the lock and the cooldown, and they were not given the same treatment.

Scenario (default epoll start, 64 connections on 2 workers: 32 conns/worker, over the 24 up-threshold)

1. The controller promotes to io_uring (sustained high load).
2. io_uring faults; the error-rate revert fires (cooldown permitting) and the controller returns to epoll.
3. After the 30 s cooldown evaluateUp promotes again (cpw is still 32). That is the third switch within 5 minutes, so recordSwitch locks switches for 5 minutes with io_uring active.
4. Until the lock expires the safety revert cannot run.

Simulation (reviewer's scenario test, TestReviewLockHoldsAFaultyIOUring, mocks only)

Production cooldown (30 s), 1 s ticks, loadDownRevert = false, 64 connections on 2 workers, an io_uring that fails 50% of its requests whenever it is active, 30 simulated minutes, recordSwitch called on every recommended switch as performSwitch does:

• 15 switches;
• io_uring active 1042 s of 1800 s;
• 892 s of those under the oscillation lock.

Relation to #1128's four test failures

TestReverseTransplant, TestReverseTransplantAsync, TestBidirectionalFlap, TestBidirectionalFlapAsync failed 0/10 at --cpus 2 because 64 conns on 2 workers are 32 conns/worker and the controller promoted back to io_uring (and logged oscillation detected) after each forced revert. This issue is the production form of that mechanism: the same re-promotion at cpw 32, with the lock turning it into a 5-minute stay on the engine that was just reverted. The tests were fixed in #1128 by turning the conns-per-worker policy off in the ForceSwitch-only helper (they test transplants, not the policy); the policy gap is this issue.

Proposed fix direction (for the maintainer to choose)

1. Let the error-rate revert bypass the lock and the cooldown, as it already bypasses the standby-build backoff: move the snap.ErrorRate > errorRevertRate check above the lock and cooldown returns (the Sample call sits after these two returns today, so a revert above them needs the sample taken earlier too; the delta-window comment at :166-170 applies, and the first sample after a long lock spans the whole lock). The revert then also becomes a switch that counts toward the lock; decide whether a revert should extend the lock (probably not: the lock exists to stop a flap, and a faulting engine is not a flap).
2. And/or, after an error revert, hold evaluateUp off for longer than the cooldown (a back-off on re-promotion to an engine that just faulted, for example doubling per consecutive error revert, reset after a quiet period), so the controller does not return to the faulty engine every 30 s.
3. Tests: the scenario as a regression test (assert io_uring is active for far less than 1042 s and never under the lock), plus a unit test that the revert fires with locked set.

Interacts with the 3-fault floor of #1128 (886 item 29): the floor makes the revert blind to low traffic that fails entirely (error_rate_floor_lowtraffic_886_linux_test.go pins it); both concern whether the safety revert can be relied on. Worth deciding together.

Risk

Behaviour change in the controller's safety path (switching). LOW code size; needs the maintainer's decision on 1 vs 2.