Repository navigation
driver/redis: a cluster or sentinel PubSub follows its subscription to another node (v1.6.0, Refs #887) - #1154
Merged
Merged
Conversation
… node and a moved slot (v1.6.0, Refs #887) Tests only; they fail on main. A fake cluster (rpCluster) hosts PubSub conns: node loss, a slot move (the old owner sends sunsubscribe and keeps the conn), a stale CLUSTER SLOTS answered with -MOVED, two subscriptions with one lost, flapping inside the backoff, Close racing a reroute, a message delivered on both conns during the switch. A real-cluster cell (redis_cluster tag) moves an empty slot.
…er node (v1.6.0, Refs #887) Option A of item 31: the client re-routes inside, nothing public changes. A pubsubRoute (unexported) tells the reconnect loop which client to dial. Triggers: the conn is lost; an unsolicited sunsubscribe of a channel still in the tracked set; -MOVED or -CLUSTERDOWN answered to a control write. The old conn is cut off before the new one is bound, a trigger that arrives during the handover is kept, the dial is cut short by Close, and a closed ClusterClient or SentinelClient ends the loop and closes Channel().
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
) F1 a guard for the closed check before the install that a PubSub with no channel exercises; F2 a caller's cancelled context does not mark a node failing, and a PubSub dial clears only the mark it set; F3 a -MOVED answering a SUnsubscribe of a channel the PubSub never had does not rebuild a healthy subscription; F4 a second slot on one PubSub does not kill the first; F7 a dead owner is dialled once by the constructor. The fake answers SUNSUBSCRIBE of a foreign slot with -MOVED and a multi-slot SSUBSCRIBE with -CROSSSLOT, as Redis does. Real-cluster cells for F3 and F4. F2, F2b, F3, F4 and F7 fail on the previous head.
F2 a caller's cancelled context no longer marks a node failing, and a PubSub dial clears only the failing mark it set. F3/F4 a cluster PubSub serves one slot, the slot of its first shard channel: a -MOVED for another slot is ignored (it answers a SUnsubscribe of a channel the PubSub never had, or a second channel that is not served) and the replay sends only the channels of that slot, so a second slot cannot take the first down with -CROSSSLOT. F6 the router reports carry their conn: a report from a conn the loop has replaced is ignored, and the trigger is raised under mu so the install's clear of pendingReroute cannot be undone. F7 the constructor dials each node at most once. F5 docs: Channel(), onConnDrop, the delivery claims. F1 and F6 get tests.
… finds no node (v1.6.0, Refs #887) TestClusterPubSubCloseLeavesNoConnBehind887 failed once on the Windows portable job with 'no cluster nodes available' (not reproduced on Linux x25 at 1 cpu, nor on darwin x60). Test-only diagnostics so a second occurrence names the cause.
This was referenced Oct 28, 2026
…e the reader holds it (v1.6.0, Refs #887) A stress test, linux and darwin only, meant for -race: on the current code it reports the data race between handshake's releaseResult and dispatch, and the healthy client's PING gets an empty reply with a nil error.
…ly was dispatched (v1.6.0, Refs #887) When a new conn's HELLO failed because the caller's context ended (or the conn closed), the request went back to the pool while the reader goroutine could still be filling it in; another conn could reuse it and be woken by the late finish with a reply that is not its own: a Do on a healthy conn returned an empty reply with a nil error, and -race reported a data race with dispatch. Only a *RedisError (the reply was dispatched) is released now, as every other caller of exec does; nothing leaks, req.owned is never set in production code. Pre-existing since 5cb0f9b.
) N2 a sunsubscribe of a slot the PubSub does not serve (the same node owns it) does not rebuild the served subscription; N6 the served slot is the first channel's, not the smallest, in the memory and in the reroute's pick (two mutants survived round 1); N5 a successful PubSub dial does not clear a failing mark the command path set after the PubSub's own; the close test checks the node table right after NewClusterClient. N2 and N5 fail on the previous head. A real-cluster cell for N2.
N2 a sunsubscribe acts only for the served slot, so the move of a second slot that the same node accepted no longer rebuilds the served subscription. N5 failing is a failFlag: Store (the command path, the topology refresh) takes the node over, and a PubSub dial clears only a mark that a failed PubSub dial was the last to set. N3/N4 the SSubscribe godoc says what happens to a channel of another slot: not subscribed if its node answers -MOVED; accepted and delivered until the next reroute if the same node owns it; and a remaining channel of another node's slot is never subscribed if the first channel is removed.
…fore it moves the slot (v1.6.0, Refs #887) TestPubSubSunsubscribeOfAWantedChannelResubscribes887 failed once in CI (resp2): the fake counts an SSUBSCRIBE when it receives it, before it records the channel, and the test sent the sunsubscribe in between, so there was nothing to unsubscribe.
) m1 the served slot is re-selected when its channels are gone (a mutant that read the cached slot survived); m2 the failing flag under concurrency (a command's Store(true) racing a PubSub success lost the mark, 1 to 16 rounds in 300000 on the current flag) and its sequential state machine, with two benchmarks; m4 a deterministic handshake test on every OS, through a hook that only handshake's own release calls (not putRequest, so no hot path pays for it); the stress test's rounds are 1000 with a note on the macOS port range. The m2 stress test fails on the previous head.
Two bits in one atomic.Uint32 (failing, set by a failed PubSub dial): Store (the command path, the topology refresh) replaces the word, markPubSub replaces it with the PubSub's verdict, clearPubSub is one compare and swap that clears only the PubSub's own mark. The previous two-field flag could lose a command's mark to a concurrent PubSub success, and its embedded atomic.Bool exposed Swap and CompareAndSwap, which skip the reset. The command path's four sites are unchanged, with no lock or allocation. The SSubscribe godoc says the remaining channel of another node's slot is subscribed at the next reroute.
…h-master subscription (v1.6.0, Refs #887) One failure in 20 runs of the stress: the test pushed the switch before the SentinelClient's background subscription existed, so nothing heard it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
driver/redis: a cluster or sentinel PubSub follows its subscription to another node. Refs #887 (item 31, quoted: "drivers, redis cluster PubSub:
ClusterClient.SubscribeandClusterClient.SSubscribebind their PubSub to the client of one node"). Maintainer decision D3, option A: the client re-routes inside, nothing public changes.Commits:
f8fdb86the tests (they fail on the base),1951ee6the change; review round 1:d11531etests (5 fail on1951ee6),a63be3ethe fixes,43a9469test diagnostics; review round 2:32e4f12+840c486(handshake race, its own pair),3f2eb6atests,3b3de72fixes,5978381a test-only wait; review round 3:722a9detests,0e9bbf1the flag,02177f8a test-only wait (the sentinel test). Round 2's5978381was the first test-only wait (the fake counted an SSUBSCRIBE before recording it; one CI failure ofTestPubSubSunsubscribeOfAWantedChannelResubscribes887/resp2).api/is untouched,mage apiChecksays "api/ matches the exported API of 53 packages".The problem
The PubSub of
ClusterClient.Subscribe,ClusterClient.SSubscribeandSentinelClient.Subscribeis bound to the*Clientof one node. Its reconnect loop dials that client's pool, that is, that one address, for ever.sunsubscribe <ch> 0and keeps the conn.routePushtook it for the ack of the PubSub's ownSUnsubscribeand dropped it, whileshardSubsstill lists the channel: the subscription is silently dead.SSUBSCRIBEsent to a node that does not own the slot is answered-MOVED, which pubsub mode dropped (empty bridge): silently dead as well.Clientis closed and the user's PubSub retries that closed client for ever.Design
Full note:
evidence/lanes-20261025/L22REDISPUBSUB/design.md(same text as below).Mechanism. One unexported hook on
PubSub:route pubsubRoute(nil for a standaloneClient: it dials its own client for ever, as before; the only differences there are thatPubSub.Closenow cuts its reconnect dial short and its backoff wait is closeCh-first). Every attempt ofreconnectLoopcallsroute.pick(ctx, shard, cur, moveOn)before it takes any lock, then dialsacquirePubSubon the returned client. The PubSub remembers which client ownsps.conn(connClient, undermu), so a conn is released to the pool it came from (async.Pool.Discardon a foreign pool would corrupt its counts).SUBSCRIBE/PSUBSCRIBE(node-local; the cluster bus delivers every PUBLISH to every node): stay on the node while it works; after a failed dial or replay move to the next node by address, skipping the ones markedfailing.DialTimeout)refreshTopology, then the owner of the served slot (reached by the smallest channel of that slot); no owner in the table: any node (as theSSubscribeconstructor does), which answers-MOVEDif wrong and the loop asks again.attempt > 0), move on.Subscribe:getClient(), the current master.ClusterClient.Subscribe/SSubscribe) try the other nodes (or the refreshed owner) when the first dial fails, each node at most once (a stale topology that keeps naming a dead owner is not dialled again).States and triggers. UP -> REROUTING (one
reconnectLoop,reconnectingset) -> UP.sunsubscribe <ch>(RESP3 push or RESP2 array): ifchis still inshardSubsthe caller did not remove it (a control call updates the set before it writes), so the server dropped it: reroute. Ifchis gone it is the ack of the caller's own call: ignored. Benign false positive:SUnsubscribe(x)thenSSubscribe(x)before the ack arrives costs one extra reroute.-MOVEDfor the slot the PubSub serves, or-CLUSTERDOWN: reroute. A-MOVEDfor another slot answers something else and is ignored (real Redis answers aSUNSUBSCRIBEof a channel it holds no subscription for, in a slot it does not serve, with-MOVED; a second channel of another slot is not served). A-CROSSSLOTor any other error is the caller's mistake and does not loop.One slot per PubSub. A cluster PubSub serves the slot of its first shard channel (
servedSlot, kept while a channel of that slot is tracked). A reroute follows that slot and replays only its channels, so a channel of another slot (documented undefined behaviour) cannot take the first one down with a-CROSSSLOT, as the first head did; a channel of another slot is not served: if another node owns its slot the SSUBSCRIBE is answered-MOVEDand it is not subscribed (as on the base); if the same node owns it, the node accepts it and delivers it until the next reroute of any kind, whose replay leaves it out (verified on real Redis; the base sent both slots in oneSSUBSCRIBEand got-CROSSSLOT). The move of that slot does not reroute the PubSub (a sunsubscribe acts only for the served slot).Reports carry their conn. The router callbacks (
sunsubscribe, error reply) are made with the router of the conn they came from; the PubSub ignores one from a conn that is no longerps.conn, and raises the trigger while holdingmu, as the install does when it clearspendingReroute. A report that was in flight when the loop replaced the conn cannot start a second reroute.T2/T3 run on the event-loop worker: they take
ps.muandreconnectingMuonly, neverctlMu.Locking and Close (the #1141 lessons).
pickand the dial run with no lock held. The install is the existing one, underctlMuandmu, with the closed check first, so a conn is never installed on a closed PubSub; a conn the loop dialled and could not install is released.PubSub.Closecancels the loop's context (ps.ctx): a dial or a topology refresh in flight ends with it.sleep): aselect{closeCh:default:}before the timer select, and a closed re-check after it.reconnecting) is kept (pendingReroute) and the loop's tail starts one more loop (the fix(redis): bind PubSub control writes to their conn and release the dropped conn on reconnect #985 item-19 class: the conn is up, so theconn.closedre-check cannot see it). The flag is cleared at the install, so the second sunsubscribe of the same move (one per channel) does not make a second loop.ErrClosed(the ClusterClient or SentinelClient was closed) ends the loop and closes the PubSub:Channel()closes, control calls returnErrClosed. That is "the reconnect loop gives up", already in thePubSubdoc.Backoff reuse. No new budget. The loop is the existing one:
ps.backoff(50 ms doubling to 5 s),ps.attemptover the whole outage, reset only after a conn stayed uppubsubStableAfter. A node that flaps, or a stale topology that keeps answering-MOVED, is retried at 50 ms, 100 ms, ... 5 s.What happens to the user's calls during a reroute.
controlholdsctlMuacross its write; the install and replay hold it too. A call before the install updates the sets (undermu) and writes to the old conn or getsErrPubSubConnDown; the replay snapshot is taken undermuafter the install, so it contains the call either way (an idempotent duplicateSUBSCRIBEat worst). A call after the install writes to the new conn, after the replay. AnUnsubscribeduring the reroute removes the channel from the set, so the replay omits it.Ordering and delivery. At most one conn is bound to the PubSub's router at a time: the old conn's router is cleared under
muimmediately before the new conn is bound and replayed (before, the old conn was detached after the replay, with a window in which both delivered). So the PubSub itself never delivers one message from two conns; a message published while it switches is lost (a gap), as pub/sub is at-most-once. The gap is bounded by the backoff delay + dial + replay. Order within one conn is kept. One caveat, for a regular clusterSUBSCRIBEonly: the cluster sends a PUBLISH to every node over its bus asynchronously, so a message in flight around a switch can in rare cases reach both the old and the new node and be delivered twice (not reproduced: the fake cluster delivers synchronously, and a real one needs bus latency). A sharded channel has one owner and a sentinel PubSub one master, so neither can.What the caller sees (release note)
Channel()andMessageare unchanged; one channel for the life of the PubSub.SUBSCRIBEa message in flight on the cluster bus around the switch can rarely arrive from both nodes.-MOVED(no error, as before the PubSub followed its slot); if the same node owns it, it is delivered until the next reroute of any kind and then dropped silently. Neither takes the first channel down on a reroute any more, and the move of such a slot does not rebuild the served subscription.Dowhose context ended during a new connection's handshake could hand a stray or empty reply to another request (a bug on main since April, fixed here: see "Handshake request recycled").SUnsubscribeof a channel the PubSub never had, in a slot another node serves, no longer rebuilds the healthy subscription (Redis answers it-MOVED).ErrPubSubConnDown(existing). Nothing new is exported.PubSub.Close).Channel()closes, control calls returnErrClosed. A PubSub whose conn stays up after the close (the standalone event loop is refcounted and shared with another Client, orWithEngine) keeps it untilPubSub.Close. This is the one semantic call in the PR, a one-branch revert (errors.Is(err, ErrClosed)inreconnectLoop).Other behaviour changes, for a ruling
failing(whichanyNodeuses forPublishand the topology refresh) unless the caller's own context ended; a successful PubSub dial clears the mark only if a failed PubSub dial was the last thing to change it (failFlag, one atomic word: the command path's and the topology refresh'sStorereplace it, and the clear is a single compare-and-swap on the PubSub's own mark, so it cannot erase a mark that changed after it looked).CLUSTER SLOTSandapplyTopology: N calls when N PubSubs get a sunsubscribe together (bounded by the backoff).PubSub.Close(before it ran to its 5 s bound).Channel()godoc: no longer says "until the reconnect loop exhausts its retries" (there is no budget); it closes onClose, or when the owning client is closed while the conn is down.SentinelClientdoes not move a PubSub whose old-master conn is still up after a switch-master (no kick, no registry).Tests
Method: commit 1 (the tests) compiles on the base and fails; commit 2 is the change. A fake cluster (
rpCluster: HELLO, CLUSTER SLOTS with a stale-reply knob, (P/S)SUBSCRIBE, sunsubscribe on a slot move,-MOVED, PUBLISH/SPUBLISH fan-out, kill a node) hosts real PubSub conns, no tag, so they also run in the portable jobs.f8fdb86(base + tests),-race, Linux,--cpus 2: 14 FAIL, 3 PASS (the 3 pass on the base by design:TestClusterPubSubCloseCutsAStalledRerouteDial887,TestClusterPubSubCloseRacesAReroute887,TestStandalonePubSubIgnoresSunsubscribe887, guards that the change must not break). Log:logs/red-final2.log.1951ee6: the new tests-race -count=20, Linux,--cpus 2: ok (495 s);go test -race ./driver/redis/...once: ok (71 s); the new tests on darwin natively,-race -count=3: ok.pubsub_reroute_internal_887_test.go: triggers on RESP3 and RESP2, ack is not a trigger, error kinds, a trigger during the handover, the old conn cut off while the loop is held, a Close between dial and install,sleepcloseCh-first) need the route and hooks, so they are in commit 2 and are proven by mutants.logs/mutants-head.log, run on1951ee6): NO-ROUTE, RELEASE-VIA-HOME-CLIENT (Close), RELEASE-OLD-VIA-HOME-CLIENT, OLD-ROUTER-NOT-CLEARED, NO-PENDING-REROUTE, PENDING-NOT-CLEARED-AT-INSTALL, SUNSUB-ALWAYS-TRIGGERS, SUNSUB-NEVER-TRIGGERS, MOVED-NOT-A-TRIGGER, CLUSTERDOWN-NOT-A-TRIGGER, ANY-ERROR-TRIGGERS, SLEEP-RANDOM-TIE, CLOSE-DOES-NOT-CANCEL-DIAL, NO-GIVE-UP-ON-CLOSED-OWNER, FLAPPING-RESETS-BACKOFF, NO-ROTATE-ON-FAILURE, NO-TOPOLOGY-REFRESH-FOR-SHARD, CONSTRUCTOR-NO-FALLBACK, SENTINEL-ROUTE-ABSENT: all red (names of the red tests in the log). INSTALL-ON-CLOSED (removing only the closed check at the install) survives: the closed check before each replay write catches the conn and releases it, so the two are layered; removing both (a two-edit mutant, run by hand) is red inTestClusterRerouteDoesNotInstallAConnOnAClosedPubSub887.test/conformance/redis/pubsub_reroute_cluster_test.go, tagredis_cluster, runs in the "Redis Cluster" job):SSUBSCRIBEon a 6-node redis 7.4.11 cluster in one container, an empty slot moved withCLUSTER SETSLOT IMPORTING/MIGRATING/NODE, moved back at the end; RED on the base ("no message of after-move arrived within 30 s"), 3 of 3 green on the change. Node loss on the same real cluster (a regularSUBSCRIBEholder, then the owner of a shard channel,SHUTDOWN NOSAVE): re-subscribed on another node after 0.2 s, and on the promoted replica after 6.5 s (a throwaway test, not committed; the destructive tag needs per-node containers).-tags redis_cluster): 0 issues.GOOS=windows/darwin/linux go vet ./driver/redis/: clean.The #1140/#1141 sweep (select with a timer and a side effect)
PubSub.reconnectLoop: fixed (sleep, closeCh first; alsotime.Afterleaked a timer per attempt).ClusterClient.refreshLoop: the ticker branch ran the refresh (dials nodes) with no look atclosed; now it returns first. No test (the period is 60 s and a fixed constant).SentinelClient.subscribeLoop: already safe, the loop top checksclosedright after the select.SentinelClient.watchFailover: the tick branch's side effect istakeSentinel/closeSentinel, which only the taker may close (redis: dial waits out the io_uring number refusal, sentinel moves on from a lost sentinel, pubsub backoff, portable unregister, wake source guard (v1.6.0, Refs #887 #890) #1129), so a tie afterCloseis harmless; the message branch goes throughhandleSwitchMaster, which checksclosedfirst.SentinelClient.handleSwitchMaster: already closed-checked after the select and before the install (driver/redis: SentinelClient and ClusterClient no longer leak or panic when closed during a failover or a refresh #1141).Review round 1 (Opus, head 1951ee6: approve with fixes) and what was done
1951ee6(d11531e)TestClusterRerouteDoesNotInstallOnAClosedPubSubWithNoChannels887failingctx.Err()==nilguard inopenPubSubTestClusterSubscribeWithACancelledContextDoesNotMarkANodeFailing887clusterNode.psFailed: clear only what the PubSub path setTestPubSubDialReportClearsOnlyTheMarkItSet887TestClusterForeignSUnsubscribeDoesNotRerouteAHealthySubscription887; real:TestClusterForeignSUnsubscribeKeepsTheSubscriptionConn(conn id fromCLIENT LIST)TestClusterSecondSlotOnOnePubSubDoesNotKillTheFirst887; real:TestClusterSecondSlotOnOnePubSubKeepsTheFirstDeliveringChannel(),onConnDrop,ClusterClient.Subscribe/SSubscribe, this body, design.mdps.conn; trigger raised undermuTestClusterRerouteIgnoresAReportFromTheReplacedConn887(loop held after the install, the old router reports)1951ee6by construction (the fix changes the callbacks' signature, so the test uses the new one); proven by the NO-CONN-IDENTITY-* mutantsTestClusterSSubscribeTriesADeadOwnerOnce887(topology refreshes counted)Counts:
d11531e(previous head + the new tests): 25 pass, 5 fail (F2, F2b, F3, F4, F7),logs/red-r1.log. Heada63be3e: the new and old tests-race -count=20Linux--cpus 2: ok (548 s);TestClusterRerouteAnswersEveryChannelOfAMoveWithOneLoop887and the F6 test-race -count=50: ok (80 s);go test -race ./driver/redis/...: ok (74 s). Real redis 7.4.11 (one container, 6 nodes), the three real cells: red on1951ee6for F3 and F4, 3 of 3 green ona63be3e(logs/real-r1-red.log,real-r1-green.log). Lint v2.14 (also-tags redis_cluster): 0 issues;GOOS=windows/darwin/linux go vet: clean;mage apiCheck: matches 53 packages.Mutants on
a63be3e(28, each alone,logs/mutants-r1.log), all red: RELEASE-VIA-HOME-CLIENT, RELEASE-OLD-VIA-HOME-CLIENT, OLD-ROUTER-NOT-CLEARED, NO-PENDING-REROUTE, SLEEP-RANDOM-TIE, CLOSE-DOES-NOT-CANCEL-DIAL, NO-GIVE-UP-ON-CLOSED-OWNER, FLAPPING-RESETS-BACKOFF, NO-ROTATE-ON-FAILURE, NO-TOPOLOGY-REFRESH-FOR-SHARD, CONSTRUCTOR-NO-FALLBACK, INSTALL-ON-CLOSED (now dies in the F1 test), PENDING-NOT-CLEARED-AT-INSTALL, SENTINEL-ROUTE-ABSENT, NO-ROUTE, SUNSUB-ALWAYS-TRIGGERS, SUNSUB-NEVER-TRIGGERS, MOVED-NOT-A-TRIGGER, MOVED-ANY-SLOT, CLUSTERDOWN-NOT-A-TRIGGER, ANY-ERROR-TRIGGERS, NO-REPLAY-SLOT-FILTER, NO-CONN-IDENTITY-SUNSUB, NO-CONN-IDENTITY-ERROR, DIAL-CTX-MARKS-FAILING, DIALED-CLEARS-COMMAND-MARK, CONSTRUCTOR-REDIALS-SAME-NODE, SERVED-SLOT-NOT-KEPT. (The earlier INSTALL-AND-REPLAY mutant, two edits, is superseded: the single-edit one is red now.)Known limitations (to be filed as issues, not fixed here)
CLUSTER SLOTS+applyTopology: N PubSubs hit by one slot move make N concurrent refreshes against the same probe node, repeated at each backoff step (no coalescing).SSUBSCRIBE; it is not grouped by slot, so a PubSub that holds several slots serves only one of them (and the base's reconnect had the same-CROSSSLOT).ClusterClient.Subscribe/SSubscribe(andClient.Subscribe) against a node that accepts the TCP connection and never answers HELLO block until the caller's context ends:DialTimeoutdoes not bound the PubSub handshake. The reconnect loop bounds its own dial with a 5 s context, the constructors do not.SUBSCRIBEthere still receives, PUBLISH is replicated).-MOVEDwas ignored when it was added; the replay then serves the re-selected slot; same as the base until then). Removing every channel and adding one of a new slot does follow it.applyTopologyaccepts aCLUSTER SLOTSreply with no parsable range and then deletes every node: a corrupt reply empties the client's node table (the Windows symptom above). It should reject such a reply.Handshake request recycled (round 2, N1; a bug on main, fixed here, its own commit pair)
redisConn.handshakegave the HELLO request back to the pool (releaseResult) on every failed exec. When the caller's context ended while the HELLO reply was arriving, the reader goroutine had already popped the request and was still filling it in (dispatch, state.go); anotherDotook the same object from the pool and the late finish woke it with whatever the object held. Proof on main with a plainClient(no PubSub): aDo(PING)on a separate healthy client got an empty reply with a nil error (wrong replies 0/2/4 over 3 runs), and-racereported the data race (handshake->releaseResultagainstdispatch). The code is from5cb0f9b(April). This PR made it reachable in the required macOS-racePortable job (about 1 run in 105: a reroute's topology refresh dials a command conn whilePubSub.Closecancels its context), and it is the only mechanism found for the Windows failure below.Fix (
840c486): release only on a*RedisError(the reply was dispatched), as every other caller ofexecdoes; a context or network error leaves the request to the garbage collector. Nothing leaks:req.ownedis never set in production code (grep '\.owned': only reset to false at state.go:42, state.go:381, pipeline.go:287), soreleaseResult's Reader branch is dead and the change only skips onesync.Poolreuse. Test (32e4f12, linux and darwin only, a stress test meant for-race): 1500 fresh clients whoseDohas a 50-350 us context against a fake server that answers HELLO after 0-300 us, while two goroutines check every PING reply of a healthy client. Red on the old code: darwin-race -count=20: 16 of 20 fail, 10 race reports, the checker gets&{Type:simple Str:[]} <nil>; it does not reproduce on Linux (the Linux standalone loop's sync path; not investigated), so the red proof is the darwin run. Green: darwin-race -count=20x3 and-count=100(0 races, 51 s), Linux-count=20(in the full run below).Release note: a
Dowhose context ended during the connection handshake could hand a stray or empty reply to another request.Windows "no cluster nodes available" (round 1)
TestClusterPubSubCloseLeavesNoConnBehind887failed once on the Windows portable job ofa63be3ewithceleris-redis: no cluster nodes availablefromSSubscribe. That text comes only from an empty node table (applyTopologydeletes every node a reply does not list), not from the F7 change. The only mechanism found is the handshake race above (a HELLO reply delivered as a CLUSTER SLOTS reply); it was not reproduced, so "found", not "proven". The Windows job passed on1951ee6and43a9469. After the fix the test checks the node table right afterNewClusterClientand prints it with the number of CLUSTER SLOTS replies, so a recurrence says where it began. Darwin-race -count=100of that test after the fix: 0 races (517 s).Review round 2 (Opus, head 43a9469: request changes, one blocker) and what was done
*RedisError(840c486)TestHandshakeWhoseContextEndedDoesNotRecycleItsRequestWhileTheReaderHoldsIt887(32e4f12)-race -count=20: 16 fail, 10 race reports; not reproducible on LinuxTestClusterSunsubscribeOfANonServedSlotDoesNotRerouteTheServedOne887; real:TestClusterMoveOfASecondSlotOnTheSameNodeKeepsTheSubscriptionConnpsFailedfailingis afailFlag:Store(command path, refresh) takes the node over, a PubSub success clears only a mark a failed PubSub dial was last to setTestPubSubDialSuccessDoesNotClearAMarkTheCommandPathSetAfterwards887TestClusterFirstChannelNotSmallestStaysServed887,TestClusterRerouteFollowsTheServedSlotNotTheSmallestChannel887NewClusterClientRuns on
3b3de72: new and old tests-race -count=20Linux--cpus 2: ok (633 s); the F6 test andTestClusterRerouteAnswersEveryChannelOfAMoveWithOneLoop887-count=50: ok (81 s);go test -race ./driver/redis/...: ok (77 s); real redis 7.4.11, four real cells x3: green (N2 red before); darwin-race -count=100of the handshake test and of the close test: 0 races. Lint v2.14 (also-tags redis_cluster): 0 issues;GOOS=windows/darwin/linux go vet: clean;mage apiCheck: matches 53 packages.Mutants on
3b3de72(33,logs/mutants-r2.log), all red: the 28 of round 1 (the sunsubscribe ones now also die in the N2 test) plus SUNSUB-ANY-SLOT, PSFAILED-STALE, SERVED-SLOT-ALWAYS-SMALLEST (the reviewer's M7), PICK-SMALLEST-OF-ANY-SLOT (the reviewer's M10) and HANDSHAKE-RELEASE-ON-ANY-ERROR (red in the handshake test, on darwin-race).Review round 3 (Opus, head 5978381: approve with fixes, nothing blocking) and what was done
TestClusterServedSlotIsReselectedWhenItsChannelsAreGone887(the reviewer's P1)failFlaginterleavings: a command'sStore(true)racing a PubSub success could lose the markfailFlagis oneatomic.Uint32(bits: failing, set by a failed PubSub dial);Storereplaces the word,clearPubSubis one CAS on the PubSub's own mark; no embeddedatomic.Bool(itsSwap/CompareAndSwapskipped the reset)TestFailFlagCommandMarkIsNotLostToAConcurrentPubSubSuccess887(300000 rounds) andTestFailFlagStateMachine8875978381: 1 to 16 lost marks in 300000 rounds, every runSSubscribegodoc)handshakereleases throughreleaseHandshakeReq, which runs a nil-checked hook; handshake runs once per connection,putRequestand the command path are untouchedTestHandshakeGivesBackNothingWhenTheContextEnds887(a server that never answers HELLO),TestHandshakeGivesBackTheRequestOfAnErrorReply887; no-race, any OS, 0.3 s-count=20uses about half of the macOS port range)The command path's four
failingsites are unchanged: no lock, no allocation. There is no redis cluster command benchmark in the repo, so two micro-benchmarks stand in (BenchmarkFailFlagStore887,BenchmarkAnyNode887, darwin arm64,-benchtime=3000000x -count=6, 0 allocs both before and after):anyNode16.2-17.4 ns before, 16.1-16.3 after;failing.Store(false)1.9-2.8 ns before, 1.6-2.3 after.Beyond the list: the sentinel test pushed
+switch-masterbefore the SentinelClient's background subscription existed (one failure in 20 stress runs); it now waits for that subscription (02177f8).Runs on
02177f8: new and old tests-race -count=20Linux--cpus 2: ok (649 s);go test -race ./driver/redis/...: ok (78 s); lint v2.14 (also-tags redis_cluster): 0 issues;GOOS=windows/darwin/linux go vet: clean;mage apiCheck: matches 53 packages. Mutants (37,logs/mutants-r3.log): all red except FAILFLAG-MARK-KEEPS-OLD, which is equivalent (markPubSubwithOr(failing|byPubSub)always yields the same word asStore). New: N2-NO-RESELECT, FAILFLAG-CLEAR-ANY, FAILFLAG-CLEAR-READ-THEN-WRITE (killed by the stress test only), and the existing HANDSHAKE-RELEASE-ON-ANY-ERROR now dies in the two deterministic tests.Not touched, as asked (to be filed):
pipeline.goslab reuse after a context error; memcacheddefer putRequestbefore the error check.