Repository navigation
Writes wait out a running arena compaction instead of being refused; hand-off test holds the lock before the daemon exits - #202
Merged
Conversation
…ng refused The engine mirror runs inside the write's control transaction, where flatsql compacts the arena only in memory, and not at all while a compaction copies its committed layout over the record stream: it answers "deferred" with nothing freed. A mirror that would pass the budget while a background pass was between its copy steps was refused although that pass was about to free the room (TestEngineArenaCompactsAtRuntimeInsteadOfRefusing, intermittently). storeOne, storeBatchChunk and importDatasetShardChunk take the store lock through lockWriteForMirror: when the mirror would pass the budget and a background pass is running, the write waits for that pass without the lock (its steps take the lock one at a time), bounded at two minutes, and checks again under the lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The case took the store lock only after its own wait on the daemon process returned, by retrying a non-blocking lock every 5 ms. The helper polls the daemon's pid every 100 ms and takes the lock as soon as it sees the exit, so whichever ran first after the exit got the lock. When the helper won, it took the free lock, swapped and restarted (ledger: daemon-exited, store-locked, apply, restart), and the case failed with "the helper did not report the held store lock". The helper behaved as designed: it found the lock free. The case now checks that the running daemon holds the lock, then queues a blocking flock on store.lock before the daemon exits. The kernel grants it as the daemon's process releases the lock, before the helper can observe the exit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Title: Fix two intermittent node tests: handoff-after-stop store lock race, arena mirror refused mid-compaction
Stack issue 736. Two tests that failed now and then on main.
1. feed-recovery/handoff-after-stop (update_feed_recovery_e2e_test.go:406), a race in the test
In the failing run (CI job 133865079959) the ledger reads stop-requested, stop-waiting, daemon-exited, store-locked, apply, restart.
store-lockedmeans the helper took the store lock itself and found it free, which is what the helper is designed to do. The case asked for the lock only after its owncmd.Waitreturned, retrying a non-blocking flock every 5 ms. The helper polls the daemon pid every 100 ms. So whichever process got there first after the exit won the lock.The fix is in the test. It checks that the running daemon holds the lock, then queues a blocking
flock(LOCK_EX)onstore.lockbefore the daemon exits. The kernel grants that lock as the daemon releases it, before the helper can see the exit. The helper and product code are unchanged.Control (Linux arm64 container, loopback only, ordinary user): the test was given a 150 ms delay after
d.exited.2. TestEngineArenaCompactsAtRuntimeInsteadOfRefusing, a product race
The engine mirror runs inside the write's control transaction. There, flatsql compacts only in memory. While a compaction is copying its committed layout over the record stream, flatsql does not compact at all: it returns "deferred" and frees nothing (flatsql 3.7.0
compactInMemoryUnlocked, phase Copying). If a mirror would pass the budget while a background pass was between copy steps, the mirror was refused, even though that pass was about to free the room. Instrumented failing run: before the refusal, the measure was 221312 bytes, the need 63232, the budget 262144, ArenaStats showed Compacting=true and DeadBytes 203072, and the background pass was running. The synchronous compaction freed nothing.The fix:
storeOne,storeBatchChunkandimportDatasetShardChunktake the store lock throughlockWriteForMirror. If the mirror would pass the budget and a background pass is running, the write waits for that pass without holding the lock (each step takes the lock by itself). The wait is bounded at 2 minutes, and the condition is checked again under the lock.Control (macOS, 20 ms delay before the compaction's settle hold, which widens the window):
internal/storagepackage with the change, macOS: PASS (exit 0, no skips).What this touches
sdn-server/internal/storage/engine_arena_compaction.go,flatsql.go,manifest.gosdn-server/cmd/spacedatanetwork/update_feed_recovery_e2e_test.go🤖 Generated with Claude Code