Skip to content

Writes wait out a running arena compaction instead of being refused; hand-off test holds the lock before the daemon exits - #202

Merged
TJKoury merged 3 commits into
mainfrom
issue-736-flaky-handoff-arena
Nov 17, 2026
Merged

TJKoury merged 3 commits into
mainfrom
issue-736-flaky-handoff-arena

Conversation

@TJKoury

@TJKoury TJKoury commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Title: Fix two intermittent node tests: handoff-after-stop store lock race, arena mirror refused mid-compaction

Stack issue 736. Two tests that failed now and then on main.

1. feed-recovery/handoff-after-stop (update_feed_recovery_e2e_test.go:406), a race in the test

In the failing run (CI job 133865079959) the ledger reads stop-requested, stop-waiting, daemon-exited, store-locked, apply, restart. store-locked means the helper took the store lock itself and found it free, which is what the helper is designed to do. The case asked for the lock only after its own cmd.Wait returned, retrying a non-blocking flock every 5 ms. The helper polls the daemon pid every 100 ms. So whichever process got there first after the exit won the lock.

The fix is in the test. It checks that the running daemon holds the lock, then queues a blocking flock(LOCK_EX) on store.lock before the daemon exits. The kernel grants that lock as the daemon releases it, before the helper can see the exit. The helper and product code are unchanged.

Control (Linux arm64 container, loopback only, ordinary user): the test was given a 150 ms delay after d.exited.

  • Before the change: 0 of 10 passed. All 10 failed at :406 with the same ledger as CI.
  • After the change: 10 of 10 passed.
  • Unmodified test after the change: 5 of 5 passed.

2. TestEngineArenaCompactsAtRuntimeInsteadOfRefusing, a product race

The engine mirror runs inside the write's control transaction. There, flatsql compacts only in memory. While a compaction is copying its committed layout over the record stream, flatsql does not compact at all: it returns "deferred" and frees nothing (flatsql 3.7.0 compactInMemoryUnlocked, phase Copying). If a mirror would pass the budget while a background pass was between copy steps, the mirror was refused, even though that pass was about to free the room. Instrumented failing run: before the refusal, the measure was 221312 bytes, the need 63232, the budget 262144, ArenaStats showed Compacting=true and DeadBytes 203072, and the background pass was running. The synchronous compaction freed nothing.

The fix: storeOne, storeBatchChunk and importDatasetShardChunk take the store lock through lockWriteForMirror. If the mirror would pass the budget and a background pass is running, the write waits for that pass without holding the lock (each step takes the lock by itself). The wait is bounded at 2 minutes, and the condition is checked again under the lock.

Control (macOS, 20 ms delay before the compaction's settle hold, which widens the window):

  • Before the change: 8 of 30 passed (22 failed with "63232 bytes refused"). In an earlier set, 6 of 10 passed.
  • A first attempt (sync the arena measure inside the step) still failed 5 of 20. It was dropped, and the instrumented run above found the cause.
  • After the change: 30 of 30 passed.
  • Without the delay: before, 50 of 50 passed, and 50 of 50 with 24 CPU hogs. After, 50 of 50 with 24 hogs. The failure did not reproduce on this Mac without the delay.
  • Full internal/storage package with the change, macOS: PASS (exit 0, no skips).
  • Linux arm64 container (4 CPUs, 4 hogs), before the change, no delay: 30 of 30 passed. The failure did not reproduce there either, so only the controlled runs show the fix.

What this touches

  • sdn-server/internal/storage/engine_arena_compaction.go, flatsql.go, manifest.go
  • sdn-server/cmd/spacedatanetwork/update_feed_recovery_e2e_test.go

🤖 Generated with Claude Code

TJKoury and others added 3 commits November 17, 2026 02:38
…ng refused

The engine mirror runs inside the write's control transaction, where flatsql
compacts the arena only in memory, and not at all while a compaction copies
its committed layout over the record stream: it answers "deferred" with
nothing freed. A mirror that would pass the budget while a background pass
was between its copy steps was refused although that pass was about to free
the room (TestEngineArenaCompactsAtRuntimeInsteadOfRefusing, intermittently).

storeOne, storeBatchChunk and importDatasetShardChunk take the store lock
through lockWriteForMirror: when the mirror would pass the budget and a
background pass is running, the write waits for that pass without the lock
(its steps take the lock one at a time), bounded at two minutes, and checks
again under the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The case took the store lock only after its own wait on the daemon process
returned, by retrying a non-blocking lock every 5 ms. The helper polls the
daemon's pid every 100 ms and takes the lock as soon as it sees the exit, so
whichever ran first after the exit got the lock. When the helper won, it took
the free lock, swapped and restarted (ledger: daemon-exited, store-locked,
apply, restart), and the case failed with "the helper did not report the held
store lock". The helper behaved as designed: it found the lock free.

The case now checks that the running daemon holds the lock, then queues a
blocking flock on store.lock before the daemon exits. The kernel grants it as
the daemon's process releases the lock, before the helper can observe the
exit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TJKoury
TJKoury merged commit 59a0248 into main Nov 17, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant