[xe-nvdk]

Covers what the first round of live testing (#30) couldn't reach. Two new rigs, both built from Arc main eab8c078.

Shared storage: tests/integration/test_live_shared.py

Opt-in via ARC_TEST_SHARED_URLS. The rig is enterprise-shared (SeaweedFS as the hot bucket) plus a dedicated compactor node, an AWS S3 cold tier, and mosquitto.

• Compactor lease: assign_compactor moves the lease to a writer and back. Assigning the current holder returns 409.
• Daily compaction: runs on the dedicated compactor and shows up in history and cycles.
• Tiering: policies (set/get/effective/list/delete), a dry run then a real migration to the cold bucket, and the rows still read from a writer and from the reader.
• MQTT: the full subscription lifecycle against a real broker. Published messages are stored and queryable.
• unflushed_nodes: a frozen writer holds buffered rows. The 503 refusal keeps the body, and allow_unreachable reports the frozen writer for both the data delete and the database drop.
• Strict xfail: tiering: GET /api/v1/tiering/files ignores tier when database is also set arc#1430, tiering files ignores tier when database is set.

Stream takeover: test_live_cluster.py

• New opt-in test test_stream_claim_after_primary_killed (ARC_TEST_CLUSTER_STREAM_TAKEOVER=1). It kills the primary while its rows are unflushed. A standby claims and publishes them exactly once. The test then checks that releasing or abandoning a finished claim returns 409, a wrong epoch returns 404, and a follower returns 409.
• Fixes test_release_stream_claim: it filtered on finished, but Arc's field is done.

Doc fixes from the runs

• MQTT: subscriptions start stopped (auto_start means start when Arc boots), and each one lives on the node that created it.
• unflushed_nodes: entries are messages naming each node, not bare ids.
• Deletes: Arc waits about 30-40 s for unreachable peers before it answers.
• tiering.files: notes arc#1430.

Arc issues found

• tiering: GET /api/v1/tiering/files ignores tier when database is also set arc#1430: tiering files ignores tier when database is set.
• cluster: a node stalled ≥15s never rejoins Raft (auth handshake failures trip raft accept backoff → livelock) arc#1431: a node stalled 15 s or more never rejoins Raft. Every delete on the cluster then fails, while the node still reports healthy. The suite restarts the frozen node to work around it.
• Deleted database still listed by GET /api/v1/databases on local storage (get returns 404) arc#1425 (comment): deleted databases are also still listed on SeaweedFS.

Results

Suite	Result
Shared storage	6 passed + 1 xfail, on 2 consecutive full runs after the last test fix
Tiering tests alone	6 consecutive passes
Stream claims	3 passed, 3 consecutive runs
Unit	1087 passed

https://claude.ai/code/session_01VMc4wAG9mUUc3wB2ingB8c