Skip to content

FlatSQL 3.12.7: a feed of over a million objects keeps every record over a restart - #13

Merged
TJKoury merged 4 commits into
mainfrom
issue-810-format5-restart-loss
Nov 17, 2026
Merged

TJKoury merged 4 commits into
mainfrom
issue-810-format5-restart-loss

Conversation

@TJKoury

@TJKoury TJKoury commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Repository: DigitalArsenal/flatsql
Branch: issue-810-format5-restart-loss (head 6913f93) into main
Title: Verify stream frames with a table bound sized to the frame (records lost over a restart past a million objects)

What was wrong

Every reader of a feed's $FMB, $FCK and $FSG frames used FlatBuffers' default Verifier, which refuses a buffer of more than 1,000,000 tables. A checkpoint's dictionary segment is one table per entry (one per object key). The first merged dictionary segment of a feed with over a million object keys was written, then refused by every reader:

  • the next background checkpoint failed, and so did the stop's checkpoint: the stop returned P4_E_CORRUPT (the node logs "format5: stop: corrupt file, quarantined (P4_E_CORRUPT)");
  • the open walked the commit chain through that segment, judged the feed index damaged and rebuilt it by replaying the stream; the replay stopped at the refused segment and cut the stream there (streamTruncate), dropping every record written after it.

Seen with the node: 2,000,000 $OMM in one lane, 1,105,920 to 1,126,400 left after a clean stop and start; 1,100,000 lost 18,656; 1,050,000 wrote the refused segment at the stop and rebuilt at the open (34 s) without losing records; 1,000,000 clean.

Control: the 1,093,635-entry dictionary segment of the failing store fails VerifyFSGBuffer with the default options and passes with max_tables = frame size / 4 + 1; a 282,624-entry segment passes both.

The change

  • cpp/src/p4/sds_frames.h: frameVerifierOptions(n): a table takes at least 4 bytes, so a frame of n bytes holds at most n/4 tables; frames are verified with that bound, never below the default. Every frame verifier in car.cpp, checkpoint.cpp, checkpoint_search.cpp, dag.cpp, frames.cpp and recovery.cpp uses it (16 sites; the SQL request verifier in mailbox.cpp is unchanged).
  • cpp/tests/p4/t_million_objects.cpp (in scripts/ci-e2e.sh): 1,100,000 distinct OMM objects in one feed, a forced compaction (a fresh generation, as a format-4 conversion writes it: the whole dictionary in one segment), 4,096 more objects, stop, open, every record read back with its bytes. Before the change: the stop answered -4 (P4_E_CORRUPT) and 1,100,000 of 1,104,096 records were held after the open. After: both stops OK, 1,104,096 held, bytes equal (33 s).
  • wasm/flatsql.wasm, wasm/integrity.json: the rebuilt artifact (wasi-sdk 30.0, FlatBuffers 8af3053e, SDS schema 1.256.0; the same build of v3.12.6 reproduces the published artifact byte for byte). sha256 19361edfb3e6613f857af8b5512fccae8c49a656ab42c99b7114e76a2f6ba175.

Proof with the node (macOS, patched runtime)

  • Node main's daemon with this engine embedded: 2,000,000 $OMM ingested into one lane, clean stop, start: lane count 2,000,000 before and after, no P4_E_CORRUPT, no quarantine line, ready 5.1 s. Every record of the stream as the stop left it (2,000,000) read back from the restarted store with its bytes: 0 missing, 0 differ. The store held a 1,085,443-entry dictionary segment.
  • Control, node main as released (flatsql 3.12.6): the same run lost records (2,000,000 before the stop, 1,105,920 after), the stop reported P4_E_CORRUPT, ready 35 s. Reopening that run's post-stop store copy with this engine: 2,000,000 held, 0 missing, 0 differ; with 3.12.6: 894,080 missing.
  • The native E2E flows of scripts/ci-e2e.sh pass (20 of 20).

Not in this change

  • A dictionary or latest segment has no size bound: a level merge or a fresh generation writes one segment for every object key. Near 3,000,000 object keys a dictionary segment (about 44 bytes an entry) passes the 128 MiB frame limit that every reader enforces, which would give the same loss. Row-order segments are already split (65,536 pairs) and their merges capped.
  • The open cuts the stream at the first complete frame it cannot verify, not only at a torn tail; that is what turned a reader's refusal into lost records.
  • An in-process 2,000,000-record test of the node trapped with bad_alloc inside checkpointPrepare on the background checkpoint thread (separate defect, reported on the stack issue).

After the release, the node takes it (pin flatsql@<version>, P4EngineSHA256).

Closes DigitalArsenal/spacedatanetwork-stack#810

🤖 Generated with Claude Code

TJKoury and others added 4 commits November 16, 2026 20:39
Every reader of a feed's $FMB, $FCK and $FSG frames used FlatBuffers'
default Verifier, which refuses a buffer of more than 1,000,000 tables.
A checkpoint's dictionary segment is one table per entry, so the first
merged dictionary segment of a feed with over a million object keys was
written and then refused by every reader: the next checkpoint failed,
the stop reported P4_E_CORRUPT, and the open judged the feed index
damaged, replayed the stream, stopped at that segment and cut the
stream there, dropping every record after it.

A table takes at least 4 bytes, so a frame of n bytes holds at most
n/4 tables: frames are verified with that bound (never below the
default).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
t_million_objects writes 1,100,000 distinct OMM objects into one feed,
compacts it (a fresh generation, as a format-4 conversion writes it:
the whole dictionary in one segment), writes 4,096 more, stops, opens
again and reads every record back with its bytes. Before the frame
verifier change the stop answered P4_E_CORRUPT and the 4,096 records
written after the compaction were gone after the open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wasm/flatsql.wasm is the wasm32-wasip1-threads build of this source
(wasi-sdk 30.0, FlatBuffers 8af3053e, SDS schema 1.256.0); the same
build of v3.12.6 reproduces its published artifact byte for byte
(sha256 aec32d3e0dd17a0f2272a5aa30761c5b6f445c7817a7c03375acf607752e836c).
New sha256 19361edfb3e6613f857af8b5512fccae8c49a656ab42c99b7114e76a2f6ba175.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ver a restart

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TJKoury
TJKoury merged commit bc9fa43 into main Nov 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant