Repository navigation
FlatSQL 3.12.7: a feed of over a million objects keeps every record over a restart - #13
Merged
Merged
Conversation
Every reader of a feed's $FMB, $FCK and $FSG frames used FlatBuffers' default Verifier, which refuses a buffer of more than 1,000,000 tables. A checkpoint's dictionary segment is one table per entry, so the first merged dictionary segment of a feed with over a million object keys was written and then refused by every reader: the next checkpoint failed, the stop reported P4_E_CORRUPT, and the open judged the feed index damaged, replayed the stream, stopped at that segment and cut the stream there, dropping every record after it. A table takes at least 4 bytes, so a frame of n bytes holds at most n/4 tables: frames are verified with that bound (never below the default). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
t_million_objects writes 1,100,000 distinct OMM objects into one feed, compacts it (a fresh generation, as a format-4 conversion writes it: the whole dictionary in one segment), writes 4,096 more, stops, opens again and reads every record back with its bytes. Before the frame verifier change the stop answered P4_E_CORRUPT and the 4,096 records written after the compaction were gone after the open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wasm/flatsql.wasm is the wasm32-wasip1-threads build of this source (wasi-sdk 30.0, FlatBuffers 8af3053e, SDS schema 1.256.0); the same build of v3.12.6 reproduces its published artifact byte for byte (sha256 aec32d3e0dd17a0f2272a5aa30761c5b6f445c7817a7c03375acf607752e836c). New sha256 19361edfb3e6613f857af8b5512fccae8c49a656ab42c99b7114e76a2f6ba175. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ver a restart Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Repository: DigitalArsenal/flatsql
Branch: issue-810-format5-restart-loss (head 6913f93) into main
Title: Verify stream frames with a table bound sized to the frame (records lost over a restart past a million objects)
What was wrong
Every reader of a feed's
$FMB,$FCKand$FSGframes used FlatBuffers' defaultVerifier, which refuses a buffer of more than 1,000,000 tables. A checkpoint's dictionary segment is one table per entry (one per object key). The first merged dictionary segment of a feed with over a million object keys was written, then refused by every reader:P4_E_CORRUPT(the node logs "format5: stop: corrupt file, quarantined (P4_E_CORRUPT)");streamTruncate), dropping every record written after it.Seen with the node: 2,000,000
$OMMin one lane, 1,105,920 to 1,126,400 left after a clean stop and start; 1,100,000 lost 18,656; 1,050,000 wrote the refused segment at the stop and rebuilt at the open (34 s) without losing records; 1,000,000 clean.Control: the 1,093,635-entry dictionary segment of the failing store fails
VerifyFSGBufferwith the default options and passes withmax_tables= frame size / 4 + 1; a 282,624-entry segment passes both.The change
cpp/src/p4/sds_frames.h:frameVerifierOptions(n): a table takes at least 4 bytes, so a frame of n bytes holds at most n/4 tables; frames are verified with that bound, never below the default. Every frame verifier incar.cpp,checkpoint.cpp,checkpoint_search.cpp,dag.cpp,frames.cppandrecovery.cppuses it (16 sites; the SQL request verifier inmailbox.cppis unchanged).cpp/tests/p4/t_million_objects.cpp(inscripts/ci-e2e.sh): 1,100,000 distinct OMM objects in one feed, a forced compaction (a fresh generation, as a format-4 conversion writes it: the whole dictionary in one segment), 4,096 more objects, stop, open, every record read back with its bytes. Before the change: the stop answered -4 (P4_E_CORRUPT) and 1,100,000 of 1,104,096 records were held after the open. After: both stops OK, 1,104,096 held, bytes equal (33 s).wasm/flatsql.wasm,wasm/integrity.json: the rebuilt artifact (wasi-sdk 30.0, FlatBuffers 8af3053e, SDS schema 1.256.0; the same build of v3.12.6 reproduces the published artifact byte for byte). sha25619361edfb3e6613f857af8b5512fccae8c49a656ab42c99b7114e76a2f6ba175.Proof with the node (macOS, patched runtime)
$OMMingested into one lane, clean stop, start: lane count 2,000,000 before and after, noP4_E_CORRUPT, no quarantine line, ready 5.1 s. Every record of the stream as the stop left it (2,000,000) read back from the restarted store with its bytes: 0 missing, 0 differ. The store held a 1,085,443-entry dictionary segment.P4_E_CORRUPT, ready 35 s. Reopening that run's post-stop store copy with this engine: 2,000,000 held, 0 missing, 0 differ; with 3.12.6: 894,080 missing.scripts/ci-e2e.shpass (20 of 20).Not in this change
bad_allocinsidecheckpointPrepareon the background checkpoint thread (separate defect, reported on the stack issue).After the release, the node takes it (pin
flatsql@<version>,P4EngineSHA256).Closes DigitalArsenal/spacedatanetwork-stack#810
🤖 Generated with Claude Code