Repository navigation
Node engine: FlatSQL 3.12.10; a lane of over a million objects keeps every record over a restart - #204
Merged
Conversation
… status, not an abort) flatsql 3.12.9 -> 3.12.10 from the registry (lockfile with integrity); nothing else. FlatSQL 3.12.10: flatsql_start creates every engine thread before any runs and returns FLATSQL_E_BUSY when one cannot be created, with nothing started; the browser guest reports it as FlatSQLThreadPoolError, Node as FlatSQLError -6. Served files that change: flatsql.wasm, integrity.json, runtime/browser-guest.mjs, index.mjs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ion object keys keep every record over a restart FlatSQL 3.12.8 verifies its stream frames with a table bound sized to the frame. A feed's merged dictionary segment holds one table per object key; past 1,000,000 keys every reader refused it under FlatBuffers' default bound, the stop answered P4_E_CORRUPT and the next open cut the stream at that segment, dropping the records written after it. FlatSQL 3.12.9 and 3.12.10 add thread start handling: flatsql_start returns FLATSQL_E_BUSY when a thread cannot be created, and starts none. The embedded engine (flatsqlrt and the embedded sdn-js copy) is the published package's wasm/flatsql.wasm byte for byte, sha256 00c92effd74897870b45345193dfa118c43cd79e72e559d9277e8e8ae3051b63 (its wasm/integrity.json), gitHead 5d5401f76751295cd47fae493c536ebc5bd97883. suite.versions.json names 3.12.10; the generated version info follows it. The store format is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…in agrees Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Repository: DigitalArsenal/space-data-network
Branch: issue-810-engine-3.12.10 into main
Title: Embed FlatSQL 3.12.10: format-5 stores past a million object keys keep every record over a restart
What was wrong
The node's Go-embedded engine was FlatSQL 3.12.6. With it, a format-5 lane holding more than about a million distinct object keys lost records over a clean restart. The engine verified its stream frames with FlatBuffers' default limit of 1,000,000 tables; a feed's merged dictionary segment holds one table per object key, so past a million keys every reader refused it. The stop logged
format5: stop: corrupt file, quarantined (P4_E_CORRUPT)and the next open cut the stream at that segment. Control on 3.12.6 (node main, same harness): 1,100,000$OMMwith distinct object keys in one lane, 1,100,000 before the stop, 1,081,344 after a clean stop and start (18,656 lost),P4_E_CORRUPTat the stop, ready 34.2 s.FlatSQL 3.12.8 verifies each frame with a table bound sized to the frame. 3.12.9 and 3.12.10 add thread-start handling only:
flatsql_startreturnsFLATSQL_E_BUSY(-6) and starts no thread when one cannot be created (the node already treats a negativeflatsql_startstatus as an error), plus browser worker-start handling.The change
suite.versions.json: flatsql 3.12.10;scripts/generate-suite-version-info.jsregeneratedsdn-js/src/version-info.generated.tsandsdn-server/internal/versioninfo/generated.go.sdn-server/internal/flatsqlrt/flatsql.wasmandsdn-server/cmd/spacedatanetwork/embedded/sdn-js/flatsql.wasm: the published package'swasm/flatsql.wasmbyte for byte, sha25600c92effd74897870b45345193dfa118c43cd79e72e559d9277e8e8ae3051b63(itswasm/integrity.json).sdn-server/internal/versioninfo/release.go:P4EngineSHA256is that sha256;sdn-server/internal/flatsqlrt/p4artifact.go:P4ThreadsPackageflatsql@3.12.10,P4ThreadsGitHead5d5401f76751295cd47fae493c536ebc5bd97883.sdn-js/package.jsonand its lock are not changed: PR 203 moves sdn-js to 3.12.10. The version check's "sdn-js flatsql matches suite manifest" passes once both are on main (on main today it fails: sdn-js 3.12.9, suite 3.12.6). The store format, stored bytes and wire formats are unchanged.Proof (macOS, patched runtime)
The direct-ingest restart run of the publication baseline (real daemons), with this branch's daemon:
$OMMwith distinct object keys ingested into one lane; lane count 1,100,000 before the stop and 1,100,000 after the restart; first record readable before and after; noP4_E_CORRUPTand no quarantine line in either daemon's log; ready 4.2 s.Checks
node scripts/check-version-consistency.jswith PR 203 merged on top: Failed 0 ("sdn-js flatsql matches suite manifest: 3.12.10", "npm registry flatsql exposes production version 3.12.10"). On this branch alone: one failure, the sdn-js mismatch (3.12.9 vs 3.12.10) that PR 203 resolves.go build ./...;go vetofinternal/flatsqlrt,internal/versioninfo,internal/storage/...,cmd/spacedatanetwork.go test -v ./internal/flatsqlrt/... ./internal/storage/...with the patched runtime: all nine packagesok; 45 tests skip, the same 45 as on the 3.12.8 engine, none for missing runtime patches../scripts/ci-local.sh deliveryand./scripts/ci-local.sh preflight: pass.Closes DigitalArsenal/spacedatanetwork-stack#810
🤖 Generated with Claude Code