Skip to content

Node engine: FlatSQL 3.12.10; a lane of over a million objects keeps every record over a restart - #204

Merged
TJKoury merged 4 commits into
mainfrom
issue-810-engine-3.12.10
Nov 17, 2026
Merged

TJKoury merged 4 commits into
mainfrom
issue-810-engine-3.12.10

Conversation

@TJKoury

@TJKoury TJKoury commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Repository: DigitalArsenal/space-data-network
Branch: issue-810-engine-3.12.10 into main
Title: Embed FlatSQL 3.12.10: format-5 stores past a million object keys keep every record over a restart

What was wrong

The node's Go-embedded engine was FlatSQL 3.12.6. With it, a format-5 lane holding more than about a million distinct object keys lost records over a clean restart. The engine verified its stream frames with FlatBuffers' default limit of 1,000,000 tables; a feed's merged dictionary segment holds one table per object key, so past a million keys every reader refused it. The stop logged format5: stop: corrupt file, quarantined (P4_E_CORRUPT) and the next open cut the stream at that segment. Control on 3.12.6 (node main, same harness): 1,100,000 $OMM with distinct object keys in one lane, 1,100,000 before the stop, 1,081,344 after a clean stop and start (18,656 lost), P4_E_CORRUPT at the stop, ready 34.2 s.

FlatSQL 3.12.8 verifies each frame with a table bound sized to the frame. 3.12.9 and 3.12.10 add thread-start handling only: flatsql_start returns FLATSQL_E_BUSY (-6) and starts no thread when one cannot be created (the node already treats a negative flatsql_start status as an error), plus browser worker-start handling.

The change

  • suite.versions.json: flatsql 3.12.10; scripts/generate-suite-version-info.js regenerated sdn-js/src/version-info.generated.ts and sdn-server/internal/versioninfo/generated.go.
  • sdn-server/internal/flatsqlrt/flatsql.wasm and sdn-server/cmd/spacedatanetwork/embedded/sdn-js/flatsql.wasm: the published package's wasm/flatsql.wasm byte for byte, sha256 00c92effd74897870b45345193dfa118c43cd79e72e559d9277e8e8ae3051b63 (its wasm/integrity.json).
  • sdn-server/internal/versioninfo/release.go: P4EngineSHA256 is that sha256; sdn-server/internal/flatsqlrt/p4artifact.go: P4ThreadsPackage flatsql@3.12.10, P4ThreadsGitHead 5d5401f76751295cd47fae493c536ebc5bd97883.

sdn-js/package.json and its lock are not changed: PR 203 moves sdn-js to 3.12.10. The version check's "sdn-js flatsql matches suite manifest" passes once both are on main (on main today it fails: sdn-js 3.12.9, suite 3.12.6). The store format, stored bytes and wire formats are unchanged.

Proof (macOS, patched runtime)

The direct-ingest restart run of the publication baseline (real daemons), with this branch's daemon:

  • 1,100,000 $OMM with distinct object keys ingested into one lane; lane count 1,100,000 before the stop and 1,100,000 after the restart; first record readable before and after; no P4_E_CORRUPT and no quarantine line in either daemon's log; ready 4.2 s.
  • The OMM stream as the stop left it holds a 1,100,003-entry dictionary segment (more than the old 1,000,000-table limit).
  • Full comparison: every record frame of that stream (1,100,000) read back from the restarted store with its bytes: 0 missing, 0 differ.

Checks

  • node scripts/check-version-consistency.js with PR 203 merged on top: Failed 0 ("sdn-js flatsql matches suite manifest: 3.12.10", "npm registry flatsql exposes production version 3.12.10"). On this branch alone: one failure, the sdn-js mismatch (3.12.9 vs 3.12.10) that PR 203 resolves.
  • go build ./...; go vet of internal/flatsqlrt, internal/versioninfo, internal/storage/..., cmd/spacedatanetwork.
  • go test -v ./internal/flatsqlrt/... ./internal/storage/... with the patched runtime: all nine packages ok; 45 tests skip, the same 45 as on the 3.12.8 engine, none for missing runtime patches.
  • ./scripts/ci-local.sh delivery and ./scripts/ci-local.sh preflight: pass.

Closes DigitalArsenal/spacedatanetwork-stack#810

🤖 Generated with Claude Code

TJKoury and others added 4 commits November 17, 2026 03:53
… status, not an abort)

flatsql 3.12.9 -> 3.12.10 from the registry (lockfile with integrity); nothing else. FlatSQL 3.12.10: flatsql_start creates every engine thread before any runs and returns FLATSQL_E_BUSY when one cannot be created, with nothing started; the browser guest reports it as FlatSQLThreadPoolError, Node as FlatSQLError -6. Served files that change: flatsql.wasm, integrity.json, runtime/browser-guest.mjs, index.mjs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ion object keys keep every record over a restart

FlatSQL 3.12.8 verifies its stream frames with a table bound sized to the
frame. A feed's merged dictionary segment holds one table per object key; past
1,000,000 keys every reader refused it under FlatBuffers' default bound, the
stop answered P4_E_CORRUPT and the next open cut the stream at that segment,
dropping the records written after it. FlatSQL 3.12.9 and 3.12.10 add thread
start handling: flatsql_start returns FLATSQL_E_BUSY when a thread cannot be
created, and starts none.

The embedded engine (flatsqlrt and the embedded sdn-js copy) is the published
package's wasm/flatsql.wasm byte for byte, sha256
00c92effd74897870b45345193dfa118c43cd79e72e559d9277e8e8ae3051b63 (its
wasm/integrity.json), gitHead 5d5401f76751295cd47fae493c536ebc5bd97883.
suite.versions.json names 3.12.10; the generated version info follows it. The
store format is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…in agrees

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@TJKoury
TJKoury merged commit 90a5875 into main Nov 17, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant