Skip to content

Browser: start the engine only when every thread has its worker (3.12.7) - #12

Merged
TJKoury merged 4 commits into
mainfrom
browser-thread-start
Nov 17, 2026
Merged

TJKoury merged 4 commits into
mainfrom
browser-thread-start

Conversation

@TJKoury

@TJKoury TJKoury commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

What was wrong

In a browser each engine thread runs on a pooled worker that must be ready (armed) before flatsql_start. The runtime bundled space-data-module-sdk 0.8.24, whose pool gave each worker 1.5 s to arm, kept the ready ones and dropped the rest without saying so. On a busy machine some workers missed the 1.5 s; flatsql_start then had fewer workers than threads, std::thread failed, and the WASM build (no exceptions) aborted: RuntimeError: unreachable at abort <- __libcpp_verbose_abort <- __throw_system_error <- flatsql::p4::startThreads <- flatsql_start.

What changes

  • wasm/browser-guest.mjs: the thread pool is created at start, after flatsql_init, sized from the configured engine's mailbox layout: service threads (nThreads) + one indexer per writer (class 1) + the slow and checkpoint threads, as startThreads creates them. Arming deadline 8 s per attempt (SDK probeTimeoutMs), requested against armed checked through spawnReport(), up to 3 attempts. The engine starts only when armed equals needed. Otherwise open() rejects with FlatSQLThreadPoolError (needed, armed = most workers ready in any attempt, armedPerAttempt); the WASM engine is never started. call('flatsql_start') is refused in the browser guest.
  • wasm/browser-host.mjs, wasm/node-host.mjs: a start() operation; wasm/index.js starts the engine through it. Node behavior unchanged (it calls flatsql_start).
  • space-data-module-sdk 0.8.24 -> 0.10.4 (exact). Runtime and demo bundles rebuilt (npm run build).
  • scripts/ci-browser-threads-e2e.mjs, now part of npm run test:browser (so both CI and the release workflow run it). The test server holds back thread-worker scripts after the first three requests:
    • slow (3 s): before this change the engine aborted at 1.5 s in startThreads (exact production stack); now it starts and answers a request.
    • never: FlatSQLThreadPoolError ("FlatSQL starts 8 engine threads; the browser armed at most 3 of 8 thread workers (3, 0, 0 in 3 attempts of 8000 ms each)"), no WASM fault on the page.
  • Docs: docs/API-3.12.md (browser hosting), docs/CI.md, CHANGELOG.md 3.12.7. Version 3.12.7.
  • wasm/flatsql.wasm and wasm/integrity.json are unchanged.

What the SDK does not expose

Nothing missing for this: createWasiThreadSpawn 0.10.4 takes probeTimeoutMs and returns spawnReport() with poolSize (requested) and armed.

Checks

  • Stack scripts/check.mjs --base origin/main --e2e "cmd node scripts/ci-browser-threads-e2e.mjs": passed.
  • npm test (Node unified E2E): pass. bash scripts/ci-e2e.sh --wasm-only: 6/6 pass. npm run test:browser: website E2E (all 286 SDS generators) and the thread E2E pass.
  • Not run here: the native part of scripts/ci-e2e.sh (no C++ change) and the release workflow's wasi-sdk rebuild (WASM unchanged).

Release steps (repository rules: release commit on main, tag v, npm publish only through .github/workflows/npm-publish.yml with OIDC)

  1. Merge browser-thread-start into main (the branch already carries the 3.12.7 version, lockfile and CHANGELOG): git checkout main && git pull && git merge --no-ff browser-thread-start && git push origin main.
  2. Wait for ci.yml on main to pass (it runs npm run test:browser, which now includes the thread E2E).
  3. Tag and push: git tag v3.12.7 <merge sha> && git push origin v3.12.7. The tag must equal package.json version (the workflow checks it).
  4. npm-publish.yml runs on the tag: rebuilds the WASM with wasi-sdk 30 and checks it equals the committed bytes, runs scripts/ci-e2e.sh and npm run test:browser, checks committed runtime/demo assets are unchanged, publishes with provenance and creates the GitHub release.
  5. Confirm: npm view flatsql@3.12.7 version and that wasm/runtime/NOTICE.txt in the tarball names SDK 0.10.4.

🤖 Generated with Claude Code

TJKoury and others added 4 commits November 16, 2026 21:48
In a browser each engine thread runs on a pooled worker that must be ready (armed) before the engine starts. The bundled SDK 0.8.24 pool gave each worker 1.5 s, kept the workers that were ready and dropped the rest; flatsql_start then had fewer workers than threads, std::thread failed and the build without exceptions aborted with RuntimeError: unreachable in flatsql::p4::startThreads.

- browser-guest: the pool is made at start, sized from the configured engine's mailbox layout (service threads + one indexer per writer + the slow and checkpoint threads), armed with an 8 s deadline, checked (requested against armed) and retried up to three times. A pool that cannot be had rejects open() with FlatSQLThreadPoolError (needed, armed, armedPerAttempt); the engine is not started.
- browser-host / node-host: a start() operation; index.js starts the engine through it.
- space-data-module-sdk 0.8.24 -> 0.10.4 (exact); runtime and demo bundles rebuilt.
- scripts/ci-browser-threads-e2e.mjs, run by npm run test:browser: the server holds back thread-worker scripts. Slow (3 s): before, the engine aborted at 1.5 s in startThreads; now it starts and answers. Never: FlatSQLThreadPoolError, no WASM fault.
- Version 3.12.7. wasm/flatsql.wasm unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d attempts are open() options (4000 ms x 2)

The pool's wait already ends as soon as every worker is ready (SDK 0.10.4 armBrowserPool settles each worker on ready, on an error event or failed instantiation at once, or on its timer; partial pools resolve when all have settled). A worker script that never loads makes createWasiThreadSpawn throw WasiThreadWorkerUnreachableError at once. spawnReport() gives counts only (poolSize, armed, failedToArm), no per-worker reason; the reason of a failed worker goes to the console.

- An attempt that ends before the arming time lost workers to errors: it is not retried, and open() rejects at once with FlatSQLThreadPoolError naming the reason (the SDK's unreachable-worker error, or the failed count). Only an attempt that ran out of time is retried, with fresh workers.
- New open() options threadArmingMs (default 4000, 100-60000) and threadAttempts (default 2, 1-5); worst case for workers that never answer: 8 s.
- Defaults from measurement in headless Chromium, two pools of 8 workers each (the console's catalog and places stores), 20 fresh contexts: pool arming 13/15/18 ms (fastest/median/slowest) at no slow-down, 47/68/95 ms with the browser's processes slowed fourfold. The rest of the budget is for the worker-script download: on the live site 1 fetch in 20 took 2.26 s (the rest 40-240 ms), and on a fresh page every worker waits for that one download.
- FlatSQLThreadPoolError carries attempts (armed, failed, ms, reason per attempt).
- ci-browser-threads-e2e: slow (script after 2.3 s) starts the engine; broken (404) rejects within a second of the last failure; stalled rejects after the 8 s budget.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rd) into browser-thread-start; version 3.12.8

No overlap: main changed cpp/src/p4, its test, wasm/flatsql.wasm and wasm/integrity.json (kept exactly), scripts/ci-e2e.sh and the 3.12.7 changelog entry. This branch's entry becomes 3.12.8; package.json and package-lock.json 3.12.8. npm run build rebuilt the runtime and demo bundles: the demo copies docs/assets/unified/flatsql.wasm and integrity.json now equal main's wasm/flatsql.wasm and wasm/integrity.json (main had left the demo copies at 3.12.6's artifact). wasm/integrity.json is unchanged and matches (check:integrity).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…) into browser-thread-start; version 3.12.9

main released 3.12.8 (tag v3.12.8, 797a3cc) with the docs copy of the engine refreshed, the same bytes this branch's build had produced. No overlap with this branch's files besides CHANGELOG.md and the version. This branch's entry becomes 3.12.9; package.json and package-lock.json 3.12.9. npm run build changes nothing further; wasm/flatsql.wasm and wasm/integrity.json stay main's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TJKoury
TJKoury merged commit 4d66843 into main Nov 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant