Repository navigation
Browser: start the engine only when every thread has its worker (3.12.7) - #12
Merged
Merged
Conversation
In a browser each engine thread runs on a pooled worker that must be ready (armed) before the engine starts. The bundled SDK 0.8.24 pool gave each worker 1.5 s, kept the workers that were ready and dropped the rest; flatsql_start then had fewer workers than threads, std::thread failed and the build without exceptions aborted with RuntimeError: unreachable in flatsql::p4::startThreads. - browser-guest: the pool is made at start, sized from the configured engine's mailbox layout (service threads + one indexer per writer + the slow and checkpoint threads), armed with an 8 s deadline, checked (requested against armed) and retried up to three times. A pool that cannot be had rejects open() with FlatSQLThreadPoolError (needed, armed, armedPerAttempt); the engine is not started. - browser-host / node-host: a start() operation; index.js starts the engine through it. - space-data-module-sdk 0.8.24 -> 0.10.4 (exact); runtime and demo bundles rebuilt. - scripts/ci-browser-threads-e2e.mjs, run by npm run test:browser: the server holds back thread-worker scripts. Slow (3 s): before, the engine aborted at 1.5 s in startThreads; now it starts and answers. Never: FlatSQLThreadPoolError, no WASM fault. - Version 3.12.7. wasm/flatsql.wasm unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d attempts are open() options (4000 ms x 2) The pool's wait already ends as soon as every worker is ready (SDK 0.10.4 armBrowserPool settles each worker on ready, on an error event or failed instantiation at once, or on its timer; partial pools resolve when all have settled). A worker script that never loads makes createWasiThreadSpawn throw WasiThreadWorkerUnreachableError at once. spawnReport() gives counts only (poolSize, armed, failedToArm), no per-worker reason; the reason of a failed worker goes to the console. - An attempt that ends before the arming time lost workers to errors: it is not retried, and open() rejects at once with FlatSQLThreadPoolError naming the reason (the SDK's unreachable-worker error, or the failed count). Only an attempt that ran out of time is retried, with fresh workers. - New open() options threadArmingMs (default 4000, 100-60000) and threadAttempts (default 2, 1-5); worst case for workers that never answer: 8 s. - Defaults from measurement in headless Chromium, two pools of 8 workers each (the console's catalog and places stores), 20 fresh contexts: pool arming 13/15/18 ms (fastest/median/slowest) at no slow-down, 47/68/95 ms with the browser's processes slowed fourfold. The rest of the budget is for the worker-script download: on the live site 1 fetch in 20 took 2.26 s (the rest 40-240 ms), and on a fresh page every worker waits for that one download. - FlatSQLThreadPoolError carries attempts (armed, failed, ms, reason per attempt). - ci-browser-threads-e2e: slow (script after 2.3 s) starts the engine; broken (404) rejects within a second of the last failure; stalled rejects after the 8 s budget. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rd) into browser-thread-start; version 3.12.8 No overlap: main changed cpp/src/p4, its test, wasm/flatsql.wasm and wasm/integrity.json (kept exactly), scripts/ci-e2e.sh and the 3.12.7 changelog entry. This branch's entry becomes 3.12.8; package.json and package-lock.json 3.12.8. npm run build rebuilt the runtime and demo bundles: the demo copies docs/assets/unified/flatsql.wasm and integrity.json now equal main's wasm/flatsql.wasm and wasm/integrity.json (main had left the demo copies at 3.12.6's artifact). wasm/integrity.json is unchanged and matches (check:integrity). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…) into browser-thread-start; version 3.12.9 main released 3.12.8 (tag v3.12.8, 797a3cc) with the docs copy of the engine refreshed, the same bytes this branch's build had produced. No overlap with this branch's files besides CHANGELOG.md and the version. This branch's entry becomes 3.12.9; package.json and package-lock.json 3.12.9. npm run build changes nothing further; wasm/flatsql.wasm and wasm/integrity.json stay main's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was wrong
In a browser each engine thread runs on a pooled worker that must be ready (armed) before
flatsql_start. The runtime bundled space-data-module-sdk 0.8.24, whose pool gave each worker 1.5 s to arm, kept the ready ones and dropped the rest without saying so. On a busy machine some workers missed the 1.5 s;flatsql_startthen had fewer workers than threads,std::threadfailed, and the WASM build (no exceptions) aborted:RuntimeError: unreachableatabort <- __libcpp_verbose_abort <- __throw_system_error <- flatsql::p4::startThreads <- flatsql_start.What changes
wasm/browser-guest.mjs: the thread pool is created at start, afterflatsql_init, sized from the configured engine's mailbox layout: service threads (nThreads) + one indexer per writer (class 1) + the slow and checkpoint threads, asstartThreadscreates them. Arming deadline 8 s per attempt (SDKprobeTimeoutMs), requested against armed checked throughspawnReport(), up to 3 attempts. The engine starts only when armed equals needed. Otherwiseopen()rejects withFlatSQLThreadPoolError(needed,armed= most workers ready in any attempt,armedPerAttempt); the WASM engine is never started.call('flatsql_start')is refused in the browser guest.wasm/browser-host.mjs,wasm/node-host.mjs: astart()operation;wasm/index.jsstarts the engine through it. Node behavior unchanged (it callsflatsql_start).npm run build).scripts/ci-browser-threads-e2e.mjs, now part ofnpm run test:browser(so both CI and the release workflow run it). The test server holds back thread-worker scripts after the first three requests:startThreads(exact production stack); now it starts and answers a request.FlatSQLThreadPoolError("FlatSQL starts 8 engine threads; the browser armed at most 3 of 8 thread workers (3, 0, 0 in 3 attempts of 8000 ms each)"), no WASM fault on the page.docs/API-3.12.md(browser hosting),docs/CI.md,CHANGELOG.md3.12.7. Version 3.12.7.wasm/flatsql.wasmandwasm/integrity.jsonare unchanged.What the SDK does not expose
Nothing missing for this:
createWasiThreadSpawn0.10.4 takesprobeTimeoutMsand returnsspawnReport()withpoolSize(requested) andarmed.Checks
scripts/check.mjs --base origin/main --e2e "cmd node scripts/ci-browser-threads-e2e.mjs": passed.npm test(Node unified E2E): pass.bash scripts/ci-e2e.sh --wasm-only: 6/6 pass.npm run test:browser: website E2E (all 286 SDS generators) and the thread E2E pass.scripts/ci-e2e.sh(no C++ change) and the release workflow's wasi-sdk rebuild (WASM unchanged).Release steps (repository rules: release commit on main, tag v, npm publish only through
.github/workflows/npm-publish.ymlwith OIDC)browser-thread-startintomain(the branch already carries the 3.12.7 version, lockfile and CHANGELOG):git checkout main && git pull && git merge --no-ff browser-thread-start && git push origin main.ci.ymlon main to pass (it runsnpm run test:browser, which now includes the thread E2E).git tag v3.12.7 <merge sha> && git push origin v3.12.7. The tag must equalpackage.jsonversion (the workflow checks it).npm-publish.ymlruns on the tag: rebuilds the WASM with wasi-sdk 30 and checks it equals the committed bytes, runsscripts/ci-e2e.shandnpm run test:browser, checks committed runtime/demo assets are unchanged, publishes with provenance and creates the GitHub release.npm view flatsql@3.12.7 versionand thatwasm/runtime/NOTICE.txtin the tarball names SDK 0.10.4.🤖 Generated with Claude Code