Skip to content

fix(sqlite): align Crab with the fixed WAL runtime - #577

Closed
forhappy wants to merge 1 commit into
codex/cache-catalog-iofrom
codex/sqlite-wal-safety
Closed

forhappy wants to merge 1 commit into
codex/cache-catalog-iofrom
codex/sqlite-wal-safety

Conversation

@forhappy

@forhappy forhappy commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Crab and its pinned Cellule runtime share a bundled SQLite 3.49.1, which is affected by the upstream WAL-reset race. Select the minimal Cellule backport and rusqlite 0.39 so all native SQLite consumers resolve to the fixed 3.51.3 engine.

Stacked on #574. Draft: the official installation build, affected-consumer CI and failed history-push rerun remain open. No merge requested.

Change

  • Align all five explicit Cellule workspace dependencies at 25809dc7a3acc37411776d79f526512088e466c3, the small backport in Cellule #78. The Cellule type package follows transitively at the same commit. This retains Crab's exact 0.1 implementation plus the SQLite correction; it does not import current Cellule's unrelated 0.2 Node Bundle migration. The equivalent current-main fix is Cellule #79.
  • Preserve range-checked unsigned SQL conversions through upstream's fallible_uint feature.
  • Propagate fallible progress-hook registration through the existing typed inspection error path before SQL. Deadline/cancellation attribution and private VFS ownership remain unchanged.
  • Update the cache contract documentation and review the lockfile closure. No patch override, vendor change, schema/configuration option or full-scan reader fallback.

Proof

  • All 309 cache library tests and three credential integration tests pass with local-cache, Xet chunk cache and remote-client enabled on the new dependency. This includes private VFS, WAL/generation replacement, corruption, cleanup and inspection cancellation/deadline coverage.
  • Strict combined-feature all-target cache Clippy, formatting and whitespace checks pass.
  • A SQL query through the compiled rusqlite archive used by Crab's qualification reports SQLite 3.51.3, source ID 2026-03-13 10:38:09 737ae4a34738ffa0c3ff7f9bb18df914dd1cad163f28fd6b6e114a344fe6d618. This is linked-engine evidence, not only manifest inspection (sqlite-linked-version-probe.json).
  • Upstream backport: the borrowed-authorizer regression fails before its required API adaptation and passes afterward; 44 primitive integration tests pass, including actual SQL mutation/publication/exact restore. LTX replica qualification passes 240 executable tests and a doctest; local-only LTX passes 35 library and ten integration tests plus its doctest. Strict LTX/runtime all-target/all-feature Clippy passes.
  • Four separate real RustFS tests pass on the pinned backport: native/bundled range compaction and reconstruction, fragmented demand reads, origin-admission release before optional cache sync, and concurrent sparse activation. Each existing test uses its own nonce-scoped prefix. Two manual provider-cost/filesystem diagnostics remain unrun.
  • Current-main Cellule qualification separately passes 56 primitive tests, 322 LTX replica executable tests plus a doctest, local-only LTX (55 library/ten integration/one doctest), and strict Clippy. Those results do not replace proof of the pinned backport.

Remaining gates

The official make install build is running with a private install prefix, external per-worktree target and locked release dependencies. All 1,218 Rust/manifest files are hashed at this commit for the live rerun. CLI/server/staging/metadata/workflow consumer builds and native-platform CI must be checked before readiness.

The previous 1,024-file history push still has a failed 900-second result. Its fresh rerun preserves both workloads and that deadline. Neither this dependency correction nor passing cache tests establishes the requested throughput improvement, unified staging, 10-GiB qualification or completion of the full initiative.

Review evidence map

Boundary Evidence
Entry/caller Shard-hint health inspection installs its bounded progress callback before entering a SQL transaction
Ownership crab-cache private Database retains its owned connection/root/generation; Cellule owns its runtime/LTX dependency adaptation
Callee rusqlite 0.39 hook registration returns Result; libsqlite3-sys 0.37 bundles the fixed SQLite
Siblings All workspace SQLite users share the single upgraded native dependency; exact Cellule pin preserves existing storage/runtime behavior
Tests Combined cache/private-VFS suite, upstream capture/restore and SQL authorization suite, four real RustFS cases
Baseline Parent #574 uses SQLite 3.49.1; official upstream release identifies the race and fixed source

Is this the best fix? Upgrade the maintained shared dependency and its actual callers, using a minimal pinned-runtime backport. This avoids a private SQLite patch, coarse global serialization, or importing a broad unrelated storage migration. Remaining build/live gates are explicit.

Primary contracts: SQLite WAL-reset fix, rusqlite 0.39 release.

Both requested large-file audit reports and unrelated advisor-plan edits are excluded.

Native/RustFS qualification completed on frozen commit 4ebfc3dcecea7ff6ee424b17eae6bb443cb30755: the original 4 MiB model / 4 KiB edit workload against 16 and 1,024 distinct 64 KiB files passed all 48 commands and 23 checks with unchanged 900-second command deadlines. Both cold hydrations passed; both raw remote-fsck results independently report passed=true, errors=0 and repair_failures=0. All 1,218 guarded source files and the officially installed FUSE binary SHA remained unchanged. The previous 1,024-file initial push timed out; this result qualifies the combined cache/WAL and SQLite stack, not isolated attribution to a single change. It is not 10 GiB or representative hosted-S3 latency qualification.

A separate fixed-model hydration probe also passed (14 commands / 26 checks): only the same 4 MiB model is reconstructed, while all 16/1,024 unrelated files remain pointers. A fresh client cache created after clone took 1,241/1,411 ms; reusing that cache after dehydrate took 284/746 ms. The source/binary guard remained unchanged. Provider/OS cache state is uncontrolled. The earlier R2 hydrate --all timings include growing output and must not be used as a fixed-file index-scaling result; request/byte/page scaling and representative hosted-S3 latency remain separate gates.

Select the minimal Cellule backport and bundled SQLite 3.51.3. Preserve checked unsigned bindings and propagate inspection hook installation errors before SQL. Keep the unrelated Cellule 0.2 storage migration out of this dependency correction.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant