feat(data): ordered version-upgrade system for persisted schemas

[krisnye]

Summary

A pluggable schema version-upgrade system for persisted databases, built on the coerce foundation (#183). Additive & minor schema changes need no code — they're converted automatically on load; only a genuinely breaking change carries a handler. One co-located guard test fails (with a fix recipe) if the schema drifts from the recorded history or a handler lacks a test. The schema version is save-format metadata (db.version), never an ECS resource, and each persisted quadrant (document + settings) carries its own stamp so the two can drift and upgrade independently.

What an author writes

The whole authoring surface is an ordered versions array — one entry per schema change, each a JSON merge patch:

const versions: VersionEntry[] = [
  { version: 0, changes: { components: { hp: F32, score: F32 }, resources: { turn: { type: "integer", default: 0 } } } }, // initial
  { version: 1, changes: { components: { mana: F32 } } },                                    // additive — no code
  { version: 2, changes: { components: { score: { maximum: 100 } } } },                      // minor (clamp) — no code
  { version: 3, changes: { resources: { difficulty: { type: "string", default: "normal" } } } }, // additive resource — no code
  { version: 4, changes: { components: { hp: { type: "object", properties: HealthProps, precision: undefined, default: undefined } } }, // MAJOR
    handler: (store) => Store.remapComponent(store, "hp", Health, (old) => ({ current: old, max: old })) },
];

• entries[i] IS version i; entries[0] is the initial schema; currentVersion = entries.length - 1.
• changes is a JSON Merge Patch applied to the folded-so-far schema — record ONLY what changes: name: { …fields… } adds or deep-merges; a key set to undefined deletes it; null is a preserved value. A shape change deletes the fields it drops (number → object ⇒ precision: undefined, default: undefined). Empty changes: {} is rejected.
• handler is present iff the change is not auto-convertible (createCoerceFunction decides — the author never classifies). It runs against the store staged to the previous version and mutates it in place; for an isolated single-component change Store.remapComponent suffices (not required) — a migration spanning several components is hand-written.

Wire it up: Database.create(plugin, { versioning: createVersionUpgrader(versions) }).

The one guard test: a single assertVersioning(...) runs BOTH the schema-fold check and the per-handler test coverage:

it("versioning is consistent", () => assertVersioning({
  database: Database.create(plugin, { versioning: createVersionUpgrader(versions) }),
  entries: versions,
  handlers: { /* a case per version that adds a handler */ },
}));

On drift it throws an actionable recipe: classifies each change (auto → record a merge-patch, no handler; breaking → record it AND add a handler), prints the exact minimal merge patch (with undefined deletes) to append, flags removals as ⚠ DROPS DATA. It also enforces the invariants: only real, typed persisted schemas may be versioned; non-persistent state must not appear; each handler touches a single quadrant. (assertVersionsMatchSchema and testUpgradeHandlers remain exported for bespoke use.)

Version as save-format metadata (not a resource)

The schema version is not an ECS component/resource — it rides the serialized envelope:

• db.version is a first-class integer on the Database type (entries.length - 1, or 0 when unversioned).
• db.toData() stamps it into the save metadata per persisted quadrant: schemaVersions: { document, settings } (the db writes its single version into both).
• db.fromData(blob) reads schemaVersions from the metadata (absent ⇒ 0, a legacy blob) and hands the per-quadrant stamps to the upgrade handler.

This keeps the version out of the content data (it describes the blob's format), lets a cloud client stamp db.version into every message envelope, and — being per-quadrant — is forward-compatible with saving the document and settings blobs separately (each carries its own quadrant's stamp).

Per-quadrant upgrade

The two persisted quadrants — document (shared+persistent) and settings (nonShared+persistent) — can be saved to different backends and drift to different versions. Whether loaded as one combined blob (unscoped) or from separate per-quadrant blobs (scoped — see below), the upgrader reads each quadrant's own saved version from the metadata and replays only that quadrant's handlers, from its own version up to current, with staging scoped to the handler's quadrant so a quadrant already ahead is never down-converted. Any quadrant newer than the app ⇒ reject ({ loaded: false }, live db untouched). One-quadrant-per-handler (guard-enforced) is what makes this sound.

Split persistence — two locations, one blob per quadrant

The document and settings quadrants can be persisted to (and loaded from) two separate, independently pluggable locations. A scoped db.fromData(blob, scope) now version-upgrades only its quadrant — from that blob's own metadata stamp — and commits scoped, leaving the other quadrant intact (previously scoped loads bypassed versioning). This is sound precisely because every handler is single-quadrant, so a quadrant upgrades on its own; it's what lets a document blob and a settings blob be saved and loaded independently, in either order, non-clobbering.

The pluggable seam is the existing createStoragePersistenceService({ database, scope, storage, … }) — instantiated once per quadrant, each with its own scope and storage backend (any Storage-shaped port: localStorage, sessionStorage, a cloud/OPFS adapter). The data-lit-todo sample demonstrates it: the todos (document) and displayCompleted (settings) each auto-load/auto-save to their own localStorage key.

How load works (model A)

db.fromData(oldBlob) → reconstruct into a bare document store → the upgrader replays each quadrant's majors (quadrant-scoped staging) → the database commit path auto-normalizes to the current schema (coerce auto-convertibles; materialize additive resources; throw on a non-auto remnant — a broken migration). The final normalize targets current (the top), so it never down-converts. Reject is data, not a throw: db.fromData resolves { loaded: true } or { loaded: false, currentVersion, schemaVersions: { document, settings } } — both quadrant stamps are reported so a caller can tell WHICH quadrant is too new. A corrupt/non-numeric saved stamp reads as +Infinity and rejects, rather than silently replaying every handler.

New / changed public surface

• db.version, createVersionUpgrader(versions), DatabaseVersioning { currentVersion, handle }
• scoped db.fromData(blob, scope) version-upgrades its quadrant + commits scoped (enables split persistence); FromDataResult reject carries per-quadrant schemaVersions
• VersionEntry, assertVersioning (combined guard), assertVersionsMatchSchema, testUpgradeHandlers, foldSchemas, createStoreAtVersion, runUpgradeStep, conformStoreToSchemas
• Store.remapComponent / Store.coerceComponent — in-place schema-change helpers, namespaced on Store per the helper-placement convention (were free remapStoreComponent/coerceStoreComponent)
• storeSchemas(store), quadrantOf, isTransientSchema
• mergePatch(target, patch, deleteSentinel?) / Patch — JSON Merge Patch; delete sentinel defaults to undefined (so null is a value), pass null for strict RFC 7396

Tests

version-upgrader.test.ts (authoring showcase: guard pass + every failure mode, v0→v4 end-to-end, reject-newer as data, assertVersioning), version-upgrader.replay.test.ts (multi-major + staged-additive + mid-history + removal-with-preservation), version-upgrader.per-quadrant.test.ts (two quadrants at different metadata stamps upgrading independently, reject, no-op, full load path), database.versioning.test.ts (seam: accept/reject/upgrade/legacy/prune/heal/backstop), database.migration-replication.test.ts (upgrade→capture delta→peer replays), plus a node-level split-persistence test (database.versioning.test.ts: one source → a document blob + a settings blob → a fresh app loads each from its own location, either order, each upgraded independently and non-clobbering) and per-quadrant reject/corrupt-stamp coverage. Full data suite passing (3241); @adobe/data build 0 errors; sample data-lit-todo passing (107); monorepo typecheck + lint clean.

Note: a sample-level unit test of the split isn't included — the split mechanism is proven by the node composition test above and every operation verified in isolation on the real MainService plugin; a fully-assembled sample vitest file hit a non-reproducible test-runner stall (not a logic defect).

🤖 Generated with Claude Code