feat(data): injectable memory allocator for the new ECS (wasm + SharedArrayBuffer) - #190
Open
krisnye wants to merge 2 commits into
Open
feat(data): injectable memory allocator for the new ECS (wasm + SharedArrayBuffer)#190krisnye wants to merge 2 commits into
krisnye wants to merge 2 commits into
Conversation
…ble SharedArrayBuffer arenas) Thread an optional MemoryAllocator through createDatabase → store → core → archetype → table → typed buffers so numeric component storage can live in a shareable arena. Reuses the legacy cache/ MemoryAllocator interface; adds a growable-SharedArrayBuffer arena allocator and rebuilds the wasm allocator on a shared free-list with a WeakMap so refresh() recovers detached views after grow. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… arena safety Adds TypedBuffer.dispose() (unsubscribe from needsRefresh + release the arena block) and wires archetype.fromData to dispose replaced columns, closing the subscription/arena leaks. Arena allocator now grows geometrically, ignores foreign views on release (no free-list corruption), tracks zeroed blocks to skip redundant fill, keeps the free list sorted on insert, supports a reserved wasm base offset, aligns SAB bounds, and throws an actionable wasm-exhaustion error. Each fix has red/green unit coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collaborator
Author
|
Addressed the fable review findings in
Notes:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds the ability to inject a custom memory allocator into the new ECS
Database/Storefactory, with off-the-shelf implementations that allocate numeric component storage into WebAssembly memory or a growableSharedArrayBuffer. This mirrors the precedent already present inold-ecs(createECS/core-ecsaccept{ allocator?: MemoryAllocator }), which the new typed-buffer-backed ECS previously lacked.The point of the SharedArrayBuffer path: every numeric column becomes a fixed
(offset, length)view into one shared backing buffer, so in principle that buffer can be posted to a worker once and its numeric components read/written across threads without copying — a foundation for future parallel system scheduling.What's included
cache/memory-allocator.tsMemoryAllocatorinterface (allocate/refresh/release/needsRefresh) rather than inventing a parallel abstraction.createSharedArrayBufferAllocator()(growable-SAB arena) +isSharedArrayBufferAllocatorSupported();createSimpleMemoryAllocator(default, per-column buffers) andcreateWasmMemoryAllocatorretained.TypedArrayreportsbyteOffset/lengthas0, so the oldrefreshcould not recover a view's location aftermemory.grow. The arena now keeps aWeakMap<view, {offset,length,ctor}>sorefreshrebuilds detached views correctly, and zero-fills reused blocks to match the always-zeroed semantics of the simple allocator.allocatorthroughcreateDatabase(plugin, { allocator })→createStore(schema, { allocator })→createCore→createArchetype→createTable→createTypedBuffer→ number / struct / boolean / enum buffers.createDataView32is now offset-aware so struct columns can be arena sub-regions.SharedArrayBuffer/ArrayBuffer), and the whole monorepo typechecks.Test evidence
New unit tests, all green (full
@adobe/datasuite: 3293 passed):cache/memory-allocator.test.ts— simple/wasm/SAB allocators, shared-arena contract, detached-view refresh after grow, zeroing of reused blocks, arena exhaustion, feature detection.typed-buffer/typed-buffer-allocator.test.ts— each storage buffer preserves data across a reallocating grow under wasm/SAB allocators; columns share one backing buffer.ecs/store/store-allocator.test.ts—createStore/createDatabaseroute component columns into the injected arena; correctness after growth past initial row capacity; default path stays per-column.Repo-wide
pnpm run lintandpnpm -r run typecheckboth pass.Known limitations / follow-ups
Subscription lifecycle: each column subscribes toResolved inneedsRefreshand never unsubscribes...e2385272: addedTypedBuffer.dispose()(unsubscribe + release the arena block), wired througharchetype.fromDatato dispose replaced columns. See the review-response comment below..copy()(detached snapshots), the persistence codec (register-typed-buffer-codecs), andremapArchetypeColumnallocate with the default allocator, so post-load / post-migration columns are not placed in an injected arena. (Still out of scope; adispose()lifecycle now exists to build on.)Related PRs
None.