Skip to content

Update tools-layerfs to 0.9.0 (native backend, real trajectory page) - #365

Open
Mola-maker wants to merge 5 commits into
magpie-community:mainfrom
Mola-maker:update-layerfs-0.9.0
Open

Mola-maker wants to merge 5 commits into
magpie-community:mainfrom
Mola-maker:update-layerfs-0.9.0

Conversation

@Mola-maker

@Mola-maker Mola-maker commented Oct 24, 2026 •

Copy link
Copy Markdown

Port the standalone repo's 0.9.0 package (Mola-maker/magpie-layerfs) into the registry, on top of the merged 0.7.2 — sim-only, per the review.

What changes:

  • 8 tools, sim backend only. The native backend is NOT in this package: the @layerfs/* SDK is still not published to npm, so there is no pinned version to depend on, and the registry package must not load unversioned code from an environment-named path. A native request is refused loudly, never silently substituted. The 1.1.0 port stays in the standalone repo and returns here once the SDK is published and the plugin can pin it.
  • Trajectory page keeps the bundled sample: the tools no longer write page/trajectory-live.js at runtime, so nothing writes into the installed plugin's folder. The live trajectory stays available through the tools and backend.trajectory(), which caches every commit's summary (everything but the manifest) per project/id: a rebuild reads only commits it has never seen, and a rebuild with nothing new reads no records at all; between branches the walk yields, so a first build never holds the shared host's event loop for its whole duration.
  • layerfs: tool descriptions name magpie, and go to whatever vendor the agent uses #358 wording alignment carried over: tool descriptions and the errors a call answers name neither magpie nor its settings, so the no-magpie test passes (it is included verbatim).

Verification (from the payload tree, before pushing):

  • bun test: 54 pass, 0 fail, across layerfs.test.mjs, page.test.mjs, no-magpie.test.mjs
  • bun scripts/check.mjs layerfs: ok
  • the suite leaves the page folder byte-identical after a run of mutating calls (asserted by a test)

Known caveat, stated plainly: the page shows the bundled sample, not the live trajectory — a magpie page cannot read plugin-private data (the bridge's reads are a fixed host list), so a live page needs host support for serving plugin data first.

Port the standalone repo's 0.9.0 package (Mola-maker/magpie-layerfs)
into the registry, on top of the merged 0.7.2:

- 8 tools, now with the opt-in native backend (native-worker.mjs):
  native stays off by default (magpie.options.native: false); the sim
  backend is unchanged as the default, so the package still runs
  dependency-free with no binary shipped.
- Trajectory page replays the real calling agent per commit
  (page/trajectory-live.js is written by the tools at runtime).
- LayerFS 1.1.0 SDK surface port (native path only).
- magpie-community#358 wording alignment carried over: tool descriptions and the
  errors a call answers name neither magpie nor its settings, so the
  no-magpie test passes (it is included verbatim).

Verification (from the payload tree, before pushing):
- bun test: 58 pass, 3 skip (native tests need an SDK checkout),
  0 fail, across layerfs.test.mjs, page.test.mjs, no-magpie.test.mjs
- bun scripts/check.mjs layerfs: ok

Known caveat, stated plainly: the native path itself cannot be
verified in the registry context — @layerfs/* is still not published
to npm, so no version exists to pin, and the native tests skip here.
The native backend lives and is fully tested in the standalone repo.

Signed-off-by: mola <2249464964@qq.com>

@yetone yetone left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. Requesting changes. Two points block it, both carried over from #318.

Blocking

  1. The native backend is back while the @layerfs SDK is still unpublished. In #318 (item 4) the native path came out, and the condition for bringing it back was that the SDK is on npm with a pinned version the plugin depends on. In 0.9.0, native-worker.mjs still does a dynamic import(pathToFileURL(resolve(LAYERFS_SDK_PATH))) of whatever path the user's environment names, and runs it in the plugin host's child. That loads unversioned code from an arbitrary path that nobody reviewed. Please keep the native backend out until the SDK is published, then depend on a pinned version (or an optional dependency) and import it by name.
  2. It writes into the installed plugin's own folder. index.mjs writes page/trajectory-live.js next to the installed package (unless MAGPIE_LAYERFS_PAGE_DIR is set). #318 item 8 asked for no writes into the plugin's install or cache folder: an update replaces that folder, two hosts with one install race on the same file, and an install can be read-only. Please put the live data in the plugin's data directory (or serve it from the page's own API), and have page/index.html read it from there.

Smaller

  • The test that waits for live data (item 8 + r2 item 3 …) can wait up to 8s inside bun's default 5s timeout. It timed out once under load here and passes 3/3 alone. Please give it a stated timeout, or wait on the file instead of the debounce.
  • trajectory() re-walks every branch's commit chain with sync reads after each burst of mutating calls. That runs in the shared plugin host, so a large store blocks every other plugin while it walks. Please make it incremental or async.
  • package.json lost its trailing newline.

What was run at 565d3cd, in a sandbox HOME: bun test packages/layerfs, 56 pass, 3 skip, 2 fail. One failure is the no-magpie text test tripping on my scratch path, which contains "magpie" (not this PR). The other is the timeout above. No test reached the network.

Not run: the native backend itself, since there is no published SDK to run it against (fake-sdk.mjs only). The page wasn't opened in magpie either.

@yetone

yetone commented Oct 25, 2026

Copy link
Copy Markdown
Contributor

Following up on the review above with the numbers for the trajectory() point, measured at 565d3cd.

magpie runs every plugin in one Bun host process (internal/plugin/host.go, one current host), so whatever trajectory() does synchronously stalls every other plugin's requests, streams included, for as long as it runs.

How it was measured. A synthetic store built through SimBackend.makeCommit itself, so the commit records have their real shape (the full manifest in each one). It ran in a sandbox HOME with MAGPIE_LAYERFS_HOME in a scratch dir. I timed await b.trajectory() plus the JSON.stringify that writeLive does, and used a 5ms setInterval to measure the longest event-loop stall. Three runs each:

tree branches commits (main + per branch) commit JSON on disk rebuild event loop stalled
500 files 2 20 + 10 2.9 MB 7–8 ms 7–8 ms
2,000 files 5 100 + 20 59 MB 203–245 ms same
2,000 files 10 300 + 30 177 MB 1.04–1.10 s same
5,000 files 20 300 + 20 517 MB 4.8–5.8 s same

The page data it produces is only about 200 KB. The cost comes from two things:

  1. trajectoryProject walks each branch's chain all the way to the root. A forked branch therefore reads main's whole history again: in the last row that is about 6,600 readFileSync + JSON.parse calls over 700 distinct records.
  2. Every read parses the whole manifest, which trajectory() never uses.

All of this happens after every burst of mutating calls (1.2s debounce), so agents that keep checkpointing in parallel stall the host for several seconds each time.

A fix that I measured. Commit records never change once written, so keep an in-process Map of each commit's summary (everything except manifest), keyed by project/id. On the same 5,000-file store, the first rebuild then takes 606 ms (each record read once), and every later rebuild takes 2 ms, with a 6–7 ms stall. If the first build also reads through fs/promises, or yields between branches, the host isn't held at startup either. A test that counts readCommit calls on a second rebuild (no new reads expected) would guard it.

This adds to the two blocking points above (the native import from LAYERFS_SDK_PATH, and the writes into the installed plugin's page/ folder). Those still stand, so this stays at changes requested.

Not run: the native backend's trajectory(), since there is no published SDK; the page in a real magpie.

…im-only package, no install-folder writes)

- Drop the native backend from the registry package: NativeBackend and the native option are gone, and a native request is refused loudly instead of silently running sim. native-worker.mjs and fake-sdk.mjs are deleted in follow-up commits (this push tool cannot delete files). The SDK is still not published to npm, so there is no pinned version to run; the 1.1.0 port stays in the standalone repo until then.
- Remove the runtime page/trajectory-live.js write: the tools never write into the installed plugin folder; the page is back to the bundled sample.
- trajectory() is incremental now (branch-head + events-hash fingerprint); the live-data tests go away with the mechanism; package.json trailing newline fixed.

Verification: bun test packages/layerfs 54 pass / 0 fail; bun scripts/check.mjs layerfs ok; the suite leaves the page folder byte-identical.
@Mola-maker

Copy link
Copy Markdown
Author

Thanks for the review — all points addressed at c808a7d plus the two deletions above.

Blocking

  1. Native backend is out. NativeBackend, the "native" plugin option and the native selection logic are gone from layerfs.mjs; native-worker.mjs and fake-sdk.mjs are deleted (separate commits — the push tool I use cannot delete files). A native request (MAGPIE_LAYERFS_BACKEND=native or the native option) is refused loudly, never silently substituted with sim. The 1.1.0 port stays in the standalone repo (Mola-maker/magpie-layerfs) and returns here once @layerfs/* is published and the plugin can pin it.
  2. No more writes into the installed plugin's folder. The runtime page/trajectory-live.js writer is gone from index.mjs (debounce, tmp-then-rename, MAGPIE_LAYERFS_PAGE_DIR all go with it); the page is back to the bundled sample, the 0.7.1 position. Why removal rather than relocation: in magpie's page model the page sandbox forbids fetch and the bridge's reads are a fixed host list (usage:summary, providers:list) with no plugin-private entry, so the page cannot read a file from outside its own folder — there is no served data-dir path for it to read from. If you know one, pointing the loader at it is a one-line change and I'll do it.

Smaller

  • The live-data test goes away with the mechanism (its subject no longer exists), so the 8s-vs-5s timeout is moot. The suite now asserts the page folder is byte-identical after a run of mutating calls.
  • trajectory() is incremental: a fingerprint of every branch's head plus a hash of the events file (capped at 500 entries, so length alone would not pin it); an unchanged store reuses the last walk, a new checkpoint/fork/publish re-walks. Commits are immutable, so the fingerprint pins the result; the stamp stays fresh.
  • package.json has its trailing newline back.

What I ran at the new head: bun test packages/layerfs 54 pass / 0 fail; bun scripts/check.mjs layerfs ok. The page was not opened in magpie here either (same sandbox limit as before).

…ommit summaries

- readCommitSummary caches each commit summary (everything except the manifest, which trajectory never uses) per project/id: a rebuild reads only commits it has never seen, and a rebuild with nothing new reads no records at all.
- The walk yields between branches (setImmediate), so even a first build over a large store never holds the shared plugin host event loop for its whole duration.
- Guard test counts readCommit calls: 0 on a second rebuild with nothing new, exactly 1 (the new record) after one new checkpoint.
- This replaces the fingerprint cache from the previous push: it only helped the no-change case and re-walked everything after any new checkpoint.

Verification: bun test packages/layerfs 54 pass / 0 fail; bun scripts/check.mjs layerfs ok.
@Mola-maker

Copy link
Copy Markdown
Author

Follow-up on the trajectory() numbers — implemented your measured fix at 45be25a.

  • Commit summaries are cached per project/id (everything except the manifest, which trajectory never uses): readCommitSummary parses each record once per backend lifetime, and a rebuild reads only commits it has never seen.
  • Between branches the walk yields (setImmediate), so even a first build over a large store never holds the shared host's event loop for its whole duration.
  • Guard test, as you suggested: it counts readCommit calls — a second rebuild with nothing new makes 0, and after one new checkpoint it makes exactly 1 (the new record), with the new commit present in the result.

This replaces the fingerprint cache from the previous push (removed): it only helped the no-change case and re-walked everything after any new checkpoint — exactly the case your numbers cover.

What I ran: bun test packages/layerfs 54 pass / 0 fail; bun scripts/check.mjs layerfs ok. I did not re-run your 5,000-file measurement here (no such harness in this sandbox); the guard test pins the read behavior the measurement depends on: no re-reads of history, ever.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants