compile: EMFILE (too many open files) silently drops most pages from embedding index on large vaults

[MarceloTrenkenchu]

Summary

swarmvault compile throws EMFILE: too many open files for a large fraction of pages during the embedding-build phase, on every run, once a vault grows to ~18k pages. The failures are swallowed into a warning and printed as [swarmvault] Warning: could not read page <path> for embedding, with no error code/message surfaced — so the real cause (EMFILE) is invisible unless you patch the CLI locally to log it.

swarmvault retrieval rebuild against the same vault does not reproduce this — only the full compile pipeline does.

Environment

• @swarmvaultai/cli 3.20.0 (swarmvault --version / package.json)
• Windows 11
• Vault size at time of repro: 7,914 sources / 18,423 pages / 15,429 nodes / 65,659 edges (swarmvault doctor --json)

Steps to reproduce

1. Have a vault with a few thousand+ pages under wiki/.
2. Run swarmvault compile from the vault root.
3. Watch stderr.

Actual behavior

Stderr fills with warnings like:

[swarmvault] Warning: could not read page graph/communities/global-module-2617.md for embedding
[swarmvault] Warning: could not read page graph/communities/handler-module-2628.md for embedding
...

In our repro this printed 159,641 such warnings across the run (before we killed it), i.e. close to the total page count (18,423) repeated across ~9 internal calls to the embedding-build step.

We patched a local copy of the CLI to log the actual caught error instead of discarding it (see root cause below), and reran. The real error for every one of these is:

EMFILE: too many open files, open 'C:\Repos\wiki\graph\communities\global-module-2617.md'

For comparison, swarmvault retrieval rebuild (a narrower command against the same, quiescent vault) completes with "warnings":[] every time — no EMFILE, no missing pages. swarmvault doctor also reports a clean vault ("ok":true) both before and after a compile run that hit this, so the graph/retrieval index itself isn't corrupted — the embedding step is just silently losing most of its input.

Root cause

loadPageContents in @swarmvaultai/engine (src/embeddings.ts, compiled into dist/chunk-*.js — we found this exact function duplicated verbatim across 30+ separate esbuild chunks, apparently once per CLI entry point):

async function loadPageContents(rootDir, graph) {
  const { paths } = await loadVaultConfig(rootDir);
  const contents = new Map();
  await Promise.all(
    graph.pages.map(async (page) => {
      const absolutePath = path.join(paths.wikiDir, page.path);
      const content = await fs.readFile(absolutePath, "utf8").catch(() => {
        process.stderr.write(`[swarmvault] Warning: could not read page ${page.path} for embedding\n`);
        return "";
      });
      if (content) contents.set(page.id, content);
    })
  );
  return contents;
}

Two compounding issues:

1. No concurrency limit. graph.pages.map(...) inside Promise.all fires a fs.readFile for every page in the vault simultaneously — 18k+ concurrent opens in our case — which exceeds the OS open-file-handle ceiling (EMFILE on Windows; likely also reachable via ulimit -n on Linux/macOS for large enough vaults).
2. The actual error is discarded. .catch(() => { ...; return ""; }) drops the caught error entirely, so there's no way to distinguish "page genuinely missing/corrupt" from "OS resource exhaustion mid-run" from the CLI's own output. We only found the real cause by patching a local copy to log err.code/err.message.

Net effect: on any reasonably large vault, compile's embedding index silently loses coverage for a large, semi-random subset of pages every time it runs, without any indication in the output that it's a resource limit rather than a real per-file problem.

Suggested fix

• Throttle loadPageContents (and any other unbounded Promise.all over graph.pages) to a bounded concurrency, e.g. a small worker-pool (p-limit-style, concurrency ~16–32) instead of firing all reads at once.
• Surface the caught error's code/message in the warning text (or at least count-and-summarize distinct error codes at the end of compile), so a resource-exhaustion failure doesn't look identical to a missing-file failure.
• Since this logic is duplicated across ~30 bundled chunks, consider whether the build should be sharing one chunk instead of triggering esbuild to duplicate it per entry point — makes any future fix here easy to under-apply.

Workaround in the meantime

None available from the CLI surface — swarmvault compile --help has no flag to skip or defer the embedding phase, so compile + a separate retrieval rebuild isn't currently possible as a split. Only mitigation right now is reducing vault size or accepting partial embedding coverage per run.

[MarceloTrenkenchu]

Opened a fix: #30