convert.py without --reclaim writes a DeepSeek-V4.1 container that can never be opened: build_engram_meta() is only called on the --reclaim path #71

A DeepSeek-V4.1-Flash conversion that reports success produces a container which
every runtime subcommand rejects with open: malformed container. The container
is not damaged — the Engram index is simply never written, because
build_engram_meta() is called on one of the two trunk paths in convert.py and
not the other. The default path is the one missing it.

Cost of the failure: ~6.5 h of conversion and a 320 GB artifact that cannot be
loaded, with no diagnostic naming what is wrong.

Symptom

$ waste info /ds41.waste        -> open: malformed container   (exit 1, stderr)
$ waste run  /ds41.waste "..."  -> open: malformed container
$ waste tokenize /ds41.waste x  -> open: malformed container
$ waste plan /ds41.waste        -> WORKS
      FLOOR                      4.86 GB
      one token's working set     2.97 GB
      fully resident           195.00 GB

waste plan parses the manifest and returns correct DS41 numbers, so the
manifest is fine. Only waste_open() fails.

Why there is no diagnostic

engram_open() is the one -2 in the load path that prints nothing —
src/model.c:1964-1966:

snprintf(path, sizeof path, "%s/engram.json", dir);
char *es = slurp(path, NULL);
if (!es) return -2;          /* silent */

Every other WASTE_E_FORMAT site (model.c:2160, 2186, 2194, 2218, 2231,
validate_text_tensors/bad_tensor) fprintfs what failed. Since DS41 is the
only architecture with engram_n != 0, this is also the only arch that can fail
this way — and it fails with a bare five-word message.

Root cause

tools/convert.py has two trunk paths. Only the first writes the Engram
metadata, and it writes it twice:

# tools/convert.py:1619-1623  (the --reclaim path)
        if ds41:
            engram = build_engram(st, args.out, cfg, args.engram_bits)
            build_engram_meta(args.src, cfg, args.out)
            build_engram_meta(args.src, cfg, args.out)   # <- called twice
        reclaim(debt, args.reclaim, ShardDebt.TRUNK, "trunk")

# tools/convert.py:1990-1991  (the default path)
        if ds41:
            engram = build_engram(st, args.out, cfg, args.engram_bits)
            # <- build_engram_meta() is never called

build_engram() writes the tables (engram-L1.bin, engram-L14.bin, 110.6 GB
here). build_engram_meta() writes engram.json and engram-tokmap.bin, which
engram_open() requires. Without --reclaim, the tables are written and the
index is not.

The container is otherwise sound

• 48/48 source shards fetched and size-verified against the remote with
  tools/fetch_weights.sh --check.
• 40 experts-L*.bin, trunk.bin 5.2 GB, codebooks.bin, tokenizer.model,
  specials.json, vision.json, manifest.json all present.
• Engram tables byte-exact against the manifest: engram-L1 expected
  55,296,888,192 B / actual 55,296,888,192 B; engram-L14 expected
  55,298,402,208 B / actual 55,298,402,208 B.
• tools/verify_container.py --container reads it:
  container: VQ3R, 360 codebooks, layers ['0'..'39'].
• Only engram.json and engram-tokmap.bin are absent.

Reproduce

tools/fetch_weights.sh --repo deepseek-ai/DeepSeek-V4.1-Flash --dest SRC
python tools/convert.py --src SRC --out ds41.waste --jobs 8   # no --reclaim
waste info ds41.waste        # -> open: malformed container
ls ds41.waste/engram.json    # -> No such file or directory

Suggested fix

Call build_engram_meta() on the path at convert.py:1991 as well, and drop the
duplicate call at line 1622.

Two smaller things noticed while tracing it, both optional:

1. engram_open() could say which file is missing. As written, the only
   architecture that can hit this path reports nothing, which is what made a
   6.5 h conversion expensive to diagnose.
2. ds41_engram.build(src, cfg, out_dir) reads engram_layer_ids from the cfg
   it is handed and does not unwrap text_config, while the DS41 release nests
   those keys under text_config. Callers must pass the inner dict; passing the
   outer one makes build() return None and write nothing, silently.

Workaround for an already-converted container

No re-conversion needed if the source weights are still on disk (~3 minutes):

import sys, json, os
sys.path.insert(0, "tools")
import ds41_engram
raw = json.load(open("SRC/config.json"))
cfg = raw.get("text_config") or raw          # see note 2 above
ds41_engram.build("SRC", cfg, "ds41.waste")

This writes engram.json (1942 B) and engram-tokmap.bin (517 KB). Its own
check passed — the compressed token map came out 99092 ids, matching the
release's stated engram_compressed_vocab_size. waste info then reports
552.37 B total, 16.62 B active/token and the model generates normally.

Environment

• WARP 5a5f7ff2085bbd56124f3582fa38e42f6776de91 (2026-09-16), the tip of main
  at the time of the run.
• WASTE 0.8.0 (container v0, backend AVX2, crc32 slice8, x86_64), plain make.
• AMD EPYC 7713, 64 physical cores / 128 threads, 1007 GB RAM, Linux, rented
  instance; container and staging on local NVMe.
• deepseek-ai/DeepSeek-V4.1-Flash, 475 GB staged, 320 GB container.

Measurements after the workaround, in case they are useful

64 decode tokens after a 12-token prompt, single run per arm, same prompt
throughout, ds41.waste on local NVMe:

Two things here may or may not be interesting to you:

• Throughput falls as the expert cache grows, monotonically from 9G to 46G,
  while the hit rate rises 71% -> 80%. The same shape appeared on GLM-5.3-Flash
  on this machine (3.07 tok/s at 16G / 75% hit, down to 2.09 at 130G / 98% hit).
• The default --threads 0 (one per logical CPU = 128 here) is roughly half the
  throughput of --threads 8-16 on a 64-physical-core part. On GLM the same
  default cost 2.3x (0.90 vs 2.07 tok/s).

For calibration against the README's 3.77 tok/s on the M5 Pro: this x86 part
reaches 1.31, whereas GLM-5.3-Flash on the same instance reached 3.07 against a
published 3.32 — so DS41 specifically loses far more on x86 than GLM does.
#70's i8mm trunk kernel being ARM-only would account for part of that; the rest
may be Engram's 144-byte random reads into two 55 GB tables. I have not
separated the two and am not claiming a cause.