perf: remaining performance ledger after the 2026-07 hotspot round

Status ledger for performance work after the #28 round and the cocoon #123 upgrade verification (bare-metal testbed, 2026-07-12). Numbers of record: warm claim 0.2ms, sequential clone claim 32-37ms, burst16 wall 287ms, cold ~320ms, fork(4) 185ms, exec RTT p50 0.27ms, fs_pull ~602 MiB/s, fs_push steady ~343 MiB/s.

Open — worth doing, needs a product decision

1. Wire protocol v2: multiplexed RPCs + binary bulk frames. The one remaining order-of-magnitude lever in this repo. Today every RPC pays a triple connection setup (SDK TCP dial + HTTP upgrade → relay hijack → fresh vsock dial): the fixed cost is the bulk of the 0.27ms RTT. A pre-dialed spare connection was measured and rejected (p50 0.20/0.48/0.36/0.35/0.39ms across 5 rounds vs a stable 0.27-0.31ms baseline; p90/p99 consistently worse — the background dial contends with the foreground RPC), so the real fix is multiplexing over one long-lived conn. Bulk data rides base64-in-JSON (+33% wire, codec CPU); binary framing should lift fs_pull from ~602 toward 800+ MiB/s. Scope: SDK + relay + silkd (Rust) — cross-repo protocol change, staleness/idempotency semantics needed for reconnects. Do both together or not at all.

2. Refill admission re-eval (refillSem=4 / per-golden). Un-blocked by cocoon #123 (sibling clones no longer serialize on the source flock). Claim-miss provisions are unbounded (verified: ClaimProvision calls provision directly), but refills and forks share the node-wide refillSem=4 — with cocoon now parallelizing, that cap may bound pool-drain recovery. One day of A/B (4 vs 8 vs per-golden admission) once a cocoon release carrying #123 is deployed.

Deferred — reopen triggers documented

item	trigger
claimStore full-map snapshot+rewrite per mutation	>120 live claims per node
idleOnce/archiveOnce O(n) scans under the manager mutex	same
checkpoint TTL sweep re-lists the full store per tick	checkpoint count grows 10x
fastBulk decode buffer per chunk (~600MB/s GC garbage at max pull)	GC pressure shows in sustained-transfer profiles; needs a payload-ownership contract on Recv
resyncEgress serial EnsureLock at restart	many egress claims + restart-time SLA
probeReady vsockOf fallback polls `vm list` (subprocess) at 100ms	the fallback branch shows up in practice (it is rare by design)

Rejected with data — do not re-litigate without new evidence

SDK pre-dialed spare connection (H-4): see above.
fs_push chunk 256K→1M: steady-state identical (~343 MiB/s) across 3 rounds.
nft Lock netlink-conn merge: conns are lazy in nftables v0.3.0; the two batch round trips cannot merge safely without kernel destroy-table support.
silkd conn scanner-buffer pooling: unsafe — Close may race a blocked Recv from another goroutine (pty/port semantics).
encodeTagged rework: already 243ns / 2 allocs per control frame.

Physical boundaries — not this repo's levers

Cold boot (~320ms, guest kernel/init), fork (~185ms, ~512MB memory snapshot I/O), CH burst restore bandwidth, and the cocoon-side 16-way residual serializer (index flock + journal + per-clone CLI process startup; per-clone p50 85ms vs 27ms serial — see cocoon#123's follow-up notes). The CLI-process model itself (one fork+exec of the cocoon binary per engine operation) is the deepest cross-repo lever here (daemon/library mode).

Related: #29 (correctness, found in the same round).

Ledger update, 2026-07-13:

Item 1 (wire protocol v2) now has a full design: #34 — framing, per-stream credit flow control, negotiation/compat matrix, reconnect semantics, rejected alternatives, acceptance targets (exec RTT p50 ≤0.15ms, fs_pull ≥800 MiB/s). The fastBulk decode-buffer deferred item folds into it (v2 DATA frames carry the payload-ownership contract).

Item 2 (refill admission re-eval) is un-blocked and has its harness + baseline. make bench gained a burst stage (BURST_N concurrent clone claims, runs last) and a warm refill-recovery stage (6615de0). Post-#33 baseline @88fda81, bare metal, two rounds:

sequential tiers unchanged: warm 0.2–0.3 ms p50, clone 33.2–39.0 ms p50, cold 354–398 ms p50, exec RTT p50 0.18–0.20 ms, fs_pull 594–612 MiB/s — feat(config): state-ownership model — persist API pools, cluster fan-out, config-drift detection #33 confirmed zero hot-path cost on hardware.
burst 16× concurrent clones: per-claim p50 126.5/149.4 ms (vs ~35 ms sequential — the contention is real and now measured in-repo), wall 331/308 ms.
warm refill recovery (0→6): 1638 ms in both rounds, identical to the millisecond — recovery is refill-cadence-quantized, not restore-bound. For the refillSem 4-vs-8-vs-per-golden A/B, measure/tune the refill tick first; the semaphore is unlikely to be the binding constraint at this pool depth.

Environment note: host cocoon moved master-be8653e → master-acfc815 between the pre-burst and burst rounds (parallel cocoon work on the bench host); all four rounds' sequential tiers agree, so the comparison stands.

Precision note on the refill-recovery number, after reading the mechanism (pool.go:35 refillInterval=2s; refillOnce call sites are startup, SetPools, and the 2s ticker only — a claim draining the pool does not kick refill; refillOne self-chains its semaphore slot, refill.go:63-67):

recovery = tick-phase wait (0–2000 ms, dominant) + ~200–400 ms of actual work (6 clones through refillSem=4, self-chained) — and the reported 1638 ms is additionally quantized by the harness's 200 ms poll loop (1638 ≈ 8 × 204.75 ms poll iterations). The round-to-round millisecond identity is the poll quantization plus the deterministic script timeline (same tick phase at drain time), not a system constant. True recovery was in (1433, 1638] both rounds.

Refined implication for item 2: at warm=6, refillSem 4→8 changes only the ~300 ms work term — invisible under the 2 s tick wait, so an A/B at this depth measures nothing. The A/B belongs on a deep pool (target ≳50, work term in seconds); if drain-recovery latency itself ever matters, the lever is an event-driven refill kick (or a shorter tick), not the semaphore. Note the 2 s tick is deliberate: refill is fully off the claim path (a miss provisions directly — the burst row measures exactly that path), so recovery speed only affects warm-hit ratio in the ~2 s after a drain.

fs_push moved: steady-state 702/741/762 MiB/s (pushbench, 128 MiB ×3, bare metal, new-silkd images @205d3f3) vs the ~343 MiB/s of record. The guest-side change is the inbound base64 decode feeding serde's borrowed slice straight to the decoder (was: an intermediate String per data frame, ~341 KB per 256 KB chunk, on musl); a rustc toolchain bump rides in the same image, so the split between the two contributors is unattributed — codec-level A/B on glibc measured only +6.5%, meaning musl allocation was most of the cost. Full round context in docs/benchmarks.md 2026-07-14 entry (e2e smoke pass, all other tiers in band). Item 1 (protocol v2, #34) remains the lever that deletes base64 entirely; its fs_pull ≥800 MiB/s target may deserve an fs_push target next to it now that push cleared 700 without protocol changes.

Perf ledger update (bare-metal testbed, 2026-07-20, cocoon master-24bd46b + CH fork dev, 512M none-lane golden):

B=64 concurrent clone, restore-mode A/B (drop_caches before each round; exec spot-checks all passing):

mode	ok	per-clone p50	resident/VM	single-VM smaps
default (eager Copy)	63/64*	2311ms	331MB	Pss_Anon 337MB — full per-VM copy, zero sharing
mmap → CH `CopyOnWrite`	64/64	1060ms (wall 1915ms)	43MB	Anon 41MB + shared page-cache file pages
ondemand (uffd)	64/64	2083ms	514MB	Anon 525MB — UFFDIO_COPY materializes everything incl. sparse holes

* one slot lost to an instrumentation accident (strace on the CLI wedges Go binaries; don't).

Conclusions for the ledger:

CopyOnWrite wins both axes at concurrency: 2.2× latency, 7.7× density. cocoon still defaults to eager copy (restore-mode ""|copy → CH default) — switching the clone default to CoW is the single biggest storm lever currently unwired.
ondemand is a density trap for pooled clones (worst of the three at B=64) — keep it for single cold restores only.
Caveat to wire alongside the default switch: CH silently falls back CoW→eager on hugepage/shared/NUMA-bound regions, and cocoon auto-enables HugePages whenever host nr_hugepages>0 — the fallback is invisible today (warn in CH stderr only). Needs surfacing as an error or metric.
Even on CoW, B=64 per-clone p50 ≈ 1.06s — the residual is CLI-per-op + CH process launch + API sequence, i.e. the daemon/library-mode item already on this ledger.
Cross-host note: DirectClone hardlinks the memory file (page-cache sharing intact); the stream-clone path materializes a full per-clone copy — C1M fleet design should land one golden per host and DirectClone-fan-out locally.

Same round: egress-lane concurrent clones now clean 16/16→64/64 after cocoon#142 (tap-attach race), egress ≈ none +10%; FC lane unblocked by FC v1.16.1 (cocoon#140) and upstream main (1.17.0-dev) also passes the vsock repro.

T1 closed (bare-metal testbed, FC none-lane per lane assignment, cocoon master-24bd46b, FC 1.17.0-dev main build, 512M guests).

Storm scaling (100% success + exec spot-checks at every B):

B	wall	p50	throughput
16	183ms	108ms	87/s
32	509ms	181ms	63/s
64	1063ms	609ms	60/s
128	6033ms	3048ms	21/s
256	18059ms	13413ms	14/s

Density ceiling: 576 × 512M sandboxes on the 60GB host — guard-stopped on MemAvailable (>3GB reserve), zero clone failures, zero OOM, exec verified per wave and at final census. Marginal footprint ~107MB/VM under pressure (84MB/VM at B=64). Nominal RSS sum 80.8GB > physical RAM — MAP_PRIVATE page-cache sharing doing its job. Extrapolation: ~9k+ VMs on a 1TB box → the T2 8-12k/box target is realistic on hardware, and ~110 × 1TB boxes for 1M at 512M guests.

Latency attribution (why p50 grows with B):

per-clone CPU: 12ms (/usr/bin/time -v across a B=64 wave; 64 clones sum 0.76s user+sys) — clones spend ~98% of wall waiting;
CLI floors at 64-way concurrency: version p50 12ms, vm list p50 12ms — process startup/config/DB-read are negligible;
sequential single clone: ~31ms;
--from-dir (bypasses snapshot store) B=64 p50 473ms vs store-path 609ms — store layer ≈ 20-25% of the wait;
superlinear wall growth B64→B256 (5.7×, 17× for 2×, 4× work) + wave slowdown at high resident-N ⇒ the dominant wait is the shared-root serialization (2ms-poll flocks + whole-file JSON DB rewrites that grow with resident count) with O(N) per-op character. /proc/locks sampling shows zero waiters — expected, the locks are non-blocking polls, so neither CPU nor lock-wait counters see it.
isolated-root A/B leg was blocked by the snapshot-metadata path-trust hardening (base-layer path outside --root-dir is rejected — correct behavior, bad for this experiment).

Ledger actions:

daemon/library mode: priority UP — the O(N) shared-state cost collapses storm throughput exactly where T2 needs 8-12k sustained; prerequisite measurement = a lock/DB-timing debug build to name the exact serialized sections.
journal delta-writes: unchanged MUST (same O(N) family).
C1M harness note: per-host golden dir + vm clone --from-dir is both the page-cache-sharing design AND a free −25% storm latency (skips the store layer entirely).
FC lane needs no restore-mode wiring (File backend natively MAP_PRIVATE lazy) — only the CH egress lane needs restore_mode: mmap in deployment config (guard: cocoon#144).

Phase decomposition of the storm superlinearity (B=64 ×3 consecutive, no cleanup between — resident N = 0/64/128; per-clone anchors: exec-stamp → ops.lock ctime → FC /proc starttime → api.sock ctime → vsock.uds ctime → exit-stamp):

Fixed B=64, only resident N grows: wall 1009 → 3653 → 4914ms. The B128/256 superlinearity is the resident-count effect, not concurrency per se.

Round-3 phase split (64/64 complete samples):

phase	p50	p90
P1 cli+locks+db-read→runDir	99ms	168ms
P2 storage-prep + record insert + FC spawn	2347ms	3902ms
P3 FC api up	249ms	252ms
P4 snapshot load (the actual restore)	14ms	22ms
P5 resume + reseed + record finalize	867ms	2013ms

Named culprit: firecracker/db/vms.json whole-file RMW under the polled exclusive flock, plus a full .prev copy on every write (measured 113KB → 226KB → 339KB at ~1.8KB/record, i.e. 3×339KB of IO per write at N=192), at least twice per clone (insert in P2, finalize in P5). Cost per RMW ∝ N ⇒ storm cost O(N·B).

Fix ladder for T2: (1) cocoon#132 — drop the second RMW (halves lock-held IO); (2) delta/segmented index instead of whole-file+.prev rewrites (kills O(N)); (3) daemon/library mode with in-memory index (removes the file RMW entirely). The restore path itself (14ms) and CLI startup (12ms floors) need nothing.

@n

T1 re-run on the sqlite meta backend (bare-metal testbed; cocoon master-5714e9e with meta P0–P3 landed, CH dev-7901a518, FC v1.16.1; same binary A/B, json round then offline meta convert then sqlite round; all storms 100% success + exec spot-checks):

metric	json (same bin)	sqlite	Δ
3×B64 ladder r1 (N=0)	1128ms	966ms	−14%
ladder r2 (N=64)	3533ms	2238ms	−37%
ladder r3 (N=128)	4032ms	3216ms	−20%
pre-spawn phase p50 @n=128	1795ms	457ms	−75%
B=128 wall	4716ms	2886ms	−39%
B=256 wall	12440ms	9553ms	−23%
CH-egress B64 (mmap)	690ms	717ms	flat (not metadata-bound)

Also validated on hardware: P0 non-regression (json engine behind the meta API reproduces the legacy ladder within noise: 1128/3533/4032 vs legacy 1009–1189/3533–3874/4016–4914) and the conversion procedure behaved per design (both json generations renamed aside with timestamps across all namespaces; records served intact from sqlite).

Design acceptance anchors, honestly scored: r1 ≤1.5s ✅ (966ms); later rounds ≤1.3×r1 ❌ (2.3×/3.3×). Cause is not a metadata shortfall but a bottleneck relocation: the restore(load) phase segment (FC api-up → vsock.uds), historically a constant ~15ms, now carries p50 599ms / p90 1622ms under sqlite. The metadata queue that used to stagger arrivals (pre-spawn 1795ms) is gone, so all 64 clones hit the next serializer nearly simultaneously — the drive-redirect window (per-golden clone-locks + bind-mount setup, kernel-global work). Confirmed no meta-store writes exist in that window (redirect uses dir flocks only).

Next ledger target therefore shifts: the drive-redirect critical section is now the dominant storm serializer at B≥64. Candidates: per-clone mount-ns cost, clone-locks hold time, symlink-fallback retirement (the TODO already in clone.go). Instrumentation of holdRedirectDirs is the next measurement step before any fix.

Attribution of the post-sqlite restore(load) p90 → it's cocoon, not FC, not a control-plane daemon. Hardware-measured (bare-metal testbed, sqlite backend, FC none-lane 512M hibernate golden):

restore(load) phase (api.sock→vsock.uds) scales purely with concurrency:

B	p50	p90	max
1	5ms	5ms	5ms
16	23ms	41ms	59ms
64	176ms	539ms	904ms

Three independent signals isolate the serializer:

Not FC: uncontended (B=1) snapshot/load is 5ms; FC's own restore work is not the cost.
Not the kernel mount lock (first suspect): a microbench of 64 concurrent Unshare(CLONE_NEWNS)+Mount(MS_REC|MS_PRIVATE,/) completes in 2ms (0.1ms/op), and all 64 FC processes are in the HOST mount namespace — so the lock-free bind fast path never ran.
cocoon's per-golden source-disk flock: flock waiters during the B=64 storm peak at p90 29 / max 51.

Causal chain, verified in source + on disk: a hibernate golden's baked source COW path (…/run/firecracker/<srcVM>/cow.raw) is MISSING once the source VM is removed post-hibernate → bindableRedirects requires a regular file, returns nil → the clone falls out of the lock-free bind path into the symlink fallback, which wraps the whole loadSnapshotFC in withSourceWritableDisksLocked (a flock on the source golden's writable disks). 64 clones of one golden share those paths → one lock → full serialization.

This is exactly what the removal of the O(N) metadata serializer (json→sqlite) exposed: pre-spawn dropped −75%, un-staggering arrivals so they all hit this next lock at once.

Fix is already scoped as a TODO in hypervisor/firecracker/clone.go:150: make dead-source (hibernate golden) clones eligible for the lock-free bind path by binding the clone's fresh disk over a placeholder mountpoint at the missing source path, instead of the locked symlink fallback. That removes the per-golden serializer from the C1M clone-from-hibernate-golden workload — the dominant B≥64 storm cost now that metadata is off the critical path.

@n

Corrected T1 baseline: both serializers removed (sqlite metadata + keep-source bind path). Same bin (cocoon master-5714e9e, FC v1.16.1), FC none-lane 512M, 100% success + exec-verified. Keep-source = golden's source VM kept stopped (not rm'd), so its baked COW path exists and clones take the lock-free bind redirect path instead of the symlink+flock fallback (attribution proven in the prior comment).

metric	legacy json	sqlite (rm src)	sqlite + keep-src
3×B64 ladder r1 (N=0)	1128ms	966ms	813ms
r2 (N=64)	3533ms	2238ms	1636ms
r3 (N=128)	4032ms	3216ms	2083ms
restore(load) p50 @n=128	15ms	599ms	49ms
restore(load) p90	—	1622ms	74ms
B=128 wall	4716ms	2886ms	2097ms
B=256 wall	12440ms	9553ms	5619ms (14/s→46/s)

The two fixes are additive (B256 12.4→9.6→5.6s). restore(load) collapses 22× (p90 1622→74ms) — confirming both the attribution and that keep-source is a valid, zero-code fix for the C1M clone-from-hibernate-golden workload. Per CMGS: keep-source is the interim answer; the placeholder-mountpoint TODO stays parked until FC upstream lands dynamic drive replacement (PR #5774), at which point the redirect dance retires entirely.

Next bottleneck is now CLI-per-op, not storage: round-3 phase split with both serializers gone shows the two largest segments are pre-spawn (p50 509ms: CLI process start + dir/COW prep) and post-uds (p50 461ms: resume + reseed + finalize), while restore(load) is 49ms. These are the per-process cost of 64 independent cocoon CLI invocations — exactly the daemon/library-mode lever (T0, this ledger). CH-egress mmap flat (~760ms, not storage-bound). The layered plan holds: each serializer removed surfaces the next — metadata → drive-redirect → CLI-per-op.

Double-checked the two suspicious T1 numbers — both explained with hardware data.

(1) restore(load) 15ms (json) → 599ms (sqlite) → 49ms (keep-src): confirmed un-staggering of a SERIAL bottleneck chain, not an anomaly. The drive-redirect flock is taken between api.sock and vsock.uds — inside the restore(load) window. sqlite B64 (rm-src) measured: api.sock arrival spread only 279ms (clones bunched) + /proc/locks waiters peak 53 (≈ the full 64-way queue on that lock) → restore(load) p50 175 / p90 663ms. In the json round, pre-spawn was 1795ms (metadata O(N)) which SPREAD arrivals, so each clone met an uncontended drive-redirect lock → 15ms. Removing the upstream metadata serializer (sqlite) just moved the queue onto the downstream drive-redirect lock. Keep-source (bind path) removes that lock → restore(load) 31-49ms. The 40× swing is the signature of two serializers in series, not a measurement error.

(2) B=256 still 5.6s with both serializers gone: it's CPU saturation of the 16-core testbed, NOT a lock/metadata/FC issue. vmstat during the B=256 storm: peak run-queue r=280, min idle=0% on 16 cores = 17.5× oversubscription. Phase split at B=256: restore(load) only 31ms (bind path working), post-uds 285ms — but pre-spawn p50 1503ms / p90 4099ms dominates, and it is now pure CPU contention (256 Go runtimes + 256 FC launches competing for 16 cores). per-clone p50 2070ms is mostly CPU-scheduling wait. This is the box being too small for B=256, not a fixable serializer.

Reframed conclusion: on 16 cores the sane per-box concurrency is ~B=64 (4× oversub, 813ms-2083ms); B=256 (17.5× oversub) is past this box's CPU wall — exactly why T2 rents a larger box. The CLI-per-op model adds real overhead (per-process Go-runtime startup ×N, sqlite cross-process WAL-writer contention on reserve+finalize = 2 writes/clone) that daemon/library mode would cut, but at B=256-on-16-cores raw core count dominates and no daemon manufactures cores. Corrected next steps: (a) daemon mode still worthwhile for per-process overhead + writer batching, measured at a concurrency the box can actually run; (b) T2 on a many-core box to find the real per-box ceiling; (c) the CLI floor is a per-process cost, so C1M's fan-out unit should be one long-lived agent per box, not N CLI forks.

B64 best-case spec (sqlite + keep-src, clean host per iteration, 4 iterations) — plus a measurement correction that reshapes the phase tables.

iter	wall	per-clone p50/p90/max	rate
1	875ms	431/703/861ms	73/s
2	821ms	415/649/798ms	77/s
3	909ms	436/732/901ms	70/s
4	884ms	427/706/878ms	72/s

All 64/64, and the fleet is genuinely usable: all-64 concurrent exec round-trips complete in 199ms wall (per-exec p50 75ms / p90 111ms). So one 16-core box turns 64 cold sandboxes into exec-ready in ~1.0s end-to-end.

Correction: the flat ~250ms 'fc-api-up' segment in every phase table was a measurement artifact, not FC. Probes: raw FC exec→api.sock = 1-2ms (PTY-wrapped identical); iter-4 pre-spawn showed p50 −8ms (negative). Root cause: phase scripts estimated FC start as btime + jiffies/100, but /proc/stat btime has whole-second resolution; the truncated fraction on this boot is 249ms (measured directly: (now−btime)−uptime), a constant that understated pre-spawn and inflated fc-api-up by exactly that amount. All cross-round deltas remain valid (same boot, same constant); only the segment attribution shifts.

Corrected B64 per-clone reality (clean host): pre-spawn ≈ 241ms (CLI start + locks + sqlite reserve + COW prep) · fc-api-up ≈ 1ms · restore(load) 12ms · post-uds 141ms (resume + reseed + finalize) · total p50 428ms. The VMM side of a clone costs ~14ms; ~89% of per-clone latency is cocoon-side per-process work — pre-spawn + post-uds — which is precisely the daemon/library-mode surface. Prior tables' pre-spawn figures (457/509/1503ms) are each ~249ms higher in truth; their fc-api-up rows collapse to ~1ms.

First many-core datapoint: 384-core / 1.5TB box, fully isolated root (own binaries/config/meta), cocoon master-66e4232 (daemon supervisor landed) + CH dev-4b542ac + FC v1.16.1, sqlite backend, keep-src golden, daemon supervising throughout. Single uncontended clone: 77ms.

Corrected storm ladder (unique names per tier, clean between tiers):

B	ok	wall	rate	pre-spawn p50	restore(load) p50	post-uds p50	store-busy fails
64	64/64	473ms	135/s	328ms	4ms	49ms	0
256	256/256	1929ms	132/s	1455ms	8ms	85ms	0
512	512/512	5469ms	93/s	4921ms	16ms	155ms	0
1024	474/1024	12197ms	38/s	6418ms	24ms	78ms	550

Two findings that reshape the C1M model:

Per-box create rate ceils at ~135/s and does NOT scale with cores. 24× the cores of the 16-core testbed bought only 1.75× the rate (77→135/s). CPU is nowhere near saturated (B=256 = 0.67× oversub) yet pre-spawn grows linearly with B (328→1455→4921ms) while restore(load) stays 4-24ms (bind path healthy) — the queue sits in the pre-spawn segment, and the prime suspect is the sqlite cross-process single-writer (every clone's reserve write serializes; ~135/s ≈ the writer drain rate through N independent CLI processes).
The safe in-flight ceiling is ≤512 per box with the current retry budget: at B=1024, 550 clones died with writer contended past retry ceiling: store busy — clean failures (no residue, tiers cleaned to zero), but they define the admission limit an orchestrator must enforce.

Neither ceiling is hardware: both are the CLI-per-op + shared-writer architecture. The daemon supervisor (P0/P1, verified running and adopting through every tier) is the vehicle — a future clone-serving phase that batches reserve/finalize writes in one process would lift both. C1M arithmetic with today's numbers: 9.6k VMs/box ÷ 135/s ≈ 71s per box fill, fleet fills in parallel → 1M in ~2 min with ~110 boxes — still comfortably inside the plan, now grounded in a many-core measurement instead of a per-core extrapolation.

ROOT CAUSE of the ~135-143/s per-box ceiling — found and independently reproduced. It is NOT sqlite (correcting my earlier comment), NOT CPU, NOT the disk: it is the kernel mount-namespace copy (unshare(CLONE_NEWNS) + MS_REC|MS_PRIVATE) that the bind-redirect fast path performs per clone, serialized on the global namespace_sem with cost O(host mount count).

Elimination trail (each independently measured on the 384-core box, B=256 unless noted):

per-op sudo in my harness: 256× cocoon version p50 653→59ms as root — harness artifact removed, storm rate unchanged (143/s) → not the wall
sqlite: 256 processes × storebench on one shared db = 2,335 creates/s cross-process → exonerated (storm needs ~286 tx/s)
FICLONE+fsync of the golden COW ×256: p50 115ms → minor term
THP (never A/B, settings saved+restored): 143/s unchanged → exonerated
GOMAXPROCS 4/16/384: ≤11% effect → minor
fsync latency sampled DURING the storm: p50 0.14ms / p99 1.38ms → XFS journal healthy
PTY allocation ×256: sub-ms p50 → minor
CPU: min idle 45-56% during storms → not saturated
FC: start→api 4ms, snapshot load 8ms → healthy

The clincher: this host carries 264 mounts (prod-ish box: 7 CH VMs + sandboxd). 256× concurrent unshare -m --propagation private /bin/true (no cocoon at all): p50 742ms, wall 872ms (baseline /bin/true: 3ms). Every clone's mount-ns copy walks the whole table under the global write-side of namespace_sem → ~3ms serialized each × 256 ≈ the entire missing chunk of the storm's dominant phase (measured phase C = 906ms ≈ 742 unshare + 115 reflink+fsync + prep). It also explains every signature: rate doesn't scale with cores (global kernel lock), CPU stays idle (sleeping on the semaphore), per-clone latencies cluster near the wall (fair queue → convoy), and the 16-core testbed never showed it (clean mount table ~40 entries → sub-ms per copy; my earlier microbench there measured 0.1ms and wrongly exonerated the kernel path — mount-table size was the hidden variable).

Fix levers, in order of leverage:

Daemon clone-serving (os-image: add the desktop flavor (GNOME + OSWorld guest server) #152 later phases): one resident process holds ONE private mount-ns; per-clone binds become plain mounts inside it — the per-clone unshare disappears entirely.
FC upstream dynamic drive replacement (PR #5774): retires the redirect dance and the unshare with it (per CMGS: the placeholder TODO stays parked pending this).
Fleet hygiene: C1M boxes should run minimal mount tables (dedicated hosts ≈ 40 mounts → ~0.4ms per copy → this component's ceiling rises to ~2,500/s on its own).

Corrected B=256 phase ledger on the 384-core box (root, keep-src, sqlite): A cli+open+reserve 492ms (CLI floor 59 + spawn stagger + reads) · C 906ms (unshare ~742 + reflink/fsync 115 + prep) · fc-api 4 · load 8 · post-uds 156 · total p50 1593ms, 143/s, 100% success. The earlier 'sqlite writer ceiling / ≤512 in-flight' framing from the first bd26 comment is superseded: the B=1024 store-busy failures were a symptom of arrival clustering behind the unshare convoy plus BEGIN IMMEDIATE polling unfairness, not of writer throughput.

Overnight T2 on the 384-core/1.5TB box — end-to-end sandboxd warm-ready, the concrete C1M single-box path. Isolated tree, machine's prod VMs untouched, all exec-verified (not scheduler-ack), cross-checked against actual vm list counts.

Definitive best-config (sandboxd master, FC none-lane small, cocoond off during fill, GOMAXPROCS=12):

N warm-ready	wall	inst rate
1,029	10.6s	97/s
2,074	22.2s	90/s
3,093	34.4s	84/s
3,932 (peak, mem-guard)	46.3s	—

Peak is memory-bound (~97MB effective/VM incl page-cache share; guard at 164GB free). Direct-cocoon density earlier hit 5,376. exec-verify 10/10 random VMs at peak.

C1M path (rate decays with pool depth, so box-count trades against fill time):

255 boxes × 3,932/box → 1M in ~46s (parallel)
489 boxes × 2,048/box → 1M in ~20s
977 boxes × 1,029/box → 1M in ~11s

1-minute-class 1M is reached today with the shipped sandboxd; sub-30s is a box-count decision. This is exec-verified warm-ready, versus Modal's 54s scheduler-ack.

Bottleneck confirmed = kernel mount-namespace copy, NOT sqlite/CPU/disk. Proof: a keep-golden-source branch (stop instead of destroy → clones take the bind fast path) REGRESSED 2.5x on this host (66/s vs 169/s) because its per-clone unshare(CLONE_NEWNS) copies the 265-mount host table (742ms at 256-way) — costing more than the flock it removes. On a thin-mount host (.79, ~40 mounts, sub-ms unshare) keep-src wins. So golden strategy is mount-table-dependent: fat-mount host → dead-source symlink path; thin-mount/dedicated fleet → keep-source bind path.

Real fix (both problems at once): a cocoon daemon clone-launch server — resident workers each hold ONE private mount namespace, so per-clone launches become plain binds inside it, killing both the per-clone unshare AND the CLI spawn cost, regardless of mount count. Prototyped on branch perf/daemon-launch; currently blocked on setns EINVAL entering the per-VM netns from a persistent pre-unshared worker thread — the remaining integration item. Other secondary levers measured: refill_concurrency default 24 is a choke (≥48 saturates); cocoond running during fill costs ~30% (O(N) reconcile) — keep it off while filling.

Fleet-hygiene corollary for C1M: provision dedicated boxes with minimal mount tables so even the current bind path stays sub-ms per clone.

CH none-lane experiment (CMGS's call): new all-time record — the FC redirect problem is simply bypassed, not fixed.

CH clones rewrite disk paths in the snapshot config before restore (patchDisks), so there is NO symlink, NO per-golden flock, NO bind/unshare, and the golden source can be destroyed freely. The one blocker was host hugepage pools forcing hugepages=true onto goldens (mmap-restore refused by the #144 preflight); fixed with a per-VM --no-hugepages opt-out.

Direct-cocoon CH mmap storms (dead source): B64 362ms=176/s, B256 1,059ms=241/s — fastest B256 of the whole campaign (FC best 1,234ms).

sandboxd end-to-end (none_backend=ch, restore_mode=mmap, RC=64, cocoond off, GOMAXPROCS=12):

milestone	wall	inst rate
2,048 warm (incl golden build)	14.3s	143/s
1,080 / 2,080 / 3,085 / 4,126	5.4 / 11.9 / 19.5 / 28.0s	200→122/s
PEAK 4,762 warm (4,830 VMs, mem guard)	33.4s	—

exec-verify 10/10 at peak. Effective memory ~99MB/VM (vs FC-lane 122MB) — mmap page-cache sharing holds at scale.

vs FC none-lane best (same box, same harness): +21% capacity (4,762 vs 3,932), −28% fill time (33.4s vs 46.3s), and the golden-strategy/mount-table dilemma disappears entirely.

C1M with CH none-lane: 1M / 4,762 per box = 210 boxes × ~34s parallel fill — half-minute-class 1M, exec-verified.

Changes (two small experiment branches, one commit each, unpushed):

cocoon exp/ch-none-lane: per-VM --no-hugepages (types field + run flag + args gate) so clone-source VMs stay mmap-restorable on hugepage hosts.
sandbox exp/ch-none-lane: none_backend: fc|ch config knob + CH cold boots take --no-hugepages.

Strategy note for review: this reverses the earlier none-lane=FC lane assignment. With CH none-lane, FC upstream PR #5774, the keep-src branch, and the daemon-launch workaround all become unnecessary for the C1M path (daemon-launch may still matter for FC lanes if kept). Residual open items: the O(resident-N) rate decay (200→122/s) remains in both backends; probe CoW cost ~30MB/VM is the price of exec-verified readiness.

O(resident-N) fill decay: root cause closed. It is the XFS journal, not sandboxd, not sqlite, not CH/cocoon code.

Instrumented direct-cocoon fill (64-worker pipeline, CH none-lane mmap, 4,000 clones, per-clone --log-level debug txn timings, bucketed by completion order):

completion bucket	done@	wall p50	sqlite txn-sum p50	txn wait
0-500	2s	224ms	12.7ms	10.8ms
1500-2000	8s	273ms	6.3ms	4.5ms
3000-3500	16s	401ms	3.8ms	1.4ms
3500-4000	20s	406ms	3.2ms	1.3ms

sandboxd exonerated: the decay reproduces bit-for-bit without sandboxd (direct fill late-bucket rate ~125/s ≈ sandboxd's 122/s at the same N; sandboxd adds only a small constant for the silkd probe). PR pool: configurable refill concurrency, auto-scaled from node CPUs #46's shared refill semaphore is innocent in the fill window (Reconcile runs once at startup; reap batches are empty with zero claims) — its cleanup-vs-refill coupling matters only in steady-state ops.
sqlite exonerated with prejudice: meta transaction time FALLS as the fill proceeds (12.7→3.2ms; writer wait 10.8→1.3ms). The whole meta layer is 1-3% of clone wall.
The +182ms growth is filesystem-side: XFS log-grant (journal) contention from the reflink CoW machinery. Every clone FICLONEs the golden cow.raw + fsyncs; every resident guest's disk write needs an xfs_reflink_end_cow_extent journal transaction. Aggregate journal traffic grows with resident N, so every clone's FS work queues longer. Past a threshold it wedges outright: leaving 4,000 write-active CH guests resident for ~40 min drove 28k+ kernel dio workers into D-state on xlog_grant_head_wait (load avg 78,000) — an XFS log-grant exhaustion incident, drained cleanly after killing the fleet (prod VMs on the same FS unaffected throughout; box self-recovered, no reboot).

Headroom answer (CH 还能更高): yes — the ceiling is the journal, and it's ops-addressable. Journal-light early-fill ran at ~250/s; a journal-relieved box should hold that flat: 4k in ~16s, C1M ≈ 210 boxes × <20s. Levers for the fleet checklist: (a) mkfs.xfs with maximum log size on pool volumes; (b) spread pools across multiple filesystems/NVMe namespaces; (c) quiet the guests (tmpfs /var/log, noatime, minimal background writers in the pool image) — fewer end_cow conversions per resident; (d) A/B eager-copy vs reflink for write-heavy pools; (e) A/B ext4 (no reflink CoW conversion path). Hard rule learned: never leave thousands of write-active reflink-backed guests resident without journal headroom — this is the real density wall for write-active fleets, distinct from the memory wall.

XFS tuning without reformat: --no-direct-io A/B lands the fix. Same instrumented 4k fill harness, golden built with --no-hugepages --no-direct-io (clones inherit from snapshot):

	direct-IO baseline	buffered (`--no-direct-io`)
4k fill avg	~200/s (late ~125/s)	207/s (late ~167/s, +34%)
decay slope	224→406ms (+81%)	217→318ms (+47%)
peak D-state during fill	dio-worker army (28k post-residency wedge)	1
peak load1 during fill	hundreds	51
exec verify	pass	pass 3/3

Buffered guest writes turn per-write synchronous end_cow journal transactions into batched writeback — the D-state/journal-storm risk class disappears outright, and for ephemeral sandboxes page-cache semantics are the right default anyway.

No-reformat tuning list (ranked): (1) pool VMs run --no-direct-io — sandboxd needs one flag in runColdArgs, clones inherit; (2) fs.xfs.xfssyncd_centisecs 3000→500, runtime-reversible, keeps the log tail moving; (3) quiet pool guest images (tmpfs /var/log, prune background writers); (4) re-verify long-residency parking under buffered IO before declaring the wall fully gone; (5) new fleet boxes mount with logbsize=256k (bd26 is at 32k, mount-time-only). Corrected earlier guidance: the bd26 log is ALREADY at the 2GB max — log size was never the lever; transaction-volume reduction is.

Residual 217→318ms slope is second-order (writeback batches + allocation txns) — no longer the bottleneck class.

Second-order slope located + concurrency ceiling mapped (buffered-IO CH fills, phase-split first-500 vs last-500 of a 3k fill):

phase	first-500 p50	last-500 p50
A cli+reserve+metacopy	85ms	32ms
B staging cow reflink+fsync+patch	4ms	164ms
C spawn / D api-up / E restore	0-4ms	0-8ms
F resume+finalize	112ms	113ms
TOTAL	218ms	334ms

The entire residual growth is segment B. Direct micro rules out refcount long-tail: FICLONE+fsync is 42ms at 0 refs vs 45ms at 3k refs — reflink itself is O(1) in sharers. The grower is the staging fsync's XFS log-force competing with resident guests' writeback end_cow traffic (grows with N). We pay full durability tax on a staging disk that is about to be dirtied by an ephemeral sandbox and is worthless after host loss.

Concurrency sweep kills the more-workers path: W=64→208/s, 96→209/s, 128→195/s (worse, load up). ~210/s is a per-clone serial-section wall, not worker starvation.

Path to 300-400/s: (a) a cocoon knob to skip/relax the COW-staging fsync for ephemeral clones erases B → ~220ms flat per clone → ~300/s; (b) 400/s then requires decomposing F (112ms flat: resume+reseed+finalize). No architectural change anywhere — the running total of code needed for the full champion config remains: cocoon --no-hugepages (done, exp branch), sandbox none_backend: ch (done, exp branch), one-line --no-direct-io in runColdArgs, plus the staging-fsync knob for the 300/s tier.

Decay eliminated + ceiling mapped. Final state of the fill-rate investigation.

The XFS journal wall fell in three costumes, each killed by a zero-code lever:

direct-IO end_cow storm (28k D-state incident) → --no-direct-io pool VMs
staging fsync log-force → os-image: complete the desktop flavor's application set and pin what it installs #160 (utils.NoSync on clone/restore COW copies)
FICLONE remap log-reservations queueing behind resident guests' writeback (phase B1: 4→196ms) → runDir on tmpfs (run_dir config key; 10.5MB RAM per VM disk, 42G for 4k VMs)

Result: decay is GONE. 4,000 clones in 14.6s = 274/s flat (buckets 218/221/226/225ms — no slope), D=0, load 56, exec-verified. Recipe is entirely config + current master (e9502f6: fsync-skip, mmap-by-default, hugepages-opt-in): CH none-lane, golden --no-direct-io, run_dir=tmpfs, sqlite, W=64. Note: tmpfs runDir is legitimately ephemeral — runDir state dies with the host anyway; records reconcile.

Concurrency sweep on the flat config: W=64→274/s, 96→240, 128→210, 192→166 — monotonically worse. A residual ~3.6ms serial section (prime suspect: per-clone netns creation through the kernel's global mount lock) plus CPU queueing caps the box at 274/s under the no-code-change constraint. C1M at this recipe: ~250 boxes × 4k × ~15s fill.

Beyond 274/s requires code (tomorrow's menu, in leverage order): (1) drop the per-VM netns for none-lane (no NICs — nothing needs it): removes the last serial suspect and likely re-enables W scaling toward 300-400/s; (2) segment H (128ms resume+reseed+finalize) and A (~80ms CLI+reserve) are the ~210ms per-clone floor — daemon/batch surface, theoretical W=64 → 600/s+ if both halve. sandboxd integration unchanged: none_backend knob + two runCold flags (exp branch ready).

sandboxd full-recipe verdict: 155-157/s — 300+ is NOT reachable without code, and the blocker is precisely located.

Full champion env (sandboxd exp bin none_backend=ch + CH cold --no-direct-io, tmpfs runDir + empty CNI dir via env, sqlite, cocoond off), warm=4000:

RC	steady rate (100→4000)	total
64	155/s	27.7s incl golden build
96	157/s	25.6s

Arithmetic closes perfectly against the direct-harness numbers: sandboxd holds the silkd ready-probe inside the refillSem slot, so slot latency = clone (~370ms at RC=64-equivalent load) + guest resume→silkd-responsive ~240ms → 64/0.61s ≈ 157/s. The probe is honest product truth but it is pure vsock waiting — it does not need to occupy a provisioning slot.

Also landed on the way here, all config-only: the netns question is CLOSED — initNetwork already runs none-lane in the HOST netns when the CNI conf dir has no conflist (run.go:559); bd26's /etc/cni/net.d had hidden conflists, so pointing cni_conf_dir at an empty dir gives netns-free none-lane (verified netns-delta=0 across 8k clones). Effect: W=96 no longer degrades (240→258/s) though the W=64 plateau (~260-275/s direct) stands — the remaining floor is per-clone A (~80ms cli+reserve) + H (~125ms resume+reseed+finalize). Security note: per-VM netns was also quarantining the VMM process; fleet decision between host-netns (bench), per-VM (today), or a shared empty netns (best of both, future).

Roadmap to 300+ (both small, both code):

sandbox: move probeReady out of the refillSem critical section (separate bounded probe pool) → sandboxd returns to clone-bound ~260/s;
cocoon: A+H daemon/batch work → 300-600/s.

Ops notes: pgrep without -f is blind to "cloud-hypervisor" (>15 chars) — false prod-down scare; sandboxd name suffix randHex saw 2 collisions in 4k draws (auto-retried, consider widening). 228 stale day-residue netns swept; box quiesced (D=0, tmpfs unmounted, prod 7 CH + sandboxd intact all day).

Final-form regression on HEAD (cocoon 81efb9e + sandboxd 973bd67/#48 both-lanes-CH, latest image) — recipe confirmed, probe decoupling verified, remaining wall re-characterized.

R1 direct core recipe (CH none-lane, tmpfs runDir, empty CNI, no-direct-io golden): 4,000 clones / 15.6s = 257/s, four buckets flat (233/246/243/240ms, zero decay), D=0, exec 3/3. Same class as yesterday's 274/s — the whole T2 recipe holds on HEAD, no regression.

R2 / RC sweep on HEAD sandboxd (warm=4000, no_direct_io+mmap config):

RC	steady 200→4000	load (384 threads)
64	152/s	585
96	167/s	673
128	158/s	906
192	180/s	858

#48 already implemented the probe/refill decoupling from #47 (refillOne releases refillSem right after the clone; probes run on a separate probeSem). The old pathological RC degradation (64>96>128 monotonically worse) is GONE — rate is now flat-to-up with RC. But the gain is only ~18% because the bottleneck moved to CPU/memory-bandwidth: sandboxd fills run at load 585-906 (severe oversubscription on 384 threads) vs R1-direct's load 40. Root cause: the silkd ready-probe forces the guest to actually become responsive (faults in more pages + runs guest userspace) beyond just reaching vsock.uds; at 4,000 concurrent resume+probe this saturates CPU. This is the physical cost of exec-verified readiness (the 274-vs-155 story), now confirmed CPU-bound, not architecture-bound — #47 item 3 is effectively done.

Path to ~260/s warm-ready is no longer sandboxd-side: it's cutting per-guest wake cost (silkd cold-start slimming, mmap first-touch prefetch) or spreading concurrency across more nodes. Box verified clean afterward (pty restored, prod 7 CH + sandboxd intact throughout).

Ledger entries from the cross-node snapshot placement branch (#52), measured on bare metal (Ryzen 9700X, linux/amd64):

Record-lock acquisition is now globally serialized. recLock gained a reference count guarded by one recLocksMu so eviction cannot strand a waiter on a dead mutex (the lock-identity split). Cost, measured head-to-head against the old bare sync.Map shape at 16 threads: 2.2ns → ~155ns per acquisition across distinct ids (~6.4M acquisitions/s ceiling). Accepted because every caller sits on 77ms+ operations (checkpoint branch, template cold provision, delete) and the warm-claim path never touches recLock.

Trigger to revisit: branch/delete rates approaching millions per second, or any change that puts recLock on the claim path.
Next step, already prototyped and measured: per-entry atomic refcount with lazy sweep-time eviction benches at ~12ns at scale, but needs a re-validate-after-increment retry loop to close its own eviction race — take it only when the trigger fires.

Ownership probing is O(peers) per distinct cold id. The per-id positive cache (5s) and singleflight kill the measured hot-id amplification (10 misses × 20 peers: 200 → 20 HEADs), but a stream of misses for distinct ids still fans out once each. Inherent query cost of gossip-free ownership; noted so nobody rediscovers it as a surprise.

Trigger: sustained cross-node branch rates of many distinct checkpoints per second on large fleets — then consider a bloom/digest hint in NodeState (small, fixed-size) to pre-filter the fan-out.

Closed by the same branch, for the record: the 1s gossip tick no longer lists the checkpoint store (was 15.4ms/tick at 1000 records), and NodeState no longer carries an unbounded checkpoint id list at all (was ~881KB wire + 4.9ms marshal/unmarshal per tick at 2000 ids/node × 20 nodes).

Static-analysis round over sandboxd + silkd + boot/init (2026-07-27). Fixes shipped in #53 (template-hash sort: epoch churn +19/min → 0; silkd chunk-render zero-fill: micro 1.77x, fs_pull 963-969 vs 843-949 MiB/s A/B). Ledger entries from the same round — noted, not taken:

silkd inbound BufReader (64KB) vs base64-expanded BULK_CHUNK (~349KB): ~6 read syscalls per inbound bulk frame. Enlarging to one-frame capacity was prototyped on paper and rejected for now: every RPC dials a fresh vsock conn (the wire-v1 triple-setup), so a ~349KB per-conn buffer moves allocation/fault cost onto every exec RTT to buy single-digit-% on fs_push. Correct home: wire protocol v2 (item 1) — a long-lived multiplexed conn makes the big inbound buffer free. Fold it in there.

Guest wake ~240ms: no silkd-code lever. Read of the full crate confirms silkd does no per-resume work (no watchers/timers/caches rebuilt; State::new is empty maps; listener accepts immediately). The wake cost is page fault-in + guest kernel, matching the R2 CPU-bound conclusion. "silkd cold-start slimming" as a code item can be closed; remaining levers are image resident-set size and fleet spreading.

Smaller notes, each with its reopen trigger:

Two randHex CSPRNG draws per claim (stampIdentity) mergeable into one — µs-class; fold into any future claim-path touch.
handleClaim computes key.Hash() eagerly; warm-hit success path never consumes it — ~0.3µs/claim.
overQuota precheck + warm-pop are adjacent separate m.mu acquisitions — ns-class merge.
refillOnce catch-up rescans all pools once per spawned refill under m.mu (O(deficit × pools)) — µs-class, deliberate round-robin fairness; trigger: many pools + large deficits + high core count showing claim-lock contention.
s3 Publish pays an unconditional LIST+diff even on first publish — ~20-100ms on an already-seconds operation; not worth the plumbing.
memberlist PushPullInterval=1s is the sole state-propagation channel (no broadcasts) — O(peers × catalog) JSON per node per second; trigger: large fleets, then move to delta broadcasts.
egress intercept leaf cache clear()s all 1024 entries on overflow — trigger: single-sandbox workloads spanning >1024 intercepted hosts.
silkd Request internally-tagged serde decode double-passes every frame (~tens-hundreds ns) — dies with wire v2.
silkd fs.find recompiles its regex per call — LRU only if the verb shows up in real load.
silkd process table: 256KB replay ring per exec, detached entries linger 300s, no aggregate cap — the attach/logs contract's price; trigger: bursty many-background-exec workloads showing guest RSS pressure.

boot/init: no material findings — cold-boot criticality is kernel + guest-ready as already ledgered; init's own path is µs-to-low-ms with correct fault-path degradation (none-lane carries no ip= params and skips the NIC wait entirely).

SDK/protocol/mcp/python round (same static pass, fixes in #53: dial-path allocs 16→2, shared NewFrameScanner, ReadFile concat, TCP_NODELAY, FS_CHUNK 256K). Noted, not taken:

Python REST path opens a fresh non-keepalive connection per call (urllib sets Connection: close by design) — ~1 TCP handshake per claim/release/etc., compounding in _redirect_fallback (up to 4 chained) and lookup probes. The SDK is deliberately stdlib-only; a hand-rolled persistent http.client.HTTPConnection with stale-conn retry is real complexity for a control-plane path. Trigger: Python-client claim latency ever matters at scale, or the stdlib-only constraint is relaxed.
Python push/pull fully materialize tar archives in memory (bytes in/bytes out; the openai adapter .read()s the whole stream too) vs Go's io.Reader/Writer streaming. Fix is an API-surface addition (Iterable[bytes]/sink overloads). Trigger: workspace archives in the hundreds of MB.
openai adapter reconstructs Client+Sandbox per operation — µs today; becomes real the moment Client gains connection reuse. Do together with the keep-alive item.
mcp: double JSON marshal per data-returning tool call (MCP content-block shape makes it largely structural), map[string]any envelopes, a 5-min timerCtx per tool call incl. metadata-only tools, append(out,'\n') possible realloc per reply. All µs-class on an LLM-cadence path — not worth motion.
sdk/go context.AfterFunc registered twice per RPC (dial + handshake windows) — load-bearing for cancel-without-deadline; merging risks cancellation semantics for 2 allocs. Rejected.
sdk/go ListDir unbounded append across Entries batches — metadata-weight; only matters for very large directories.

Deferred items from the template-content-identity round:

store/dir generation-addressed layout replacing the cross-process flocks -> store/dir: generation-addressed records to replace cross-process flocks #63. Trigger: shared-FUSE flock friction (broken/local-only locks, ms-scale lock latency, or an observed starved re-promote). Measured today: 17us lock cycle / 31us Fetch+release locally; ms-scale per claim on FUSE.
Promote-path digest cost (full export re-read + SHA-256, ~0.5-1s/GB): engine-side emission at export time -> snapshot export: emit the content digest while writing (--digest) cocoon#193 (zero extra read on both backends). Interim option if S3 promote latency surfaces first: tee the hash into the existing upload read. Depends on the v1->v2 digest stream decision recorded in the cocoon issue, which must be settled before the digest API first ships.

Rows from the SOCKS5 / locked-NIC batch (#164, #165, #168), measured on cocoon-test1 with both arms interleaved A-B-B-A in one run:

Warm claim, second listener (egress: serve SOCKS5 beside the HTTP proxy so non-HTTP protocols can leave the sandbox #164). A pool whose policy has a tunnel-eligible rule (a bare {"host": ...} rule counts) binds a second unix socket per claim (unlink+socket+bind+listen). µs driver, n=96 per arm, 3 rounds: no policy 432 vs 442 µs (A/A drift 14–49 µs, no effect); bare-rule policy 496 vs 532 µs, +36 µs ≈ 7 % (A/A drift 7–18 µs). This is the only measured hot-path cost of the batch and the argument for making the SOCKS5 door opt-in per pool or tenant rather than implied by the rule shape.
Exec RTT, guest lane check (egress: tell the guest when its NIC is locked so the bridge lane gets the proxy variables #165). silkd's routes_directly() adds one stat("/etc/silkd-nic-locked") per spawned process on a guest with a NIC and, when the marker exists, the proxy variables in the child's env; the none lane short-circuits and pays nothing. execab driver, 500 execs of /bin/true per arm, 3 rounds, host under another session's 1000-VM clone fill (load 45–80): egress lane p50 750 vs 745 µs, per-round A/A drift 11–18 µs; none lane 775 vs 767 µs, A/A drift 1–33 µs. Below the noise floor; not a ledger item.

Closing after a full re-verification of every item in this thread against main (ca6d08f, 2026-09-14). Ledger, one line per item:

Wire protocol v2 / bulk-frame buffer — tracked in protocol: silkd binary bulk frames (multiplexing parked; per-RPC setup solved by keep-alive) #34.
Refill admission re-eval — done (perf(pool): trigger a coalesced refill after consuming a warm VM #41, sandboxd: adopt the CH none-lane C1M recipe (backend knob, buffered pool disks, probe outside the refill semaphore) #47): configurable cap, auto-scale, probe outside the semaphore.
Claim did not kick refill — done (perf(pool): trigger a coalesced refill after consuming a warm VM #41). Restore-mode CoW default + hugepage guard — done (pool: pass --restore-mode mmap on CH-lane clones (egress burst wall -53%) #40). XFS journal / direct-IO — done (no_direct_io, deploy.md). Staging fsync + tmpfs run dir + empty CNI dir — done (cocoon os-image: complete the desktop flavor's application set and pin what it installs #160, deploy.md recipe). VM-name suffix collisions — done (6ba90d6). Generation-addressed dir store — done (store/dir: generation-addressed records to replace cross-process flocks #63). Promote-path digest emission — done (store-owned v2 chunked export digest: PublishDigested with dir parallel hash and s3 upload tee #64, cocoon desktop: write an xauth entry for :1 once Xvfb is up #193). s3 Publish list+diff — resolved by the generation rework.
FC drive-redirect / keep-source / unshare convoy — obsolete under the Cloud Hypervisor lane decision.
Pre-dialed spare connection, fs_push chunk size, nft netlink merge, silkd scanner-buffer pooling, encodeTagged rework, silkd cold-start slimming — rejected with data, unchanged.
SOCKS5 second listener (+36 µs per warm claim) — the door is opt-in per policy in egress: make the SOCKS5 door opt-in per policy #169.
Locked-NIC exec RTT — below noise, not a ledger item.
Still deferred with their triggers, unchanged in code: claimStore whole-map rewrite, idleOnce/archiveOnce scans, checkpoint TTL sweep, resyncEgress serial lock, probeReady fallback poll, recLock serialization, ownership probing per cold id, the static-analysis and SDK micro items — carried to perf: remaining performance ledger after the 2026-09 SOCKS5 batch #172 with the triggers.
Cocoon clone-serving / CLI-per-op — the largest remaining fill-rate lever, unowned; carried to perf: remaining performance ledger after the 2026-09 SOCKS5 batch #172 as the open item.

The headline numbers of record now live in docs/benchmarks.md and docs/performance.md; the C1M campaign measurements stay in this thread and #172 points back here for them.

Navigation Menu

Repository navigation

perf: remaining performance ledger after the 2026-07 hotspot round #30

Open — worth doing, needs a product decision

Deferred — reopen triggers documented

Rejected with data — do not re-litigate without new evidence

Physical boundaries — not this repo's levers

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

perf: remaining performance ledger after the 2026-07 hotspot round #30

Description

Open — worth doing, needs a product decision

Deferred — reopen triggers documented

Rejected with data — do not re-litigate without new evidence

Physical boundaries — not this repo's levers

Activity

CMGS commented on Jul 13, 2026

CMGS commented on Jul 13, 2026

CMGS commented on Jul 14, 2026

CMGS commented on Jul 20, 2026

CMGS commented on Jul 20, 2026

CMGS commented on Jul 20, 2026

CMGS commented on Jul 21, 2026

CMGS commented on Jul 21, 2026

CMGS commented on Jul 21, 2026

CMGS commented on Jul 21, 2026

CMGS commented on Jul 21, 2026

CMGS commented on Jul 21, 2026

CMGS commented on Jul 21, 2026

CMGS commented on Jul 21, 2026

CMGS commented on Jul 22, 2026

CMGS commented on Jul 22, 2026

CMGS commented on Jul 22, 2026

CMGS commented on Jul 22, 2026

CMGS commented on Jul 22, 2026

CMGS commented on Jul 22, 2026

CMGS commented on Jul 22, 2026

CMGS commented on Jul 26, 2026

CMGS commented on Jul 26, 2026

CMGS commented on Jul 26, 2026

CMGS commented on Aug 11, 2026

CMGS commented on Sep 14, 2026

CMGS commented on Sep 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions