perf: remaining performance ledger after the 2026-09 SOCKS5 batch #172

Activity

1. CMGS commented on Sep 14, 2026

   

   CMGS

   on Sep 14, 2026

   ContributorAuthor

   More actions

   Two rows from the 2026-09-14 Rust ownership audit (all of silkd and boot/init read in full), kept out of the review PR #173 as their own change:

   • Eager broadcast ring per process — silkd/src/proc.rs Table::register calls broadcast::channel(OUTPUT_FANOUT) (256) for every exec and every pty open, and tokio allocates the whole slot buffer at construction: measured Chunk 32 B, Slot 48 B, Mutex<Slot> 56 B on Linux → about 14 KB of heap per registered process, for a live fan-out most execs never attach to. Present since silkd's first commit (ddeddc7, 2026-07-05), where the ring backed attach/logs on detached processes and the concern was reaping and pid reuse, not per-exec memory. CPU is under one percent of a fork+exec; the real cost is retention: a detached exec keeps its entry for REAP_DELAY (300 s), so a thousand detached execs pin about 14 MB of empty rings on top of the log ring. Fix shape: create the sender lazily on the first attach_stream (OnceLock<Sender>; emit already checks receiver_count() > 0), or cut the capacity. Trigger: detached-exec-heavy workloads, or any memory budget per guest.
   • Per-git-verb strings — silkd/src/git.rs builds the format!("git {verb}") label before subprocess_result decides whether it needs it (only on failure), and every session command allocates two strings for its sentinel (session.rs / sysutil::rand_token). Noise next to a fork and a shell round trip; recorded so nobody re-finds them.
2. CMGS

   added 2 commits that reference this issue on Sep 14, 2026

   pool: bind a warm clone's egress doors at refill, not at claim

   ...

   a6f01d1

   pool: bind a warm clone's egress doors at refill, not at claim (#177)

   ...

   3db7456
3. CMGS commented on Sep 14, 2026

   

   CMGS

   on Sep 14, 2026

   ContributorAuthor

   More actions

   Two rows closed on 2026-09-14:

   • Eager broadcast ring per process (the Rust ownership-audit row above) — silkd: build a process's fan-out ring on its first attacher #175. Table::register no longer allocates the 256-slot channel; attach_stream builds it on the first attacher under the ring lock. cocoon-test1, same sandboxd, guests differing only in silkd, 500 timed execs per run, A B B A × 3: exec p50 726 → 699 µs, p90 780 → 754 (n=3000 each, every round lower); silkd VmRSS after 1000 detached spawns 5040 → 20768 kB before, 4920 → 5636 kB after. The 256 KB log ring and the 300 s detached linger in the micro list are a different item and stay.
   • SOCKS5 second listener, +36 µs per warm claim — pool: bind a warm clone's egress doors at refill, not at claim #177 (row removed from the table). Refill binds a warm clone's doors and the claim serves the pre-bound pair after a one-accept probe. cocoon-test2 warm claims, A B B A × 3, n=288 per arm: no policy 243 vs 237 µs p50 (equal), HTTP door 307 → 263, both doors 351 → 274. The remaining 26–37 µs over the no-policy floor is the proxy build and the two serve goroutines.

   Also landed from the same batch, not perf rows: #176 (idle sweep vs live connections, #146) and #174 (spawn error kinds, #145).
4. CMGS commented on Sep 14, 2026

   

   CMGS

   on Sep 14, 2026

   ContributorAuthor

   More actions

   Rows from the whole-repo review round (PR #179), measured or estimated by the readers, none changed in that PR:

   • wire fastBulk decodes each data frame into a fresh buffer and copies it once more into the caller's slice; append-decode into the caller's buffer would drop one copy per 256 KiB chunk on every file read.
   • usage and audit journals append under one mutex with json.Marshal plus an unbuffered write(2) per event, shared by the claim path (recordUsage in finalizeBatch) and every guarded-egress decision; ~2–4 µs marshal plus 1–3 µs syscall per claim, and at 10k egress requests/s the claim path queues behind the data plane on one mutex. A size- and time-bounded buffer or a single writer goroutine.
   • reap tick holds m.mu across three full scans of m.claimed (reapOnce, idleOnce, archiveOnce) and hashes a PoolKey per claim in two of them; ≈185 µs of global-lock hold per 5 s at 1000 live claims. One scan collecting all three victim sets.
   • claims.json commit is O(live claims) per claim/release/hibernate/wake (maps.Clone + marshal + whole-file write under writeMu); ≈1.7 ms at 1000 live claims. Per-claim records or an append-only op log.
   • egress usage writes one billing line per proxied request; a guest at 1000 req/s rotates the 64 MiB journal in ~9 minutes and evicts other sandboxes' records. Aggregate per sandbox, emit at release.
   • reached() probe takes syscall.ForkLock on every guarded-egress claim; a claim that overlaps a refill's cocoon exec waits for that fork window. Accept4(SOCK_CLOEXEC) (Linux-only file) drops the lock; needs its own A/B.
5. CMGS commented on Sep 14, 2026

   

   CMGS

   on Sep 14, 2026

   ContributorAuthor

   More actions

   One more row from the same round (PR #179), unchanged there:

   • tap nft lock at claim — armEgress runs netfilter.Lock(tap) on the claim path: two netlink connections and two flush round trips per guarded-egress claim (the ~36 µs measured in review: batch-end round on the SOCKS5 opt-in, ports and lane-file state #173). The tap is already known at refill (startVM records it), so the lock could move next to the door pre-bind of pool: bind a warm clone's egress doors at refill, not at claim #177 and leave the claim path with the map write; a trimmed warm VM would unlock in destroy. Constrained by the hibernate rule (cocoon resumes before the lock can be re-applied), so it needs its own A/B and a look at what a warm VM may reach before its claim.
6. CMGS commented on Sep 17, 2026

   

   CMGS

   on Sep 17, 2026

   ContributorAuthor

   More actions

   Ledger addition: #213 — a redirected claim counts as an arrival on the entry node as well as on the node that serves it, so a burst against one node raises the warm target across the mesh. It rode on the back of #181: until the shrink lands, every inflated pool keeps its high-water mark, so the cluster strands N times the single-node residue. #211 bounds that in time; what gets counted is still open, and the reopen trigger is a mesh whose total warm VMs exceed its peak concurrent claims.
7. CMGS commented on Sep 22, 2026

   

   CMGS

   on Sep 22, 2026

   ContributorAuthor

   More actions

   Re-verified every row of this ledger against main at 24d9993 (2026-09-22). All twelve are still open — nothing here has been fixed since the ledger was written, and no named file or symbol has moved. The only two structural changes touching these rows, netfilter.LockedTaps() prefetch in resyncEgress and the seq/written coalescing in claimStore, both landed in d19b677 on 2026-09-03, before this ledger, so neither counts against it.

   Two corrections to the wording, so nobody optimizes against a shape the code does not have:

   The reap tick does not hold m.mu across the three scans. reapOnce, idleOnce and archiveOnce are three separate calls in the tick (sandboxd/pool/pool.go:497-503), and each takes and releases m.mu on its own (sandboxd/pool/claim.go:400-428, sandboxd/pool/hibernate.go:181-192, sandboxd/pool/archive.go:46-61). The cost stands — three O(n) passes over m.claimed per tick, two of them hashing a PoolKey per claim — but there is no single critical section spanning all three.

   fastBulk is not where the extra copy is. It allocates one buffer per data frame and returns out[:n] (protocol/wire/frame.go:738-744); there is no second copy inside it. The second copy is at the consumers — sandboxd/engine/silkd.go:109 (data = append(data, resp.Data...)) and sdk/go/utils.go:172-175. The row's cost claim holds; the fix belongs on the consumer side, or in an append-decode API that lets the caller supply the buffer.

   Current file:line for each row, for whoever picks one up:

   row	site
   claimStore whole-map marshal + rename	sandboxd/pool/claims.go:130-142
   idleOnce / archiveOnce scans	sandboxd/pool/hibernate.go:181-192, sandboxd/pool/archive.go:46-61
   checkpoint TTL sweep full list	sandboxd/pool/checkpoint.go:339 → checkpoint.go:91
   resyncEgress serial pass	sandboxd/pool/reconcile.go:170-199
   probeReady fallback poll	sandboxd/pool/refill.go:429-439, :461
   recLock global serialization	sandboxd/pool/reclock.go:7-15, sandboxd/pool/template.go:256-269
   peer ownership probe	sandboxd/store/peer/probe.go:62-73
   journal mutex + unbuffered write	sandboxd/pool/journal.go:49-64
   armEgress nft lock on the claim path	sandboxd/pool/claim.go:352 → sandboxd/pool/egress.go:139 → sandboxd/netfilter/nft_linux.go:19-53,77-85
   cocoon CLI per op	sandboxd/engine/engine.go:342-353

   #213 was re-checked too and is unchanged: p.noteArrival(start) still runs before the warm check at sandboxd/pool/claim.go:46.
8. CMGS commented on Oct 17, 2026

   

   CMGS

   on Oct 17, 2026

   ContributorAuthor

   More actions

   Closed since 09-22:

   • the claims.json commit encodes only changed claims (pool: a claims.json commit encodes only the changed claims #305);
   • the idle sweep skips its scan when no key idles (pool: a VM event scans for its own claim, and the idle sweep skips the scan when no key idles #315);
   • reap-tick purges share one fsync (pool: a reap tick marks its archive purges with one directory fsync #321);
   • the Go SDK decodes bulk frames into a reused buffer (wire, sdk/go: streaming consumers decode bulk frames into one reused buffer #328); sandboxd/engine/silkd.go:118 still appends;
   • Python keeps its connection between calls (keep-alive, silkd: serve RPCs back to back on one connection #196/sdk/go: keep a handle's relay connection across calls #198/sdk/python: keep a handle's relay connection across calls #199).

   Still open: one cocoon CLI process per engine op; the archive scan; the checkpoint TTL full list; the serial resyncEgress pass at restart; the probeReady fallback poll; recLock serialization on template records; the peer ownership probe; the journal mutex with an unbuffered write; egress usage written per request; the ForkLock in outbound/doors.go:119; the nft lock taken on the claim path; #213.
9. CMGS commented on Oct 26, 2026

   

   CMGS

   on Oct 26, 2026

   ContributorAuthor

   More actions

   Addition, off the claim path: the archive checkpoint pins.

   • fix(sandboxd): retain checkpoints through failed claim writes #380: the pin set now comes from the claims journal's reference counts. With 5,000 claims each holding a checkpoint, it went from 340 to 163 µs and from 48 to 18 allocs.
   • pool: O(1) archive pin check, follow-ups to #380 #381: a single-id pin check (DeleteCheckpoint, once per delete marker in the retry sweep, dropStaleRow) is now a lookup. It went from 162 µs, 218 KB and 18 allocs to 25 ns and 0 allocs. It no longer copies the whole set under the manager mutex.

   Measured on a mac M2 Max with -cpu 1, both arms interleaved in one run. The cost on the claim path is one extra map lookup per journal set/del, about 25 ns, with no allocation.