Repository navigation
snapshot: stop retaining ~3x the export image in RSS after a snapshot - #289
Conversation
The export kept three copies of the same state alive at its peak and left the freed pages with glibc instead of the OS, so RSS never came back: 1. state_json was built only to hash it and then stayed alive through the second serialization and the compression of the whole snapshot. Release it right after the hash (swap with an empty string). 2. fc::variant(mutable_variant_object) takes its argument BY VALUE, so fc::variant(snapshot) copied the entire state tree - on the testnet stand about 1.1 GB of extra live objects next to the JSON string. Move instead. 3. Freeing a few hundred thousand small objects fragments the arenas: no arena end is wholly free, so the webserver's periodic malloc_trim(0) returned nothing and the RSS stayed at the peak for 30+ minutes. Trim right after the export, under the glibc guard that the Windows build needs, and log the released amount so the next measurement can tell whether it worked. Measured on the testnet stand 20.11.2026 (image 468 MB): anon 289 MB -> 2 549 MB peak -> 1 685 MB and flat; ten 128 MB anonymous regions fully resident; the production node with a 37 MB image shows +2 MB, i.e. the price scales with the exported state.
|
Adversarial self-review of my own diff (the project rule requires it for a node change): 1. Does
2. Is 3. Does anything read 4. Does the file format change? No: same 5. What if the export throws before the trim? One honest caveat: the trim only helps where the pages are still in glibc's hands; if the retention turns out to live in buffers owned by the serializer itself, the numbers after the next snapshot will show that, and I will say so instead of claiming the fix worked. |
Problem
After every snapshot export the node keeps the memory it needed for the export. Measured on the testnet stand (
testnet.viz.world, node carries ~48k prediction markets, snapshot image 468 MB):[heap]157 MB + ten anonymous 128 MB regions, all fully residentheap trim: released ...never appears in the log, i.e. the periodicmalloc_trim(0)returns nothing: the pages are not free chunks, they are fragmented arena pages. Reproduced on two consecutive snapshots (03:02Z -> 1 727 MB, 05:32Z -> 1 685 MB); a restart returns to 76 MB.Scale is the image size, not traffic: production
api.viz.worldwith a 37 MB image shows +2 MB.With
snapshot-every-n-blocks = 9000(~2.5 h) the stand consumes ~1.4 GB per cycle against a 12 GiB cgroup limit -> ~18-20 h until OOM.Cause
Three separate issues, all in
write_snapshot_to_file:state_jsonis built only to compute the payload checksum, but stayed alive through the second serialization and the compression - hundreds of MB held longer than needed.fc::variant(mutable_variant_object)takes its argument by value (thirdparty/fc/include/fc/variant.hpp:289), sofc::variant(snapshot)copied the whole state tree - ~1.1 GB of extra live objects at the peak, right next to the JSON string. This is what brings the peak to ~5x the image.malloc_trim(0)cannot return the pages. Trimming right after the export - when we know exactly what was just freed - is both cheaper and more effective.Change
state_jsonimmediately after hashing (std::string().swap);fc::variant(std::move(snapshot))instead of copying the tree;release_snapshot_heap()-malloc_trim(0)right after the export, under#if defined(__GLIBC__)(an unguarded glibc call broke the Windows build before, see the heap-trim guard inwebserver_plugin.cpp), logging the released amount when it is >= 16 MiB so the next measurement can tell whether it worked.No behavioural change: file contents, checksum,
.part+ rename, fsync and the log lines of the export itself are untouched.How to verify
vizd:fix/snapshot-export-memoryon this PR.Anonymous:in/proc/<pid>/smaps_rollupbefore, at the peak, and 5/30 minutes after the snapshot at block 86 877 000 (or the next multiple of 9000).Snapshot: snapshot export released N MiB to the OS ...line, or a residue near zero without it.Checked locally: single-TU
-fsyntax-onlyonplugins/snapshot/plugin.cppwith the flags fromcompile_commands.jsonpasses (rc=0); full build and tests run in CI.