Skip to content

snapshot: stop retaining ~3x the export image in RSS after a snapshot - #289

Merged
On1x merged 1 commit into
masterfrom
fix/snapshot-export-memory
Nov 20, 2026
Merged

On1x merged 1 commit into
masterfrom
fix/snapshot-export-memory

Conversation

@On1x

@On1x On1x commented Nov 20, 2026

Copy link
Copy Markdown
Member

Problem

After every snapshot export the node keeps the memory it needed for the export. Measured on the testnet stand (testnet.viz.world, node carries ~48k prediction markets, snapshot image 468 MB):

value
anon before 289 MB
peak during export 2 549 MB
after the export, flat for 30+ min 1 685 MB
profile of the residue [heap] 157 MB + ten anonymous 128 MB regions, all fully resident

heap trim: released ... never appears in the log, i.e. the periodic malloc_trim(0) returns nothing: the pages are not free chunks, they are fragmented arena pages. Reproduced on two consecutive snapshots (03:02Z -> 1 727 MB, 05:32Z -> 1 685 MB); a restart returns to 76 MB.

Scale is the image size, not traffic: production api.viz.world with a 37 MB image shows +2 MB.

With snapshot-every-n-blocks = 9000 (~2.5 h) the stand consumes ~1.4 GB per cycle against a 12 GiB cgroup limit -> ~18-20 h until OOM.

Cause

Three separate issues, all in write_snapshot_to_file:

  1. state_json is built only to compute the payload checksum, but stayed alive through the second serialization and the compression - hundreds of MB held longer than needed.
  2. fc::variant(mutable_variant_object) takes its argument by value (thirdparty/fc/include/fc/variant.hpp:289), so fc::variant(snapshot) copied the whole state tree - ~1.1 GB of extra live objects at the peak, right next to the JSON string. This is what brings the peak to ~5x the image.
  3. After the tree is freed, the arenas are fragmented: no arena end is wholly free, so a generic malloc_trim(0) cannot return the pages. Trimming right after the export - when we know exactly what was just freed - is both cheaper and more effective.

Change

  • release state_json immediately after hashing (std::string().swap);
  • fc::variant(std::move(snapshot)) instead of copying the tree;
  • release_snapshot_heap() - malloc_trim(0) right after the export, under #if defined(__GLIBC__) (an unguarded glibc call broke the Windows build before, see the heap-trim guard in webserver_plugin.cpp), logging the released amount when it is >= 16 MiB so the next measurement can tell whether it worked.

No behavioural change: file contents, checksum, .part + rename, fsync and the log lines of the export itself are untouched.

How to verify

  1. CI builds vizd:fix/snapshot-export-memory on this PR.
  2. Deploy it to the testnet stand and compare Anonymous: in /proc/<pid>/smaps_rollup before, at the peak, and 5/30 minutes after the snapshot at block 86 877 000 (or the next multiple of 9000).
  3. Expect either the new Snapshot: snapshot export released N MiB to the OS ... line, or a residue near zero without it.

Checked locally: single-TU -fsyntax-only on plugins/snapshot/plugin.cpp with the flags from compile_commands.json passes (rc=0); full build and tests run in CI.

The export kept three copies of the same state alive at its peak and left the
freed pages with glibc instead of the OS, so RSS never came back:

1. state_json was built only to hash it and then stayed alive through the
   second serialization and the compression of the whole snapshot. Release it
   right after the hash (swap with an empty string).
2. fc::variant(mutable_variant_object) takes its argument BY VALUE, so
   fc::variant(snapshot) copied the entire state tree - on the testnet stand
   about 1.1 GB of extra live objects next to the JSON string. Move instead.
3. Freeing a few hundred thousand small objects fragments the arenas: no arena
   end is wholly free, so the webserver's periodic malloc_trim(0) returned
   nothing and the RSS stayed at the peak for 30+ minutes. Trim right after the
   export, under the glibc guard that the Windows build needs, and log the
   released amount so the next measurement can tell whether it worked.

Measured on the testnet stand 20.11.2026 (image 468 MB): anon 289 MB -> 2 549 MB
peak -> 1 685 MB and flat; ten 128 MB anonymous regions fully resident; the
production node with a 37 MB image shows +2 MB, i.e. the price scales with the
exported state.
@On1x

On1x commented Nov 20, 2026

Copy link
Copy Markdown
Member Author

Adversarial self-review of my own diff (the project rule requires it for a node change):

1. Does std::move actually move here, or is it a no-op? Checked the whole chain in thirdparty/fc, because a move that silently copies would make fix #2 worthless:

  • mutable_variant_object(mutable_variant_object &&obj) : _key_value(fc::move(obj._key_value)) — variant_object.cpp:244, generated explicitly, so the unique_ptr is transferred;
  • variant(mutable_variant_object obj) takes by value, then new variant_object(fc::move(obj)) — variant.cpp:136;
  • variant_object(mutable_variant_object &&obj) : _key_value(fc::move(obj._key_value)) — variant_object.cpp:115.
    So the vector of entries is never copied. snapshot is not referenced after that line.

2. Is state_json still needed after the hash? No: the checksum is the only consumer; the state is serialized again as part of snapshot_json. Releasing it right after the hash removes an overlap of two several-hundred-MB strings.

3. Does anything read state after std::move(state)? No — it is moved into snapshot["state"] and never touched again.

4. Does the file format change? No: same header + state keys, same checksum input, same compression, same .part + rename + fsync. Only lifetimes change.

5. What if the export throws before the trim? release_snapshot_heap is at the end of the function; on an assertion the trim is skipped and we are back to the old behaviour — no new failure mode.

One honest caveat: the trim only helps where the pages are still in glibc's hands; if the retention turns out to live in buffers owned by the serializer itself, the numbers after the next snapshot will show that, and I will say so instead of claiming the fix worked.

@On1x
On1x merged commit 2acf1fc into master Nov 20, 2026
8 checks passed
@On1x
On1x deleted the fix/snapshot-export-memory branch November 20, 2026 08:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant