Skip to content

webserver: return free heap pages to the OS (malloc_trim), not only cap arenas - #287

Merged
On1x merged 1 commit into
masterfrom
fix/heap-trim
Nov 19, 2026
Merged

On1x merged 1 commit into
masterfrom
fix/heap-trim

Conversation

@On1x

@On1x On1x commented Nov 19, 2026

Copy link
Copy Markdown
Member

Problem

Measured 19.11.2026 on the production node (api.viz.world): anon memory stepped +7.25 GB in one minute and the node was OOM-killed 47 minutes later.

Cause, measured A/B with a single variable (same client, same image, 86 000 block headers in batches of 2000):

stand MALLOC_ARENA_MAX anon growth
vizprobe3 2 +67 MiB
vizprobe3u not set (256 arenas) +9 494 MiB

Elements of one JSON-RPC batch are handed to different pool threads, each with its own malloc arena; a grown arena is never returned by glibc. Production runs without the variable (143–233 arenas).

Setting MALLOC_ARENA_MAX in the container environment stops the growth but (a) leaves what is already held and (b) makes the node's memory behaviour depend on how someone started the container.

Change

Return the free part of every arena to the OS from inside the process:

  • malloc_trim(0) every webserver-heap-trim-sec (default 60 s, 0 = off), driven by the existing 1 s stats tick — no new thread;
  • the trim logs one line with the RSS delta from /proc/self/statm, only when it released ≥ 16 MiB, so "trim ran and gave nothing back" is distinguishable from "trim did nothing" and from "trim claimed success";
  • option at 0 restores the previous behaviour exactly.

Verification

  • single-TU -fsyntax-only rc=0; the same harness on a deliberately broken TU reports an error (control)
  • end-to-end proof is on a stand: uncapped node, grow the heap with a batch burst, then the trim line + falling anon (will be posted here after CI builds the image)

Not a substitute for the client-side fix (large batches are still expensive to serve); it removes the need to configure the allocator correctly to avoid an OOM.

Draft: merge is the owner's by project policy.

…renas

Root cause measured 19.11.2026: elements of one JSON-RPC batch are handed to
several pool threads, each allocates in its own malloc arena, and glibc keeps a
grown arena for the life of the process. A single client sending 86 000 block
headers in batches of 2000 cost +67 MiB on a stand with MALLOC_ARENA_MAX=2 and
+9 494 MiB on the identical image without it; the production node (no cap,
143-233 arenas) stepped anon by +7.25 GB in one minute and was OOM-killed 47
minutes later.

Capping arenas is an environment variable: the node's memory behaviour depends
on how the container was started, and the cap stops growth without returning
what is already held. This change cleans up from inside the process:

- malloc_trim(0) every webserver-heap-trim-sec (default 60, 0 = off), driven by
  the existing 1 s stats tick so no new thread is created
- the trim logs a line with the RSS delta read from /proc/self/statm, but only
  when it released at least 16 MiB: a trim that gives nothing back must not look
  like a trim that worked (and a log line every minute with 0 would be noise)

Behaviour is otherwise unchanged; with the option at 0 the node does exactly what
it did before.

Verified: single-TU -fsyntax-only rc=0 for the touched file, the same harness on
a deliberately broken TU fails (control). End-to-end proof is the stand: grow the
heap with an uncapped node, then watch the trim line and anon fall.
@On1x

On1x commented Nov 19, 2026

Copy link
Copy Markdown
Member Author

Stand proof (19.11.2026, 21:4xZ)

Bed: a steady node with real blocks (all 10 000 responses counted, none null), then THIS image on a copy of its data, two runs with one difference — --webserver-heap-trim-sec.

batch of 2000 real blocks trim=15 s trim=0 (control)
1 +246 MiB +246 MiB
2 +479 +494
3 +710 +710
4 +940 +941
5 +1171 +1170
after 12–60 s idle +10 MiB +1152 MiB

Instrument line from the same run:

heap trim: released 1142 MiB to the OS (rss 1215 -> 72 MiB, total 1142 MiB over 1 trims)

Also measured on the same bed: one 2000-block batch costs +248 MiB, while the same 2000 blocks in ten 200-block batches cost +1 MiB — the shape of the request drives the cost, 350×.

This is the production symptom reproduced: 43 such batches in one minute (the 14:35Z step) ≈ 10 GB against the observed +7.25 GB step.

So the trim is not hygiene here — it is the cure for this class: the heap never returns the arena pages by itself, and malloc_trim releases them from inside the process, without any environment variable and without a restart.

@On1x

On1x commented Nov 19, 2026

Copy link
Copy Markdown
Member Author

Cost of the trim: measured, no latency penalty

A trim walks every arena, so the question "does it stall a live API" is fair. Measured on this image: a probe thread issues a single get_block_header every 20 ms while a parallel load grows the heap by ~1.2 GB (5 batches of 2000 blocks), and the trim fires every 10 s.

p50 p99 max
trim = 10 s 1.3 ms 1.9 ms 38.2 ms
trim = 0 (control) 1.3 ms 5.8 ms 35.1 ms

Two trims fired in that window and released 1141 MiB (rss 1223 -> 82 MiB); the worst single-call latency stayed in the tens of milliseconds, same as without the trim. Named boundary: on a heap of tens of GB the arena walk costs proportionally more, so the first production run should be read against the heap trim: released line (its timestamp shows when it ran) and against the API latency series if one exists.

@On1x

On1x commented Nov 19, 2026

Copy link
Copy Markdown
Member Author

Concurrency (production shape): also cured

The sequential A/B above is not how the production node grew. On 19.11 memory stepped while several clients sent batches simultaneously. Measured on a steady node (anon 15 MiB, own data dir, chain following), 8 concurrent clients × 2 batches × 2000 real blocks (30 000 headers, all returned):

peak after 10–40 s idle
--webserver-heap-trim-sec=10 +1967 MiB +52 MiB (2 released lines)
--webserver-heap-trim-sec=0 (control) +3844 MiB +3711 MiB

Two things this shows: concurrency is more expensive than the sequential case for the same volume (3844 vs ~1170 MiB), which matches the production step; and the trim returns 97 % of it, with a peak half the control because it fires during the burst rather than after.

Also observed: with nothing free to release the trim logs nothing and releases nothing — it only returns free pages, so it cannot disturb a node that is loading state.

@On1x
On1x marked this pull request as ready for review November 19, 2026 19:41
@On1x
On1x merged commit cf42955 into master Nov 19, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant