Repository navigation
webserver: return free heap pages to the OS (malloc_trim), not only cap arenas - #287
Conversation
…renas Root cause measured 19.11.2026: elements of one JSON-RPC batch are handed to several pool threads, each allocates in its own malloc arena, and glibc keeps a grown arena for the life of the process. A single client sending 86 000 block headers in batches of 2000 cost +67 MiB on a stand with MALLOC_ARENA_MAX=2 and +9 494 MiB on the identical image without it; the production node (no cap, 143-233 arenas) stepped anon by +7.25 GB in one minute and was OOM-killed 47 minutes later. Capping arenas is an environment variable: the node's memory behaviour depends on how the container was started, and the cap stops growth without returning what is already held. This change cleans up from inside the process: - malloc_trim(0) every webserver-heap-trim-sec (default 60, 0 = off), driven by the existing 1 s stats tick so no new thread is created - the trim logs a line with the RSS delta read from /proc/self/statm, but only when it released at least 16 MiB: a trim that gives nothing back must not look like a trim that worked (and a log line every minute with 0 would be noise) Behaviour is otherwise unchanged; with the option at 0 the node does exactly what it did before. Verified: single-TU -fsyntax-only rc=0 for the touched file, the same harness on a deliberately broken TU fails (control). End-to-end proof is the stand: grow the heap with an uncapped node, then watch the trim line and anon fall.
Stand proof (19.11.2026, 21:4xZ)Bed: a steady node with real blocks (all 10 000 responses counted, none null), then THIS image on a copy of its data, two runs with one difference —
Instrument line from the same run: Also measured on the same bed: one 2000-block batch costs +248 MiB, while the same 2000 blocks in ten 200-block batches cost +1 MiB — the shape of the request drives the cost, 350×. This is the production symptom reproduced: 43 such batches in one minute (the 14:35Z step) ≈ 10 GB against the observed +7.25 GB step. So the trim is not hygiene here — it is the cure for this class: the heap never returns the arena pages by itself, and |
Cost of the trim: measured, no latency penaltyA trim walks every arena, so the question "does it stall a live API" is fair. Measured on this image: a probe thread issues a single
Two trims fired in that window and released 1141 MiB ( |
Concurrency (production shape): also curedThe sequential A/B above is not how the production node grew. On 19.11 memory stepped while several clients sent batches simultaneously. Measured on a steady node (
Two things this shows: concurrency is more expensive than the sequential case for the same volume (3844 vs ~1170 MiB), which matches the production step; and the trim returns 97 % of it, with a peak half the control because it fires during the burst rather than after. Also observed: with nothing free to release the trim logs nothing and releases nothing — it only returns free pages, so it cannot disturb a node that is loading state. |
Problem
Measured 19.11.2026 on the production node (
api.viz.world): anon memory stepped +7.25 GB in one minute and the node was OOM-killed 47 minutes later.Cause, measured A/B with a single variable (same client, same image, 86 000 block headers in batches of 2000):
MALLOC_ARENA_MAXvizprobe3vizprobe3uElements of one JSON-RPC batch are handed to different pool threads, each with its own malloc arena; a grown arena is never returned by glibc. Production runs without the variable (143–233 arenas).
Setting
MALLOC_ARENA_MAXin the container environment stops the growth but (a) leaves what is already held and (b) makes the node's memory behaviour depend on how someone started the container.Change
Return the free part of every arena to the OS from inside the process:
malloc_trim(0)everywebserver-heap-trim-sec(default 60 s,0= off), driven by the existing 1 s stats tick — no new thread;/proc/self/statm, only when it released ≥ 16 MiB, so "trim ran and gave nothing back" is distinguishable from "trim did nothing" and from "trim claimed success";0restores the previous behaviour exactly.Verification
-fsyntax-onlyrc=0; the same harness on a deliberately broken TU reports an error (control)anon(will be posted here after CI builds the image)Not a substitute for the client-side fix (large batches are still expensive to serve); it removes the need to configure the allocator correctly to avoid an OOM.
Draft: merge is the owner's by project policy.