Skip to content

image: bake MALLOC_ARENA_MAX=2 into the runtime image - #291

Merged
On1x merged 1 commit into
masterfrom
fix/arena-max-in-image
Nov 20, 2026
Merged

On1x merged 1 commit into
masterfrom
fix/arena-max-in-image

Conversation

@On1x

@On1x On1x commented Nov 20, 2026

Copy link
Copy Markdown
Member

Why

The glibc arena cap has been living in the launch command (-e MALLOC_ARENA_MAX=2), not in the image. That is the wrong place for it:

  • it is invisible — docker inspect shows it, but nothing in the node's own logs does;
  • it disappears silently — anyone recreating the container from an older command line drops the flag, and the leak comes back with no trace. On 19.11.2026 the testnet stand lived 16.7 h without it and reached 9.05 GB;
  • it makes every validator read an announcement and edit their command line, for something that is a property of how this binary wants to be run.

Evidence

malloc_info on a production node, before and after the cap:

before after
arenas 274 2
taken from the OS 1 149 MB 71 MB
free chunks 1 074 MB (93 %) 16 MB (23 %)
live objects 76 MB 56 MB

Consequence of the "before" state: three OOM kills in one day (35.4 / 16.9 / 14.6 GB) on a node whose live data is tens of megabytes; each kill costs the network a slot. After the cap: zero OOM.

Cost on real traffic: batch of 200 get_block_header calls — 49–72 ms, unchanged (p50 16 ms vs 17 ms before).

Change

ENV MALLOC_ARENA_MAX=2 in the runtime stage of both Dockerfiles (production and testnet), next to the existing KILL_PROCESS_TIMEOUT defaults, with the reasoning written down where the next person will look for it.

PID 1 of the image is my_init, which passes its environment to the runit service, so vizd inherits the value. An explicit -e MALLOC_ARENA_MAX=<n> on the command line still takes precedence — operators who need a different value keep that door.

After merge

The flag becomes optional in the announcement: image first, -e only if you want a different value. Verified by docker exec <c> sh -c 'tr "\0" "\n" < /proc/1/environ | grep MALLOC' on a container started without -e.

The arena cap was an operator's duty: an `-e MALLOC_ARENA_MAX=2` flag in the
docker run command line. That is exactly the kind of measure that disappears
silently - recreating the container from an older command line drops the flag,
and nothing in any log says so (on 19.11 the testnet stand lived 16.7 h without
it and reached 9.05 GB).

Measured on a production node with malloc_info: 274 arenas holding 1149 MB, 93%
of it free chunks, while live objects were 76 MB; the host ran out of memory and
the node was OOM-killed three times in a day. With the cap: 2 arenas, 71 MB, no
OOM since. Cost on real traffic is nil (batch of 200 calls: 49-72 ms, unchanged).

Setting it as ENV in the runtime stage means PID 1 (my_init) carries it into the
runit service and the daemon inherits it. An explicit -e on the command line
still overrides, so operators who need another value keep that door.
@On1x
On1x merged commit e18f29a into master Nov 20, 2026
8 checks passed
@On1x
On1x deleted the fix/arena-max-in-image branch November 20, 2026 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant