Repository navigation
image: bake MALLOC_ARENA_MAX=2 into the runtime image - #291
Merged
Merged
Conversation
The arena cap was an operator's duty: an `-e MALLOC_ARENA_MAX=2` flag in the docker run command line. That is exactly the kind of measure that disappears silently - recreating the container from an older command line drops the flag, and nothing in any log says so (on 19.11 the testnet stand lived 16.7 h without it and reached 9.05 GB). Measured on a production node with malloc_info: 274 arenas holding 1149 MB, 93% of it free chunks, while live objects were 76 MB; the host ran out of memory and the node was OOM-killed three times in a day. With the cap: 2 arenas, 71 MB, no OOM since. Cost on real traffic is nil (batch of 200 calls: 49-72 ms, unchanged). Setting it as ENV in the runtime stage means PID 1 (my_init) carries it into the runit service and the daemon inherits it. An explicit -e on the command line still overrides, so operators who need another value keep that door.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The glibc arena cap has been living in the launch command (
-e MALLOC_ARENA_MAX=2), not in the image. That is the wrong place for it:docker inspectshows it, but nothing in the node's own logs does;Evidence
malloc_infoon a production node, before and after the cap:Consequence of the "before" state: three OOM kills in one day (35.4 / 16.9 / 14.6 GB) on a node whose live data is tens of megabytes; each kill costs the network a slot. After the cap: zero OOM.
Cost on real traffic: batch of 200
get_block_headercalls — 49–72 ms, unchanged (p50 16 ms vs 17 ms before).Change
ENV MALLOC_ARENA_MAX=2in the runtime stage of both Dockerfiles (production and testnet), next to the existingKILL_PROCESS_TIMEOUTdefaults, with the reasoning written down where the next person will look for it.PID 1 of the image is
my_init, which passes its environment to the runit service, sovizdinherits the value. An explicit-e MALLOC_ARENA_MAX=<n>on the command line still takes precedence — operators who need a different value keep that door.After merge
The flag becomes optional in the announcement: image first,
-eonly if you want a different value. Verified bydocker exec <c> sh -c 'tr "\0" "\n" < /proc/1/environ | grep MALLOC'on a container started without-e.