Skip to content

os-image: cocoon-agent keeps running when the OOM killer kills one of its exec children - #378

Merged
CMGS merged 1 commit into
mainfrom
m6/agent-oom-continue
Oct 23, 2026
Merged

CMGS merged 1 commit into
mainfrom
m6/agent-oom-continue

Conversation

@CMGS

@CMGS CMGS commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

The cocoon-agent.service written by os-image/base/install-agent.sh now sets OOMPolicy=continue. This matches silkd.service, which got the same line in #377.

Why

Commands run through cocoon-agent (cocoon vm exec, kubectl exec) live in cocoon-agent.service's cgroup, because KillMode=control-group keeps them there. With the default OOMPolicy=stop, one kernel OOM kill of any such command makes systemd stop the whole unit. That kills the agent and every other in-flight exec, and vsock exec fails until Restart=always brings the agent back.

The same mechanism was reproduced on silkd.service, which had the same unit shape (#377). On a 7.9 GiB guest with no swap, a growing memory hog was started through the daemon's exec:

  • The kernel kills only the hog (Killed process … (python3) … oom_score_adj:0).
  • systemd logs "A process of this unit has been killed by the OOM killer", then "Failed with result 'oom-kill'", then "Scheduled restart job". NRestarts goes up by one and MainPID changes.
  • A vsock probe gets EOF for about 2.5 s.
  • A sleep started earlier through the same daemon dies with the unit.

The same guest runs cocoon-agent.service with OOMPolicy=stop and adj 0, so a hog started through cocoon vm exec takes the agent down the same way.

With OOMPolicy=continue, the OOM killer still picks the hog, and the agent and its other execs keep running. systemctl stop still tears the cgroup down as before. The setting has no runtime cost.

Checks

  • The heredoc was extracted; systemd-analyze verify (systemd 255) on the rendered unit returns rc 0.
  • make sh-lint (shellcheck over every tracked .sh): rc 0.

@CMGS
CMGS merged commit 2654727 into main Oct 23, 2026
1 check passed
@CMGS
CMGS deleted the m6/agent-oom-continue branch October 23, 2026 06:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant