Skip to content

feat(tunnel): reap stray cloudflared processes on startup - #207

Open
luthermonson wants to merge 1 commit into
mainfrom
feat/cloudflared-reap-on-start
Open

luthermonson wants to merge 1 commit into
mainfrom
feat/cloudflared-reap-on-start

Conversation

@luthermonson

Copy link
Copy Markdown
Contributor

#206 stopped ephemerd leaking a cloudflared per restart on Windows — but a Job Object only governs children this binary spawns. It cannot adopt one already running, so orphans from an older binary (or from a bind that failed) survive until the host reboots. Every Windows node upgrading from a pre-#206 build still carries its entire backlog.

They are not idle

Each orphan keeps re-registering edge connections for the same tunnel, and past some count Cloudflare starts refusing. mfl-win-amd64-102 held 25 of them across 96 edge connections and logged:

ERR ... unknown error registering the connection

~300 times an hour for three days, while the live daemon ran on 3 of its 4 connections. Reaping them dropped it to ~0 within minutes:

2026-09-23T22  x305        2026-09-24T05  x54
2026-09-23T23  x300        2026-09-24T06  x51
2026-09-24T00  x300        2026-09-24T07  x3   <- after the reap

Cloudflare load-balances across every connection registered for a tunnel, so a webhook handed to an orphan is one the live daemon never sees. That is a plausible mechanism for the webhook wedges seen after upgrades.

Fix

start() sweeps before spawning.

  • Matching is by full image path, not process name. Only cloudflared running out of this ephemerd’s data dir is touched, so an operator’s own tunnel is safe. This is the property most worth protecting — killing someone else’s tunnel would be a worse bug than the leak.
  • Processes parented to us are skipped, so a restart never kills its own live child.
  • No-op off Windows — Pdeathsig makes an orphan impossible to create on Linux, and darwin does not host a tunnel.

Testing

Runs for real on Windows. The tests had to spawn genuine orphans via a launcher that exits: spawning directly makes them our own children, which the reaper deliberately spares — so the obvious setup silently tests nothing. My first version did exactly that and passed while proving nothing.

Revert-verified twice:

  • no-op reaper → stray pid 41292 survived the reap
  • match on name instead of path → killed an unrelated cloudflared ... matching is not path-scoped

Builds and vets clean for linux, windows, darwin.

#206 stopped ephemerd leaking a cloudflared per restart on Windows, but a
Job Object only governs children THIS binary spawns. It cannot adopt one
already running, so orphans from an older binary — or from a bind that
failed — survive until the host reboots. Every Windows node upgrading
from a pre-#206 build still carries its whole backlog.

Those orphans are not idle. Each keeps re-registering edge connections
for the SAME tunnel, and past some count Cloudflare starts refusing.
mfl-win-amd64-102 held 25 of them across 96 edge connections and logged

    ERR ... unknown error registering the connection

roughly 300 times an hour for three days, while the live daemon ran on 3
of its 4 connections. Killing them dropped that to ~0 within minutes.
Cloudflare load-balances across every connection registered for a tunnel,
so a webhook handed to an orphan is a webhook the live daemon never sees
— a plausible mechanism for the webhook wedges seen after upgrades.

start() now sweeps before spawning. Matching is by full IMAGE PATH, not
process name, so only cloudflared running out of THIS ephemerd's data dir
is touched and an operator's own tunnel is left alone. Processes parented
to us are skipped, so a restart never kills its own live child.

No-op off Windows: Pdeathsig makes an orphan impossible to create on
Linux, and darwin does not host a tunnel.

Tests run for real on Windows and had to spawn genuine orphans via a
launcher that exits — spawning directly makes them our own children,
which the reaper deliberately spares, so the obvious setup silently
tests nothing. Revert-verified twice: a no-op reaper leaves the stray
alive, and matching on name instead of path kills an unrelated
cloudflared.
@ephpm

ephpm Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

ePHPm Preview — deployed (health check pending)

URL https://ephpm-ephemerd-pr-207.preview.ephpm.dev
Framework PHP
PHP 8.5
Deployed in 62.1s

Preview updates automatically on each push to this PR.

@ephpm
ephpm Bot temporarily deployed to preview-pr-207 September 24, 2026 14:08 Inactive
@ephpm
ephpm Bot temporarily deployed to preview-pr-207 September 24, 2026 14:08 Inactive

This branch was previously deployed

1 inactive deployment
preview-pr-207 — d873026f Deployed Sep 24, 2026 by ephpm[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant