feat(tunnel): reap stray cloudflared processes on startup - #207
Open
luthermonson wants to merge 1 commit into
Open
luthermonson wants to merge 1 commit into
luthermonson wants to merge 1 commit into
Conversation
#206 stopped ephemerd leaking a cloudflared per restart on Windows, but a Job Object only governs children THIS binary spawns. It cannot adopt one already running, so orphans from an older binary — or from a bind that failed — survive until the host reboots. Every Windows node upgrading from a pre-#206 build still carries its whole backlog. Those orphans are not idle. Each keeps re-registering edge connections for the SAME tunnel, and past some count Cloudflare starts refusing. mfl-win-amd64-102 held 25 of them across 96 edge connections and logged ERR ... unknown error registering the connection roughly 300 times an hour for three days, while the live daemon ran on 3 of its 4 connections. Killing them dropped that to ~0 within minutes. Cloudflare load-balances across every connection registered for a tunnel, so a webhook handed to an orphan is a webhook the live daemon never sees — a plausible mechanism for the webhook wedges seen after upgrades. start() now sweeps before spawning. Matching is by full IMAGE PATH, not process name, so only cloudflared running out of THIS ephemerd's data dir is touched and an operator's own tunnel is left alone. Processes parented to us are skipped, so a restart never kills its own live child. No-op off Windows: Pdeathsig makes an orphan impossible to create on Linux, and darwin does not host a tunnel. Tests run for real on Windows and had to spawn genuine orphans via a launcher that exits — spawning directly makes them our own children, which the reaper deliberately spares, so the obvious setup silently tests nothing. Revert-verified twice: a no-op reaper leaves the stray alive, and matching on name instead of path kills an unrelated cloudflared.
|
ePHPm Preview — deployed (health check pending)
Preview updates automatically on each push to this PR. |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#206 stopped ephemerd leaking a cloudflared per restart on Windows — but a Job Object only governs children this binary spawns. It cannot adopt one already running, so orphans from an older binary (or from a bind that failed) survive until the host reboots. Every Windows node upgrading from a pre-#206 build still carries its entire backlog.
They are not idle
Each orphan keeps re-registering edge connections for the same tunnel, and past some count Cloudflare starts refusing.
mfl-win-amd64-102held 25 of them across 96 edge connections and logged:~300 times an hour for three days, while the live daemon ran on 3 of its 4 connections. Reaping them dropped it to ~0 within minutes:
Cloudflare load-balances across every connection registered for a tunnel, so a webhook handed to an orphan is one the live daemon never sees. That is a plausible mechanism for the webhook wedges seen after upgrades.
Fix
start()sweeps before spawning.Pdeathsigmakes an orphan impossible to create on Linux, and darwin does not host a tunnel.Testing
Runs for real on Windows. The tests had to spawn genuine orphans via a launcher that exits: spawning directly makes them our own children, which the reaper deliberately spares — so the obvious setup silently tests nothing. My first version did exactly that and passed while proving nothing.
Revert-verified twice:
stray pid 41292 survived the reapkilled an unrelated cloudflared ... matching is not path-scopedBuilds and vets clean for linux, windows, darwin.