Skip to content

fix: do not stop accepting connections on transient errors - #577

Open
AndreyOsipuk wants to merge 4 commits into
9seconds:masterfrom
AndreyOsipuk:fix/accept-loop-resilience-upstream
Open

AndreyOsipuk wants to merge 4 commits into
9seconds:masterfrom
AndreyOsipuk:fix/accept-loop-resilience-upstream

Conversation

@AndreyOsipuk

Copy link
Copy Markdown

mtg can stay alive but stop accepting connections until it is restarted:

  1. Proxy.Serve returns on any Accept error, and runProxy starts it as go proxy.Serve(listener) //nolint: errcheck. When a connection flood exhausts file descriptors (EMFILE), the accept loop ends while the process keeps running; process-level monitoring does not notice.
  2. utils.Listener.Accept returns an error when socket options cannot be set on one accepted connection (for example, a client that resets right after the TCP handshake), which ends the accept loop because of a single bad client.

Changes:

  • temporary accept errors (EMFILE, ENFILE, ENOBUFS, ENOMEM, ECONNABORTED, ECONNRESET, timeouts) are retried with a pause growing from 5ms to 1s, as net/http does, with a warning in the log; shutdown interrupts the pause;
  • a connection whose socket options cannot be set is closed and skipped;
  • if Serve still returns an error, runProxy shuts down and returns it, so systemd/docker restarts the process.

Tests: acceptWithRetry survives EMFILE, stops on a closed listener and on shutdown during the pause; IsTemporaryAcceptError and the delay cap; utils.Listener skips a connection with a broken socket (fails on current master). go vet, tests and GOOS=windows/darwin go build ./... pass.

The same fix for mtg-multi (which also covers its MultiListener): dolonet#46.

Proxy.Serve returned on any Accept error and run_proxy started it with
'go proxy.Serve(listener)' ignoring the result, so running out of file
descriptors under a connection flood (EMFILE) left the process alive but no
longer accepting connections. utils.Listener also returned an error when
socket options could not be set on one accepted connection (e.g. reset by the
client right away), which stopped the accept loop because of one bad client.

Temporary accept errors are now retried with a pause growing from 5ms to 1s,
like net/http; a connection with broken socket options is closed and skipped;
and if Serve still returns, runProxy shuts down and returns the error so a
supervisor restarts the process.
The listener is closed during shutdown, so Serve returns an accept error
and every stop or restart exited with code 1. Only a Serve that failed
before shutdown started is reported now.
@AndreyOsipuk

Copy link
Copy Markdown
Author

Found on our production nodes after deploying this change: an ordinary systemctl stop/restart exited with code 1, because shutdown closes the listener and the resulting accept error was reported as a failure. Fixed in the last commit: only a Serve that failed before shutdown started is reported (waitAndShutdown, with tests for both cases). Checked live: SIGTERM now exits with 0.

@dolonet

dolonet commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

@AndreyOsipuk this change also went into dolonet/mtg-multi (#46), and review there produced follow-ups in dolonet#50. Two of them apply to this PR as it stands:

  • Socket options error is now silent. utils.Listener.Accept closes and skips a connection whose options cannot be set, without a log line. If setting options starts failing for every connection (a bad option, an unsupported platform), the proxy drops everything quietly, where before it exited with an error. In the fork this is logged as a warning, at most once per 10 s with a count of suppressed lines (dolonet/mtg-multi@d82cff6).
  • New public API. mtglib.IsTemporaryAcceptError and mtglib.AcceptRetryDelay are exported from mtglib only to be shared with internal/utils. In the fork they moved to an internal package (dolonet/mtg-multi@411fe39), so mtglib's public surface does not grow.

The third follow-up there (the pause applied twice) is about the fork's multi-listener and does not apply here. Could you port the two above, so that @9seconds reviews the version that was already reviewed?

utils.Listener skipped such a connection silently. If setting the options
starts failing for every connection (a bad option, an unsupported platform),
the proxy would drop everything without a single log line, while before
it died loudly.

The skip is now logged as a warning with the error and the client address,
at most once per 10 seconds with the number of suppressed lines, so a flood
of broken connections does not flood the log. The rate limiter lives in a
small internal/logthrottle package. The listener gets its logger from
runProxy.
@codecov-commenter

codecov-commenter commented Oct 22, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.79570% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.33%. Comparing base (753d051) to head (7f6ce74).
⚠️ Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
internal/cli/run_proxy.go 40.00% 12 Missing ⚠️
internal/utils/net_listener.go 92.00% 2 Missing ⚠️
mtglib/accept.go 93.75% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #577      +/-   ##
==========================================
+ Coverage   66.65%   67.33%   +0.68%     
==========================================
  Files         110      112       +2     
  Lines        4000     4075      +75     
==========================================
+ Hits         2666     2744      +78     
+ Misses       1189     1185       -4     
- Partials      145      146       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

IsTemporaryAcceptError and AcceptRetryDelay were exported from mtglib,
but only acceptWithRetry in the same package uses them. They are now
unexported (isTemporaryAcceptError, acceptRetryDelay), so the change does
not widen the library API.
@AndreyOsipuk
AndreyOsipuk force-pushed the fix/accept-loop-resilience-upstream branch from 7200c3f to 7f6ce74 Compare October 22, 2026 14:32
@AndreyOsipuk

Copy link
Copy Markdown
Author

Ported both follow-ups from dolonet#50. The head is now 7f6ce74.

  • Socket options error is now silent (7d0e7d4, from dolonet/mtg-multi@d82cff6). utils.Listener now logs a skipped connection as a warning, with the error and the client address. It writes at most one line per 10 s, with a suppressed count. The listener gets its logger from runProxy (logger.Named("listener")). The rate limiter is a small internal/logthrottle package, the same file as in feat: opt-in per-IP limit on pending handshakes #579, so the two PRs merge cleanly in either order. Tests:
    • TestListenerSkipsConnectionWithBrokenSocket checks that the skip is logged;
    • TestListenerRateLimitsSkipWarning checks that 10 broken connections give 1 warning.
  • New public API (7f6ce74, from dolonet/mtg-multi@411fe39). Here only acceptWithRetry in mtglib uses the helpers. A shared internal package is not needed, so they are just unexported: isTemporaryAcceptError and acceptRetryDelay. mtglib's public API is unchanged by this PR.
  • The pause applied twice (dolonet/mtg-multi@4672e7b) is about the fork's multi-listener and does not apply here, as you said.

Checks: go vet ./..., go test -race ./..., GOOS=windows go build ./..., GOOS=darwin go vet ./... and golangci-lint v2.12.2 (0 issues) all pass. The touched files pass gofmt, except mtglib/proxy.go, whose import order is already unformatted on master.

CI: Test, Lint, Fuzzing and Build release artifacts are green. Docker is still running. Test vulnerabilities fails on GO-2026-6809 (crypto/x509@go1.26.9, fixed in 1.26.10). The runner takes Go 1.26.9 from its tool cache, and master's push run of be6e176 fails the same way. #579's run got 1.26.10 and passed. I re-pushed the same tree once (7200c3f -> 7f6ce74, only the commit date changed) to get another runner, but it got 1.26.9 again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants