Repository navigation
fix: do not stop accepting connections on transient errors - #577
AndreyOsipuk wants to merge 4 commits into
Conversation
Proxy.Serve returned on any Accept error and run_proxy started it with 'go proxy.Serve(listener)' ignoring the result, so running out of file descriptors under a connection flood (EMFILE) left the process alive but no longer accepting connections. utils.Listener also returned an error when socket options could not be set on one accepted connection (e.g. reset by the client right away), which stopped the accept loop because of one bad client. Temporary accept errors are now retried with a pause growing from 5ms to 1s, like net/http; a connection with broken socket options is closed and skipped; and if Serve still returns, runProxy shuts down and returns the error so a supervisor restarts the process.
The listener is closed during shutdown, so Serve returns an accept error and every stop or restart exited with code 1. Only a Serve that failed before shutdown started is reported now.
|
Found on our production nodes after deploying this change: an ordinary |
|
@AndreyOsipuk this change also went into dolonet/mtg-multi (#46), and review there produced follow-ups in dolonet#50. Two of them apply to this PR as it stands:
The third follow-up there (the pause applied twice) is about the fork's multi-listener and does not apply here. Could you port the two above, so that @9seconds reviews the version that was already reviewed? |
utils.Listener skipped such a connection silently. If setting the options starts failing for every connection (a bad option, an unsupported platform), the proxy would drop everything without a single log line, while before it died loudly. The skip is now logged as a warning with the error and the client address, at most once per 10 seconds with the number of suppressed lines, so a flood of broken connections does not flood the log. The rate limiter lives in a small internal/logthrottle package. The listener gets its logger from runProxy.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #577 +/- ##
==========================================
+ Coverage 66.65% 67.33% +0.68%
==========================================
Files 110 112 +2
Lines 4000 4075 +75
==========================================
+ Hits 2666 2744 +78
+ Misses 1189 1185 -4
- Partials 145 146 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
IsTemporaryAcceptError and AcceptRetryDelay were exported from mtglib, but only acceptWithRetry in the same package uses them. They are now unexported (isTemporaryAcceptError, acceptRetryDelay), so the change does not widen the library API.
7200c3f to
7f6ce74
Compare
|
Ported both follow-ups from dolonet#50. The head is now
Checks: CI: Test, Lint, Fuzzing and Build release artifacts are green. Docker is still running. Test vulnerabilities fails on GO-2026-6809 ( |
mtg can stay alive but stop accepting connections until it is restarted:
Proxy.Servereturns on anyAccepterror, andrunProxystarts it asgo proxy.Serve(listener) //nolint: errcheck. When a connection flood exhausts file descriptors (EMFILE), the accept loop ends while the process keeps running; process-level monitoring does not notice.utils.Listener.Acceptreturns an error when socket options cannot be set on one accepted connection (for example, a client that resets right after the TCP handshake), which ends the accept loop because of a single bad client.Changes:
EMFILE,ENFILE,ENOBUFS,ENOMEM,ECONNABORTED,ECONNRESET, timeouts) are retried with a pause growing from 5ms to 1s, asnet/httpdoes, with a warning in the log; shutdown interrupts the pause;Servestill returns an error,runProxyshuts down and returns it, so systemd/docker restarts the process.Tests:
acceptWithRetrysurvivesEMFILE, stops on a closed listener and on shutdown during the pause;IsTemporaryAcceptErrorand the delay cap;utils.Listenerskips a connection with a broken socket (fails on current master).go vet, tests andGOOS=windows/darwin go build ./...pass.The same fix for mtg-multi (which also covers its
MultiListener): dolonet#46.