Repository navigation
Retry a query once when MySQL closed it for idling - #530
Merged
Merged
Conversation
Prod MySQL closes a connection after 60 seconds without a query (wait_timeout = 60), and mysql2 drops an idle pool connection only after its own idleTimeout, which also defaults to 60 seconds. On that tie the pool hands out a connection the server already closed, and the query fails with errno 4031. Between Oct 16 and 20 prod logged it 6 times, twice on PR saves (ifixit#63906 and an ops PR) that only landed on a later update. Dropping idle connections at 30 seconds, half the server's limit, means the pool always lets go first. TCP keepalive wouldn't help: MySQL counts time without queries, not time without packets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review found the idleTimeout change did nothing: mysql2 runs its idle check only while maxIdle is below connectionLimit, and both default to 10. Turning the check on keeps a timer alive, so every test and bin that imports the DB without db.end() hangs. Instead, retry once when a query fails with errno 4031. The server closed the connection for idling before the query arrived, so the query never ran, and the pool discards that connection on the error: the retry gets a fresh one, writes included. Nothing else is retried; a connection lost mid-query may have run its write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
jarstelfox
changed the title
mysql2 ignores idleTimeout while maxIdle equals connectionLimit, so the setting and its comment described something that didn't happen. The retry on errno 4031 is the fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖
Prod MySQL closes a connection after 60 seconds without a query (
@@wait_timeout = 60), and the pool only finds out when the next query on that connection fails:Prod logged it 6 times between Oct 16 and 20. Two were PR saves (ifixit#63906 and an ops PR) that only landed on a later update.
db.querynow retries once on errno 4031. The server closed the connection before the query arrived, so the query never ran, and the pool discards that connection on the error. So the retry gets a fresh connection and is safe for writes too. Nothing else is retried: a connection lost mid-query might have run its write.Why not mysql2's
idleTimeout(this PR's first commit)Review found that setting
idleTimeoutalone does nothing. mysql2 runs its idle check only whilemaxIdleis belowconnectionLimit(lib/base/pool.js), and both default to 10. Turning the check on starts a timer that keeps Node alive, so every test and bin that imports the DB withoutdb.end()hangs. The suite hung when I tried it.test/db-retry.test.js: errno 4031 runs the query once more; another error, or a second 4031, is passed on.🤖 Generated with Claude Code