Skip to content

zulip: Retry requests over stale keep-alive connections. - #947

Open
harshvardhan178251 wants to merge 1 commit into
zulip:mainfrom
harshvardhan178251:fix-761-stale-connection
Open

harshvardhan178251 wants to merge 1 commit into
zulip:mainfrom
harshvardhan178251:fix-761-stale-connection

Conversation

@harshvardhan178251

Copy link
Copy Markdown
Collaborator

Fixes: #761

Some networks silently kill idle TCP connections (e.g. GitHub Actions' ~270s idle timeout). The requests session Zulip uses pools keep-alive connections, so the next request over such a stale pooled connection fails with a connection error even though the server is healthy; that is what was happening in the referenced oneclick deployments. urllib3's default retry policy does not retry non-idempotent methods on connection errors, so the error propagated to callers.

This PR mounts an HTTPAdapter with a urllib3 Retry (3 attempts, 0.5s backoff, connection/read errors only, HTTP statuses untouched) on http:// and https:// in Client.ensure_session(). The retry covers the stale-connection class of connection errors; status-code handling and the existing error_retry/has_connected logic in do_api_query are unchanged. allowed_methods=None (retrying POST included) because over a stale pooled connection the request is never delivered.

One design decision worth flagging for review: if a request reaches the server but the response is lost, a retried POST could be delivered twice (e.g. a duplicate message). This is the same trade-off the issue describes for "a transparent retry for this class of connection error", and the failure mode is rare and self-limiting; alternative would be to scope retries to idempotent methods only, which would not fix the common case (Zulip API calls are mostly POSTs). Happy to narrow the retry policy if maintainers prefer.

How did you test this PR?

  • New unit test zulip/tests/test_session_retries.py asserts the mounted adapter's retry configuration for both schemes.
  • Synthetic local server that answers one request then force-closes the pooled keep-alive connection (SO_LINGER RST); the next request over the pooled connection now succeeds.
  • Full existing test suite for the zulip package passes locally (13 passed). ruff check and mypy clean. (zulint/tools/lint requires Unix executables and was run via ruff/mypy directly; zulip_bots/ has 41 pre-existing failures due to missing optional third-party bot dependencies, unrelated to this change.)
  • Lint environment note: dependency pins in requirements.txt (e.g. #egg=zulint==1.0.0) are rejected by current pip; installed with equivalent requirements locally without modifying the repo.
Self-review checklist
  • Self-reviewed the changes for clarity and maintainability (variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

  • Explains differences from previous plans (e.g., issue description).
  • Highlights technical choices and bugs encountered.
  • Calls out remaining decisions and concerns.
  • Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

  • Each commit is a coherent idea.
  • Commit message(s) explain reasoning and motivation for changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve error handling when a request fails due to the persistent connection being expired

2 participants