Skip to content

feat(client): retry uncharged failures by default, with full-jitter backoff and a time budget - #58

Draft
gonzalo-zenrows wants to merge 1 commit into
mainfrom
feat/client-retry-jitter
Draft

gonzalo-zenrows wants to merge 1 commit into
mainfrom
feat/client-retry-jitter

Conversation

@gonzalo-zenrows

Copy link
Copy Markdown

What changed

fetch(), extract() and post() now share one retry policy, the same one the Python SDK, Go SDK and CLI are adopting in parallel PRs:

  • Retried: 429, 502, 503 and 504 (the API never charges these), and network errors before any response.
  • Never retried: a timeout or abort (the server may still finish the request and charge it), and every other status. 422 is no longer retried: the API no longer sends a retryable 422. no_data_extracted was already final; now every 422 is.
  • Wait between tries: the server's Retry-After in whole seconds, capped at 30 s. Otherwise full jitter: uniform(0.25 s, min(retryMaxWait, 1 s * 2^attempt)), where attempt is the 0-based retry index (first retry 0.25–1 s, second 0.25–2 s). An HTTP-date Retry-After falls back to jitter.
  • Budget: no retry starts if the time spent since the first try plus the next wait would pass retryBudget. The last response or error is returned as-is.
  • The extract() fallback to autoparse on REQS007 / AUTH010 is unchanged: still one extra request (which gets its own retries, as before).

It stays on fetch-retry, with no second retry layer: the policy lives in src/retry.ts and is passed per request as retryOn / retryDelay, so each request has its own clock and budget. A retried response's body is cancelled so its connection is freed.

New defaults

Option Before Now
retries 0 2
retryMaxWait (seconds) — 30
retryBudget (seconds) — 90

The retries default is a public behaviour change, so this is a feat (minor release with a changelog entry via release-please). Callers who already set retries get the new wait curve and status set.

How to opt out

const client = new ZenRows(apiKey, { retries: 0 });

Tests

pnpm check is clean. pnpm test -- --run: 403 passed, 6 skipped (e2e). Retry tests use fake timers and an injectable clock and random source, so they run in milliseconds:

  • tests/retries.test.ts: retried and non-retried statuses, network error vs abort/timeout, jitter bounds per attempt, retryMaxWait cap, Retry-After honoured and capped at 30 s, non-numeric Retry-After falls back to jitter, budget respected (unit and end to end), no retry on 400/422, autoparse fallback unchanged.
  • tests/retry-count.test.ts: default makes 3 calls on a 503; retries: 0 makes 1.
  • tests/no-data-extracted.test.ts: a non-no_data_extracted 422 is now not retried.
  • tests/index.test.ts: two exact-argument assertions on the wrapped fetch now use objectContaining, since each call also carries retryOn / retryDelay.

README updated to describe the policy and the new options.

Not covered

  • The Batch and Crawl clients keep their own retry logic; this PR does not change them.
  • Not exercised against the live API.

… and a time budget

A 429, 502, 503 or 504 from the API is never charged and means "try again", so the
client now retries those (and network errors before any response) by default, waiting
the server's Retry-After or a full-jitter backoff instead of a fixed doubling delay.
A 422 is no longer retried: the API no longer sends a retryable 422. A timeout or abort
is never retried, because the server may still finish and charge the request.

- retries defaults to 2 (was 0); retries: 0 turns retries off
- new retryMaxWait (seconds, default 30) caps one backoff wait
- new retryBudget (seconds, default 90): no retry starts past it
- Retry-After in whole seconds is honoured, capped at 30 s
@gonzalo-zenrows

Copy link
Copy Markdown
Author

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant