Skip to content

fix(client): surface HTTP status, bound requests, and type API errors - #29

Open
karaposu wants to merge 1 commit into
brightdata:mainfrom
karaposu:fix/http-boundary-hardening
Open

fix(client): surface HTTP status, bound requests, and type API errors#29
karaposu wants to merge 1 commit into
brightdata:mainfrom
karaposu:fix/http-boundary-hardening

Conversation

@karaposu

@karaposu karaposu commented Aug 27, 2026

Copy link
Copy Markdown

Three defects in src/utils/client.ts — the one function every API call in the CLI goes through. They're bundled because they're the same root cause (the boundary hides what it knows) and touch the same 30 lines.

1. The client discards the HTTP status, so polling guesses

On success the client returned only the parsed body. 202 is res.ok, so a caller could not distinguish "job accepted, still building" from "here is your data" — both arrived as a body with no status attached.

Snapshot polling therefore infers readiness from the body's shape (extract_status + RUNNING_STATUSES). That works while the body is a parsed object, i.e. --format json. It silently fails for --format csv|ndjson|jsonl, where the client returns text:

const extract_status = (result: unknown)=>{
    if (!result || typeof result != 'object')   // a string body exits here
        return undefined;                        // → "not running" → treated as data

So a not-ready snapshot requested as JSONL gets printed as if it were the dataset, and the CLI exits 0. Silent wrong data is the worst outcome for exactly the ETL workflows those formats exist to serve.

Fix: an opt-in get_with_status() returning {status, headers, body}. request() / get() / post() still return the body, so no existing call site changes — only code that must reason about the protocol opts in. Readiness now checks, in order of authority: HTTP 202 → parsed-object status → text body that parses to a status. The predicate returns the real status string, so progress output still shows startingbuildingrunning instead of collapsing to one token.

Headers are exposed too — that's the plumbing a future Retry-After-aware backoff needs.

2. No request timeout — the CLI could hang forever

fetch() has no default timeout and was called without a signal. A black-holed connection (VPN drop, hung LB) produced no error at all, so the retry loop never engaged: spinner spinning, no output, Ctrl-C the only exit.

Fix: AbortSignal.timeout per attempt (fresh signal each time — hoisting it would abort retries instantly), default 120s, overridable via Request_opts.timeout_ms.

Two deliberate choices worth reviewing:

  • 120s, not 30s. The abort must bound hung connections without cutting off slow-but-alive work — protected-site scrapes and large snapshot bodies legitimately run long. Happy to lower it if you have real numbers on the long tail.
  • A timed-out attempt retries once, not 3×. Reusing the generic retry budget would mean 4 × 120s ≈ 8 minutes of silent stall — multiplying the very wait this fix exists to bound. The retry is also narrated rather than silent.

3. Retry was decided by error message prose

if (e instanceof Error && e.message.startsWith('Error:'))
    throw e;   // treat as final API error, don't retry

Retry semantics were coupled to message wording: rewording a template silently changes behavior, and a network failure whose message happens to start with Error: was misclassified as final and never retried.

Fix: a Client_api_error class (carrying status and hint) discriminated with instanceof. Message bytes are unchanged — scraper-studio matches on error prose (REALTIME_LIMIT_MARKER = 'realtime job limit', clean_error_message), so that contract is preserved and now has a test pinning it.

Testing

  • 22 new tests, 395 total. The request() loop had no test coverage at all before this — these are the first.
  • The readiness predicate is table-tested across every combination of {200, 202} × {object body, text body}, because that matrix is precisely where the silent-corruption case lives.
  • vi.mocks utils/config so URL assertions don't depend on the developer's local config.json (load_config() runs on every request and can override api_url).
  • Verified by running the built CLI.

Scope

Deliberately not included: retry policy/backoff changes (that's #20's area), migrating other call sites (status.ts is correctly excluded — for /datasets/v3/progress the status metadata is the payload), and retiring scraper.ts's parallel fetch_raw helper. That helper exists only because the shared client couldn't expose status codes; get_with_status removes its reason to exist, but that's a follow-up, not this diff.

Independent of #28 (Node 20 fix) — different files, no conflicts, either can merge first.

Note: this PR reports no CI checks because main has no CI workflow yet (#28 adds one — type-check + tests on a Node 20.17.0 / 24 matrix). Verified locally instead: tsc --noEmit clean, 391 tests passing, and the built CLI smoke-run.

Three defects in src/utils/client.ts, the function every API call goes
through:

1. Protocol erasure. On success the client returned only the parsed body
   and discarded the status code, so a caller could not tell HTTP 202
   (job accepted, still building) from 200 (this is your data) — 202 is
   res.ok, so both looked identical. Snapshot polling therefore inferred
   readiness from the body's *shape*. That misses the not-ready case
   whenever the body is text, which is exactly what --format csv/ndjson/
   jsonl produce: the status stub is printed as if it were the dataset
   and the CLI exits 0. Silent wrong data is the worst failure mode for
   the ETL use case these formats exist for.

2. No request timeout. fetch() has no default timeout and was called
   without a signal, so a black-holed connection (VPN drop, hung load
   balancer) hung forever — no error ever arrived for the retry loop to
   react to. The only way out was Ctrl-C.

3. Retry decided by message prose. The catch block told API errors from
   network errors with message.startsWith('Error:'), so rewording an
   error template silently changed retry behavior, and a network failure
   worded that way was misclassified as final and never retried.

Changes:
- Add Response_envelope {status, headers, body} and an opt-in
  get_with_status(). request()/get()/post() keep returning the body, so
  all existing call sites are untouched; only pollers that must reason
  about the protocol opt in. Headers are exposed too, which is what a
  future Retry-After-aware backoff would need.
- Add Client_api_error (carrying status and hint) and discriminate with
  instanceof. Message bytes are unchanged — scraper-studio matches on
  error prose (e.g. 'realtime job limit'), so that contract is preserved
  and now covered by a test.
- Abort each attempt with AbortSignal.timeout (default 120s, override via
  Request_opts.timeout_ms). The default is deliberately generous so it
  cannot cut off slow-but-alive work such as protected-site scrapes or
  large snapshot downloads. A timed-out attempt retries at most once
  rather than reusing the 3-retry budget, because timeout x attempts
  multiplies the stall the fix exists to bound, and the retry is narrated
  instead of being silent.
- Migrate the pipelines snapshot poller to decide readiness from the
  protocol first, keeping body-shape checks as fallback and adding the
  text-body case that the object-only check missed. The predicate returns
  the real status string, so progress output still distinguishes
  starting/building/running.

Tests: 22 new (395 total). The client request loop had no coverage at all
before this; the readiness predicate is table-tested across every
combination of {200, 202} x {object body, text body} because that matrix
is where the silent-corruption case lives.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant