fix(client): surface HTTP status, bound requests, and type API errors - #29
Open
karaposu wants to merge 1 commit into
Open
fix(client): surface HTTP status, bound requests, and type API errors#29karaposu wants to merge 1 commit into
karaposu wants to merge 1 commit into
Conversation
Three defects in src/utils/client.ts, the function every API call goes
through:
1. Protocol erasure. On success the client returned only the parsed body
and discarded the status code, so a caller could not tell HTTP 202
(job accepted, still building) from 200 (this is your data) — 202 is
res.ok, so both looked identical. Snapshot polling therefore inferred
readiness from the body's *shape*. That misses the not-ready case
whenever the body is text, which is exactly what --format csv/ndjson/
jsonl produce: the status stub is printed as if it were the dataset
and the CLI exits 0. Silent wrong data is the worst failure mode for
the ETL use case these formats exist for.
2. No request timeout. fetch() has no default timeout and was called
without a signal, so a black-holed connection (VPN drop, hung load
balancer) hung forever — no error ever arrived for the retry loop to
react to. The only way out was Ctrl-C.
3. Retry decided by message prose. The catch block told API errors from
network errors with message.startsWith('Error:'), so rewording an
error template silently changed retry behavior, and a network failure
worded that way was misclassified as final and never retried.
Changes:
- Add Response_envelope {status, headers, body} and an opt-in
get_with_status(). request()/get()/post() keep returning the body, so
all existing call sites are untouched; only pollers that must reason
about the protocol opt in. Headers are exposed too, which is what a
future Retry-After-aware backoff would need.
- Add Client_api_error (carrying status and hint) and discriminate with
instanceof. Message bytes are unchanged — scraper-studio matches on
error prose (e.g. 'realtime job limit'), so that contract is preserved
and now covered by a test.
- Abort each attempt with AbortSignal.timeout (default 120s, override via
Request_opts.timeout_ms). The default is deliberately generous so it
cannot cut off slow-but-alive work such as protected-site scrapes or
large snapshot downloads. A timed-out attempt retries at most once
rather than reusing the 3-retry budget, because timeout x attempts
multiplies the stall the fix exists to bound, and the retry is narrated
instead of being silent.
- Migrate the pipelines snapshot poller to decide readiness from the
protocol first, keeping body-shape checks as fallback and adding the
text-body case that the object-only check missed. The predicate returns
the real status string, so progress output still distinguishes
starting/building/running.
Tests: 22 new (395 total). The client request loop had no coverage at all
before this; the readiness predicate is table-tested across every
combination of {200, 202} x {object body, text body} because that matrix
is where the silent-corruption case lives.
karaposu
force-pushed
the
fix/http-boundary-hardening
branch
from
August 27, 2026 07:21
1bf0d98 to
c9f5a0d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three defects in
src/utils/client.ts— the one function every API call in the CLI goes through. They're bundled because they're the same root cause (the boundary hides what it knows) and touch the same 30 lines.1. The client discards the HTTP status, so polling guesses
On success the client returned only the parsed body. 202 is
res.ok, so a caller could not distinguish "job accepted, still building" from "here is your data" — both arrived as a body with no status attached.Snapshot polling therefore infers readiness from the body's shape (
extract_status+RUNNING_STATUSES). That works while the body is a parsed object, i.e.--format json. It silently fails for--format csv|ndjson|jsonl, where the client returns text:So a not-ready snapshot requested as JSONL gets printed as if it were the dataset, and the CLI exits 0. Silent wrong data is the worst outcome for exactly the ETL workflows those formats exist to serve.
Fix: an opt-in
get_with_status()returning{status, headers, body}.request()/get()/post()still return the body, so no existing call site changes — only code that must reason about the protocol opts in. Readiness now checks, in order of authority: HTTP 202 → parsed-object status → text body that parses to a status. The predicate returns the real status string, so progress output still showsstarting→building→runninginstead of collapsing to one token.Headers are exposed too — that's the plumbing a future
Retry-After-aware backoff needs.2. No request timeout — the CLI could hang forever
fetch()has no default timeout and was called without a signal. A black-holed connection (VPN drop, hung LB) produced no error at all, so the retry loop never engaged: spinner spinning, no output, Ctrl-C the only exit.Fix:
AbortSignal.timeoutper attempt (fresh signal each time — hoisting it would abort retries instantly), default 120s, overridable viaRequest_opts.timeout_ms.Two deliberate choices worth reviewing:
3. Retry was decided by error message prose
Retry semantics were coupled to message wording: rewording a template silently changes behavior, and a network failure whose message happens to start with
Error:was misclassified as final and never retried.Fix: a
Client_api_errorclass (carryingstatusandhint) discriminated withinstanceof. Message bytes are unchanged — scraper-studio matches on error prose (REALTIME_LIMIT_MARKER = 'realtime job limit',clean_error_message), so that contract is preserved and now has a test pinning it.Testing
request()loop had no test coverage at all before this — these are the first.{200, 202} × {object body, text body}, because that matrix is precisely where the silent-corruption case lives.vi.mocksutils/configso URL assertions don't depend on the developer's localconfig.json(load_config()runs on every request and can overrideapi_url).Scope
Deliberately not included: retry policy/backoff changes (that's #20's area), migrating other call sites (
status.tsis correctly excluded — for/datasets/v3/progressthe status metadata is the payload), and retiringscraper.ts's parallelfetch_rawhelper. That helper exists only because the shared client couldn't expose status codes;get_with_statusremoves its reason to exist, but that's a follow-up, not this diff.Independent of #28 (Node 20 fix) — different files, no conflicts, either can merge first.
Note: this PR reports no CI checks because
mainhas no CI workflow yet (#28 adds one — type-check + tests on a Node 20.17.0 / 24 matrix). Verified locally instead:tsc --noEmitclean, 391 tests passing, and the built CLI smoke-run.