feat(extract): remove the autoparse fallback and add zenrows extract prepare

[zenrows-dev-pool]

Summary

zenrows extract no longer retries with the deprecated Autoparse when Extract has not prepared a domain. It reports the error with the exact command to prepare the domain, and this PR adds that command (zenrows extract prepare), so users can prepare a domain themselves and extract again.

Flow: zenrows extract <url> → REQS007 → zenrows extract prepare <domain> --url <url> --wait → zenrows extract <url>.

Breaking / behaviour changes

• The Autoparse fallback is removed. On 403 REQS007 (or the legacy 402 AUTH010), zenrows extract makes one request and reports the error with the prepare step. No code path retries with autoparse=true any more, whatever the caller passes. The adapter option fallbackAutoparse is still accepted so existing callers keep working, but it is deprecated and has no effect; when it is true, the CLI prints one deprecation warning on stderr. --autoparse is a different thing (the user asks for Autoparse directly) and is unchanged: it still runs Autoparse and warns that it is deprecated.
• fellBackToAutoparse is removed from the adapter outcome, the run artifact and the --json output, since it can never be true now.
• Stricter empty-result check. A result whose fields are all null, "" or [], at any depth, is now empty: true. Before, only {}, [], null and blank strings were empty, so page scaffolding read as data. --validate fails on such results and the plain run prints the empty-result warning. 0 and false still count as data.
• REQS007 codes renamed to say what they mean. A domain not prepared yet is now EXTRACT_DOMAIN_NOT_PREPARED (it was EXTRACT_DOMAIN_NOT_ENABLED). A domain Extract is not enabled for is EXTRACT_DOMAIN_NOT_ENABLED. The page case is the new EXTRACT_PAGE_NOT_RECOGNISED. Code that compared against the old EXTRACT_DOMAIN_NOT_ENABLED did so to fall back to Autoparse, and that fallback no longer exists, so there is nothing left for that comparison to do.

REQS007: three cases, three answers

The API answers REQS007 in three cases. It names the case in a reason field, which arrives with an upcoming API deploy. The CLI reads that field first. Until the deploy, it falls back to the problem title and detail, which the API sends today. A comment in src/core/http.ts marks that fallback for removal.

reason	Body title today	CLI error	Next step it prints
domain_not_prepared	Extract is not prepared for the requested domain (…)	EXTRACT_DOMAIN_NOT_PREPARED	the exact zenrows extract prepare <domain> --url <url> command, --wait, status, and --css meanwhile
page_not_recognised	Extract did not recognise this page on …	EXTRACT_PAGE_NOT_RECOGNISED	the API's own advice, in the API's order: try a product, article or listing page; if this page should work, retry it once in a few minutes, not in a loop; otherwise --css. Preparing again does not change it
domain_not_enabled	Extract is not enabled for the requested domain (…)	EXTRACT_DOMAIN_NOT_ENABLED	--css, or contact support to have the site enabled; preparing does not help

page_not_recognised is not a missing page kind. On a prepared domain Extract learns a new kind of page on its first request, so this answer means the page is not one Extract extracts (a home, search, login or category page), or a transient failure kept it from trying. Sending the agent to prepare would loop, because resubmitting a prepared domain returns its ready record at once. That is why a submit answering ready at once sets already_prepared: true, and why the ready next step repeats the page advice. None of the messages offer Autoparse.

zenrows extract prepare

• zenrows extract prepare <domain> --url <u> [--url <u>...] [--wait [--timeout <s>]] calls POST /v1/extract/prepared-domains (X-API-Key) and returns at once. --url takes 1 to 5 full http(s) URLs; anything else is rejected before any request. --wait polls every 30 seconds (the same interval the MCP server tells agents) until ready or failed; default timeout 900 seconds.
• zenrows extract prepare status <domain> shows status, state, roles, the failure, the monthly allowance (used / limit / remaining) and the next step.
• It never creates a Free plan account: it needs a key the user already has.
• The domain is the host, minus a leading www.; it is not reduced to the registrable domain, because Extract prepares each host on its own (verified in production, below).

Answer	Result
202 preparing	Status record, next step (prepare status …)
202 ready at once	already_prepared: true; do not loop, use --css
409 in_progress	Follows the running preparation; EXTRACT_PREPARE_IN_PROGRESS only when this account cannot read it
400 invalid_argument / 409 idempotency_conflict	EXTRACT_PREPARE_INVALID (with reason)
404 not_found (status)	EXTRACT_PREPARE_NOT_FOUND, with the prepare command
429 out_of_allowance	EXTRACT_PREPARE_OUT_OF_ALLOWANCE (with used / limit); do not retry this month, use --css
503 overloaded / disabled	EXTRACT_PREPARE_UNAVAILABLE; retry in a few minutes, do not loop
status failed	EXTRACT_PREPARE_FAILED, exit 1, with the step for the failure code; whether the attempt used an allowance slot comes from failure.slot_consumed
--wait runs out	EXTRACT_PREPARE_TIMEOUT, with the status command

Docs and skills

skills/extract, skills/zenrows, agent-plugin/skills/*, README.md, docs/capabilities.md, the competitor-intelligence workflow, the pricing template and registry/*.json teach the flow, the do-not-loop rule and the host rule. They mark Autoparse deprecated and say "any prepared domain (you can prepare one yourself)". The agent-plugin extract skill describes the MCP server's prepare_domain / prepared_domain_status tools, which ship in the matching zenrows-mcp PR.

Production smoke test of the fallback removal

Local build of this branch against https://api.zenrows.com, key from the environment, every request to the API counted. Target: https://web-scraping.dev/product/1, a domain that is not prepared (absent from GET /v1/extract/domains) and not forbidden (a plain fetch returns 200).

Step	Requests	Result
a) zenrows extract <url>, defaults	1 (extract=auto&mode=auto, 403)	exit 1, EXTRACT_DOMAIN_NOT_PREPARED (server code REQS007), first suggested command zenrows extract prepare web-scraping.dev --url https://web-scraping.dev/product/1 --wait
b) adapter runExtract with fallbackAutoparse: true	1 (extract=auto&mode=auto, 403)	the same error, plus one stderr line: "fallbackAutoparse is deprecated and has no effect: Extract no longer falls back to Autoparse. …"

No step called prepare, so no allowance slot was used.

Verified against production

Local build of this branch against https://api.zenrows.com, no base override, key from the environment. Second pass, with the final texts. The production API does not send reason yet, so these runs also exercise the title fallback. They ran before the code rename: the same output now starts ✗ [EXTRACT_DOMAIN_NOT_PREPARED].

$ zenrows extract https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html
✗ [EXTRACT_DOMAIN_NOT_ENABLED] Extract hasn't prepared books.toscrape.com yet. Prepare it with `zenrows extract prepare books.toscrape.com --url https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html` (a few example URLs, a few minutes, no credits), then retry; meanwhile use --css.
  cause: (REQS007) Extract is not prepared for the requested domain (books.toscrape.com) (REQS007)
  next:  Run `zenrows extract prepare books.toscrape.com --url https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html --wait` (up to 5 --url pages, one per page kind you want), then retry `zenrows extract https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html`. Without --wait, check progress about every 30 seconds with `zenrows extract prepare status books.toscrape.com`. Meanwhile extract with --css and explicit selectors.
  $ zenrows extract prepare books.toscrape.com --url https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html --wait
  $ zenrows extract prepare status books.toscrape.com
  $ zenrows extract https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html --css '{"title":"h1"}'

$ zenrows extract prepare scrapingcourse.com --url https://www.scrapingcourse.com/ecommerce/ --wait
✓ scrapingcourse.com: ready
scrapingcourse.com is already prepared, so nothing new was submitted: run the extract (`zenrows extract https://www.scrapingcourse.com/ecommerce/`). If the extract then fails with EXTRACT_PAGE_NOT_RECOGNISED, Extract did not recognise that page. Try a product, article or listing page on the same site instead. If this page should work, retry it once in a few minutes; do not retry in a loop. Otherwise use --css with explicit selectors. Preparing the domain again does not change this.

$ zenrows extract prepare status sandbox.oxylabs.io
✗ [EXTRACT_PREPARE_NOT_FOUND] This account has not submitted sandbox.oxylabs.io for preparation.
  cause: (not_found) This domain has never been submitted.
  next:  Prepare it first: zenrows extract prepare sandbox.oxylabs.io --url <page url> (1 to 5 --url).

The already-prepared submit used no allowance slot: the allowance stayed at 1 of 10 before and after.

The page_not_recognised REQS007 still does not reproduce in production: search, login, cart and listing pages on prepared domains all came back 200 with data. That path is covered by tests that use the API's body, with and without reason.

From the first pass, still valid:

• Hosts are keyed separately: pt.wikipedia.org answers REQS007 although es.wikipedia.org is ready, and www.scrapethissite.com is stored as scrapethissite.com.
• End to end on scrapeme.live: REQS007, then prepare --wait reached ready in 2 min 31 s, then the extract returned 200 with data. That run used one slot.

Ordering

Release this only after the API deploy that adds the reason field to REQS007 is live in production. The client reads reason first and keeps a title match only for an API without it, but the docs describe reason, and the docs and both client releases should go out after the API carries it.

Release

No version change here. Per docs/contributing.md, release-please sets the version from the feat(extract) title after merge.

Tests

• npm run typecheck and npm run build: clean. The repo has no linter beyond tsc.
• npm test: 394 passed, 0 failed.
  • fallbackAutoparse: true on REQS007 and on AUTH010: one request, EXTRACT_DOMAIN_NOT_PREPARED with the prepare command, and exactly one deprecation warning on stderr. No warning without the option or with it false.
  • All three REQS007 cases, read from reason and, separately, from the title only. Each test checks the advice order for page_not_recognised, and that no prepare command and no Autoparse appear where they must not.
  • refusalReason prefers the field, and ignores a reason it does not know.
  • prepare: 202 preparing, ready at once (already_prepared), --wait ending in ready, failed and timeout, 409 followed and 409 unreadable, 400, 429, 503 overloaded and disabled, --url validation, no automatic account creation.
  • status with and without the allowance, and on 404.
  • Every failure code, with and without slot_consumed.
  • The empty-result rule, the capability notes, and routing from zenrows extract.