feat(crawl): add ZenRowsCrawlClient for the Crawl API

[StefanoGuerrini]

Summary

This PR adds ZenRowsCrawlClient, a typed client for the Crawl API, in a new zenrows.crawl package. The shared HTTP transport now serves both the Batch and the Crawl clients. It caps a Retry-After wait at 30 s for both clients. stop retries a 503 crawl_busy answer. The PR also adds a first-install acceptance test for the Crawl client under acceptance/crawl/.

This PR also carries #32 and #35, merged into this branch.

Squash commit message

feat(crawl): add ZenRowsCrawlClient for the Crawl API

Add the zenrows.crawl package with ZenRowsCrawlClient: create, get,
list, results, stop, content, download and wait. Parse responses into
pydantic v2 models with open enums. Raise CrawlAPIError for every
non-2xx response, and export it and the client from zenrows.

Let stop retry a 503 crawl_busy up to `retries` more times. It waits
the Retry-After value, at most 30 seconds, or 1 second when the value
is missing or unusable. Any other error still raises at once.

Let the shared HTTP transport take an error mapping, a logger and
per-request retry statuses. Cap a Retry-After wait at 30 seconds and
ignore a NaN or infinite value; this also applies to
ZenRowsBatchClient. CrawlAPIError.code is None when the body has no
code. Batch still reports a missing code as "internal", in both
ProblemDetail.code and BatchAPIError.code.

Ship a py.typed marker. Add an opt-in e2e test (make test-e2e), a
README quickstart, examples/11_crawl.py and DEVELOPMENT.md notes.

Add acceptance/crawl/, a first-install acceptance test. It builds the
wheel, installs it in a clean container, runs the README Crawl
quickstart and checks every Crawl feature against the live API.
Exclude /acceptance from the sdist. make check also lints acceptance/.

Squash-merge with this message: the branch commits carry trailer lines that should not land.

Changes since #30

This PR replaces #30. The review of the first version is here. Since that review, these things changed:

• content() names its argument content_id.
• The top-level zenrows package exports CrawlAPIError.
• The transport waits at most 30 s for a Retry-After.
• stop retries a 503 crawl_busy.
• The branch adds the acceptance test.

What this adds

Import the client from the package or from the top level: from zenrows import ZenRowsCrawlClient, CrawlAPIError. README.md and the docstrings call Crawl "(new)". README.md and the package docstring add one line that Crawl is still evolving. The client gives no runtime warning.

Method	Behavior
ZenRowsCrawlClient(api_key, *, base_url=None, timeout=30.0, retries=3, verify_ssl=True, user_agent=DEFAULT_USER_AGENT, httpx_args=None)	Default User-Agent zenrows-crawl-python/<version>. Default base URL https://api.zenrows.com/v1; override it with base_url= only. The client reads no environment variable.
create(url, *, depth, max_items=None, max_pages=None, include_patterns=None, exclude_patterns=None, output_format: Literal["html"] | None = None, idempotency_key=None) -> Crawl	Sends only the fields you set. Only the type hint checks output_format; the API decides. Without output_format, a crawl returns URLs only. A crawl stays on the start URL's registrable domain; subdomains count.
get(crawl_id, *, cursor=None, limit=None) -> CrawlWithResults	One crawl with one page of results.
list(*, cursor=None, limit=None) -> CrawlList	One page of crawls, newest first. next_cursor is None on the last page.
results(crawl_id, *, limit=None)	Follows next_cursor, limit URLs per request, and stops at the first empty page or a null cursor. It does not poll. On a running crawl it yields only what the crawl kept so far; call wait first.
stop(crawl_id) -> CrawlStop	Idempotent. Retries a 503 crawl_busy; see the next section.
content(crawl_id, content_id) -> str	One kept URL's HTML. content_id is a CrawlResult, its content_url, or a content id.
download(crawl_id) -> CrawlDownload	Opens the NDJSON file. Iterate lines for parsed DownloadLines; read .status from X-Crawl-Status. It is also a context manager. On a running crawl, status is running and the file holds what the crawl kept so far. DownloadLine.content is str | dict | None.
wait(crawl_id, *, timeout=600.0) -> Crawl	Polls with backoff (2 s, x1.5, up to 15 s) until the crawl ends or timeout passes. It never raises for the crawl's state. On timeout it returns the crawl, still running. It returns a failed crawl; read crawl.error.code and crawl.error.detail.

Retries: the client retries up to retries times on 429, 502, 503, 504 and network errors. It retries GETs, and a create that carries an idempotency_key. create never retries a 429. The client uses jittered exponential backoff and honors a Retry-After up to 30 s.

Models: responses parse into pydantic v2 models. The models ignore unknown fields. Every response enum is open: a value added later parses as UNKNOWN and keeps its raw value.

Errors: the client raises CrawlAPIError, with status_code, code, detail, problem and retry_after. code is None when the body has no code. retry_after is None when Retry-After is missing, negative, NaN or infinite. README.md lists every code:

Status	Code	Meaning
400	invalid_request, unknown_parameter, invalid_cursor	Fix the request.
403	REQS008	Crawl is not enabled for this account. Compare exc.code with the exported CRAWL_NOT_ENABLED_CODE.
404	crawl_not_found, content_not_found	Not found.
409	idempotency_request_in_flight	Retry after the first request finishes.
422	invalid_parameter, invalid_start_url, domain_not_allowed	Fix the request.
422	idempotency_key_reused	Use a new key, or no key.
429	too_many_crawls	The account reached its limit of active jobs (3 by default), shared with its Batch jobs. create never retries it.
503	crawl_busy	The API did not save the stop, and the crawl is still running. stop retries it, then raises it. A second stop is safe.

Packaging: the package now ships a py.typed marker, so type checkers use its types.

Shared transport: it now takes the error mapping and the logger as constructor arguments. It gains request() and stream() helpers and takes the retryable statuses per request. These changes apply to ZenRowsBatchClient too. Batch still reports a missing code as "internal", in both ProblemDetail.code and BatchAPIError.code. Otherwise the existing clients behave as before.

Stop on 503 crawl_busy

A stop can collide with the crawl while the crawl saves its own progress. The API retries that collision for under 2 s. If the collision stays, the API answers 503 with code: crawl_busy and Retry-After: 1. Stop is a POST without an Idempotency-Key, so the shared transport never retries it. Without this change, the caller saw an error while the crawl kept running.

Case	stop behavior
503 with code crawl_busy	Retries up to retries more times (default 3), then raises the CrawlAPIError. retries=0 turns the retry off.
Wait between tries	The Retry-After value, at most 30 s. A missing, negative, NaN or infinite value gives 1 s.
503 with another code, any other error	Raises at once.

Acceptance test

acceptance/crawl/run.sh tests the release artifact the way a new user meets it. It builds the wheel with python -m build in its own container. Then it installs the wheel in a clean, digest-pinned container with no source, caches or tokens. It runs the README.md Crawl quickstart as written, then checks 1 to 21 and a mypy --strict check. It stops every crawl it started, also on a failure or Ctrl-C. It writes acceptance/crawl/out/crawl-first-install-report.md and makes sure that no file under out/ holds the key. acceptance/crawl/README.md lists each check with its expected result. The harness runs on macOS and Linux, and make check lints it.

Input	Default
CRAWL_URL	https://www.scrapingcourse.com/ecommerce/
CRAWL_INCLUDE	/product/
CRAWL_API_BASE	https://api.zenrows.com/v1
REF	unset: this checkout
OUT_DIR	moves the output

Cost: a run starts five crawls, each with depth 1 and max_pages 10 or less. On the default CRAWL_URL, a run fetches about 10 pages, and at most 50. Check 9 can start a sixth crawl, which adds at most 10 pages. At most 2 crawls run at the same time, under the account limit of 3 active jobs. Do not run two copies against one account at the same time.

Packaging: pyproject.toml excludes /acceptance from the sdist, so the harness never ships in the package.

Tests

Test	Result or coverage
make check (ty, ruff check, ruff format --check)	Passes.
make test	Passes. It deselects the e2e test.
wait	Returns a running crawl on timeout, and returns a failed crawl.
list, results	list pages with a cursor, and results sends limit.
Transport Retry-After	3600 waits 30 s; an unusable value gives retry_after=None.
content	A caller can pass content_id= by keyword.
stop on 503 crawl_busy	crawl_busy then 200 returns after one 1.0 s wait. With retries=2 and crawl_busy on every call, it raises after 3 calls, waits [1.0, 1.0]. A 503 with another code does not retry.
stop wait per Retry-After	3600 waits 30.0 s, -1 and nan wait 1.0 s, 2 waits 2.0 s.
tests/e2e/test_crawl_e2e.py, marked e2e	Skips unless ZENROWS_API_KEY, ZENROWS_CRAWL_BASE_URL and ZENROWS_E2E_CRAWL_URL are set; ZENROWS_E2E_CRAWL_INCLUDE is optional. Passes ZENROWS_CRAWL_BASE_URL as base_url=. DEVELOPMENT.md has a "Running the Crawl e2e test" section.

The e2e test passed against a live deployment before the method renames. It has not run since then, nor since the content_id rename.

How to try it

make check
make test
make test-e2e   # needs the three environment variables above

examples/11_crawl.py runs a crawl end to end. README.md has a Crawl API (new) quickstart, with the client options base_url, retries and timeout.

The acceptance test needs bash, git and Docker. Run it from the repository root:

export ZENROWS_API_KEY="$(cat path/to/key-file)"
bash acceptance/crawl/run.sh              # every check
bash acceptance/crawl/run.sh --no-key     # offline checks only; the live checks are SKIP
bash acceptance/crawl/run.sh --report     # rebuild the report from out/
REF=v1.2.3 bash acceptance/crawl/run.sh   # test a tag, branch or SHA from GitHub

Without ZENROWS_API_KEY, run.sh stops and asks for it.

Release

This PR does not change the version. Do a minor bump (1.6.0) in the release commit.