feat(crawl): add ZenRowsCrawlClient for the Crawl API

DEVELOPMENT.md

@@ -19,14 +19,19 @@ Makefile targets below.
 
 ```
 src/zenrows/
-├── __init__.py               # re-exports both clients
+├── __init__.py               # re-exports the clients
 ├── client.py                 # ZenRowsClient (legacy sync scraper)
-└── batch/
-    ├── __init__.py           # ZenRowsBatchClient + key models
+├── batch/
+│   ├── __init__.py           # ZenRowsBatchClient + key models
+│   ├── client.py             # hand-written typed facade
+│   ├── _transport.py         # httpx wrapper, RFC 7807 → exceptions
+│   ├── errors.py             # BatchAPIError, ProblemDetail
+│   └── models.py             # GENERATED — pydantic v2 (do not edit)
+└── crawl/
+    ├── __init__.py           # ZenRowsCrawlClient + models
     ├── client.py             # hand-written typed facade
-    ├── _transport.py         # httpx wrapper, RFC 7807 → exceptions
-    ├── errors.py             # BatchAPIError, ProblemDetail
-    └── models.py             # GENERATED — pydantic v2 (do not edit)
+    ├── errors.py             # CrawlAPIError
+    └── models.py             # hand-written pydantic v2 models
 ```
 
 The Batch SDK is split deliberately:
@@ -42,12 +47,21 @@ This way the wire types stay in lockstep with the OpenAPI document
 while the ergonomic surface (method names, helpers, retries, URL
 override) stays in our control.
 
+The Crawl client uses the shared HTTP transport (`_transport.py`: key
+header, retries, problem+json) with `CrawlAPIError.from_response` as
+its error mapping. Its models are
+hand-written: the surface is a few small schemas, so there is no Crawl
+codegen step.
+Keep them tolerant: unknown fields are ignored, and every response enum
+gets `_missing_ = classmethod(open_enum_missing)`.
+
 ## Common tasks
 
 | Make target     | What it does |
 |-----------------|--------------|
 | `make sync`     | `uv sync --all-extras` |
-| `make test`     | `uv run pytest` |
+| `make test`     | `uv run pytest` (offline; e2e tests deselected) |
+| `make test-e2e` | End-to-end tests against a live API (see below) |
 | `make check`    | `ty check` + `ruff check` + `ruff format --check` (CI mode) |
 | `make typecheck`| `ty check src` (static types; `models.py` excluded) |
 | `make lint`     | `ruff check --fix` |
@@ -56,6 +70,38 @@ override) stays in our control.
 | `make build`    | Build wheel + sdist via hatchling |
 | `make clean`    | Drop caches + build outputs |
 
+## Running the Crawl e2e test
+
+`tests/e2e/test_crawl_e2e.py` drives a real crawl through the client:
+create (with `include_patterns`, `output_format="html"`), wait, read the
+results, one page and the NDJSON download, list, stop, and a 404. It
+costs a few credits. It is marked `e2e`, which `make test` deselects,
+and it skips unless the three required variables below are set.
+
+| Variable | |
+|---|---|
+| `ZENROWS_API_KEY` | required: a key with Crawl access |
+| `ZENROWS_CRAWL_BASE_URL` | required: the API base, e.g. `https://api.zenrows.com/v1`; the test passes it as `base_url=` |
+| `ZENROWS_E2E_CRAWL_URL` | required: the page the crawl starts from |
+| `ZENROWS_E2E_CRAWL_INCLUDE` | optional: an include pattern; every result must contain it |
+
+```bash
+export ZENROWS_API_KEY=zr_...                       # a key with Crawl access
+export ZENROWS_CRAWL_BASE_URL=https://api.zenrows.com/v1
+export ZENROWS_E2E_CRAWL_URL=https://example.com/products/
+export ZENROWS_E2E_CRAWL_INCLUDE=/product/          # optional
+make test-e2e
+```
+
+Pick a start page whose links include a few pages matching the pattern:
+the test crawls at depth 1 with `max_items=3` and `output_format="html"`.
+
+To test against a local or staging deployment, point
+`ZENROWS_CRAWL_BASE_URL` at its `/v1` base instead. When the account
+has reached its limit of active jobs (429 `too_many_crawls`: 3 by default,
+shared with its Batch jobs), the test waits and retries (up to 5 minutes)
+before creating its crawl.
+
 ## Refreshing the OpenAPI spec
 
 `docs/openapi.yaml` is the SDK-local copy of the spec. `make generate`

Makefile

@@ -1,28 +1,33 @@
-.PHONY: install sync test lint format typecheck check generate docs clean build bump release
+.PHONY: install sync test test-e2e lint format typecheck check generate docs clean build bump release
 
 # Bootstrap: install + dev deps, build the local venv.
 install sync:
 	uv sync --all-extras
 
-# Run the suite.
+# Run the suite (offline; e2e tests are deselected).
 test:
 	uv run pytest
 
+# Run the end-to-end tests against a live API. Needs ZENROWS_API_KEY,
+# ZENROWS_CRAWL_BASE_URL and ZENROWS_E2E_CRAWL_URL; see DEVELOPMENT.md.
+test-e2e:
+	uv run pytest -m e2e -s tests/e2e
+
 # Static type check (ty — Astral). Shipped surface only; tests are
 # covered by the suite. Generated models.py is excluded in pyproject.
 typecheck:
 	uv run ty check src
 
 # Lint + format + type check (CI mode — no fixes).
 check: typecheck
-	uv run ruff check src/ tests/
-	uv run ruff format --check src/ tests/
+	uv run ruff check src/ tests/ acceptance/
+	uv run ruff format --check src/ tests/ acceptance/
 
 # Lint + format (writes fixes).
 lint:
-	uv run ruff check --fix src/ tests/
+	uv run ruff check --fix src/ tests/ acceptance/
 format:
-	uv run ruff format src/ tests/
+	uv run ruff format src/ tests/ acceptance/
 
 # Regenerate the pydantic v2 models from the backend's canonical spec.
 # docs/openapi.yaml is the SDK-local copy of the spec; refresh it from the

README.md

@@ -10,14 +10,17 @@
 SDK to access [Zenrows](https://www.zenrows.com/) APIs directly from Python.
 Zenrows handles proxies rotation, headless browsers, and CAPTCHAs for you.
 
-This package ships two clients:
+This package ships three clients:
 
   - **`ZenRowsClient`** — the original synchronous scraping client.
     One URL in, one HTML/JSON response out. Best for ad-hoc scraping.
   - **`ZenRowsBatchClient`** — the async-job / Batch API client. Submit
     thousands of URLs as one job, poll for results, optionally upload a
     CSV of URLs in one call. Best for offline / bulk pipelines.
     _(Private beta — [contact support](mailto:support@zenrows.com) for access.)_
+  - **`ZenRowsCrawlClient`** — the Crawl API client. Give it one start
+    URL; it follows links and returns the URLs (and, optionally, the
+    pages) it finds behind it. _(New.)_
 
 ## Table of Contents
 
@@ -37,6 +40,10 @@ This package ships two clients:
   - [Act on an id without a GET](#act-on-an-id-without-a-get)
   - [Scheduled jobs & webhooks](#scheduled-jobs--webhooks)
   - [Error handling](#error-handling)
+- [Quickstart — Crawl API (new) (`ZenRowsCrawlClient`)](#quickstart--crawl-api-new-zenrowscrawlclient)
+  - [Read the pages](#read-the-pages)
+  - [List and stop crawls](#list-and-stop-crawls)
+  - [Crawl errors](#crawl-errors)
 - [Contributing](#contributing)
 - [License](#license)
 
@@ -485,6 +492,129 @@ The full Batch surface (jobs, runs, tasks, results, content, history,
 file_inputs, HMAC keys) is reachable via methods on `ZenRowsBatchClient`.
 See `src/zenrows/batch/client.py` or `help(ZenRowsBatchClient)`.
 
+## Quickstart — Crawl API (new) (`ZenRowsCrawlClient`)
+
+Crawl is still evolving: new features are coming, limits may be tuned, and the changelog announces each change.
+
+A crawl starts from one URL, follows the links on each page, and keeps
+the URLs that match your patterns, up to `max_items`. It stops at
+`max_items` or `max_pages`, or when no link is left. `depth` limits the
+link hops from the start URL; without it the crawl has no depth limit.
+It stays on the start URL's registrable domain;
+subdomains count. It runs in the background: `create` returns at once with
+`status="running"`, and `wait` blocks until it ends. Without
+`output_format` a crawl returns URLs only; `output_format="html"` also
+returns each kept URL's page.
+
+```python
+import os
+from zenrows import ZenRowsCrawlClient
+
+client = ZenRowsCrawlClient(api_key=os.environ["ZENROWS_API_KEY"])
+
+crawl = client.create(
+    "https://www.scrapingcourse.com/ecommerce/",
+    depth=1,                         # link hops from the start URL (default: no limit)
+    max_items=50,                    # stop after keeping 50 URLs (default 100)
+    max_pages=100,                   # stop after fetching 100 pages (default 100)
+    include_patterns=["/product/"],  # keep only URLs containing this
+)
+crawl = client.wait(crawl.crawl_id)            # polls up to timeout=600 s
+print(crawl.status.value, crawl.stop_reason, crawl.coverage)
+
+for result in client.results(crawl.crawl_id):  # follows next_cursor
+    print(result.url)
+```
+
+`wait` returns the crawl when it ends, or when `timeout` (seconds,
+default 600) runs out. It does not raise: on timeout the crawl is still
+`running` (call `wait` again, or `stop`), and a `failed` crawl comes
+back with `crawl.error` set. `results` does not poll: it returns at the first empty page, so on a
+running crawl it yields only the URLs kept so far. Call `wait` first to
+read every result. `results(crawl_id, limit=...)` sets the page size of
+each request. `get(crawl_id, cursor=..., limit=...)` returns one raw page.
+
+The client takes these options: `base_url` (default
+`https://api.zenrows.com/v1`), `retries` (default 3; set 0 to disable)
+and `timeout` (seconds per request, default 30):
+
+```python
+client = ZenRowsCrawlClient(api_key, base_url="https://api.zenrows.com/v1", retries=3, timeout=30)
+```
+
+### Read the pages
+
+Pass `output_format="html"` to also fetch the page of every kept URL:
+
+```python
+crawl = client.create(start_url, depth=1, output_format="html")
+client.wait(crawl.crawl_id)
+
+for result in client.results(crawl.crawl_id):
+    if result.content_url:                       # content_status == fetched
+        html = client.content(crawl.crawl_id, result)
+
+# Or everything in one NDJSON file: {"url", "content_status", "content"} per line.
+with client.download(crawl.crawl_id) as download:  # streamed, parsed
+    print(download.status)                         # from X-Crawl-Status
+    for line in download.lines:
+        print(line.url, line.content_status)
+```
+
+`content` takes a result, its `content_url`, or its content id.
+`download` returns a `CrawlDownload`: its `status` is the crawl's status
+when the file was read, and its `lines` yields parsed lines. On a `running` crawl the
+file holds only the results kept so far.
+
+### List and stop crawls
+
+```python
+page = client.list(limit=20)            # newest first, without results
+for c in page.crawls:
+    print(c.crawl_id, c.status.value)
+# Pass page.next_cursor as cursor= for the next page; it is None on the last.
+
+client.stop(crawl_id)                   # idempotent; read final counts with get()
+```
+
+### Crawl errors
+
+Every non-2xx raises `CrawlAPIError`. Branch on `status_code` and `code`:
+
+```python
+from zenrows import CrawlAPIError
+from zenrows.crawl import CRAWL_NOT_ENABLED_CODE
+
+try:
+    client.create(start_url, depth=1)
+except CrawlAPIError as exc:
+    if exc.code == CRAWL_NOT_ENABLED_CODE:  # 403 REQS008
+        print("Crawl is not enabled for this account")
+    elif exc.code == "too_many_crawls": # 429: active jobs limit reached
+        print("retry in", exc.retry_after, "s")
+    elif exc.status_code in (400, 422):
+        print(exc.detail)
+    else:
+        raise
+```
+
+| Status | `code` | What to do |
+|---|---|---|
+| 400 | `invalid_request`, `unknown_parameter`, `invalid_cursor` | Fix the request. Do not retry it as is. |
+| 403 | `REQS008` | Crawl is not enabled for the account (`CRAWL_NOT_ENABLED_CODE`). |
+| 404 | `crawl_not_found`, `content_not_found` | Check the id. |
+| 409 | `idempotency_request_in_flight` | Retry after the first request with this key finishes. |
+| 422 | `invalid_parameter`, `invalid_start_url`, `domain_not_allowed` | Fix the request. Do not retry it as is. |
+| 422 | `idempotency_key_reused` | Use a new key, or no key. Do not retry it as is. |
+| 429 | `too_many_crawls` | The account has reached its limit of active jobs (3 by default), shared with its Batch jobs. Retry after `exc.retry_after` seconds. `create` does not retry it. |
+| 503 | `crawl_busy` | `stop` could not be saved yet; the crawl is still running. `stop` retries it up to `retries` times, waiting `exc.retry_after` seconds, then raises it. Repeating the stop is safe. |
+
+`code` is None when the response has no problem body with a code (for
+example a 502 from a proxy). A crawl that ended `failed` is not an error
+response: read `crawl.error.code` and `crawl.error.detail` on it. Response enums (`CrawlStatus`, `StopReason`,
+`ContentStatus`, ...) are open: a value added later parses as an
+`UNKNOWN` member that keeps the raw value.
+
 ## Contributing
 Pull requests are welcome. For significant changes, please open an issue first to discuss what you would like to change.
 

acceptance/crawl/.gitignore

@@ -0,0 +1 @@
+out/