Skip to content

test(crawl): add a first-install acceptance test for the Crawl client - #35

Merged
StefanoGuerrini merged 4 commits into
feat/crawl-client-v2from
test/crawl-acceptance
Oct 12, 2026
Merged

StefanoGuerrini merged 4 commits into
feat/crawl-client-v2from
test/crawl-acceptance

Conversation

@StefanoGuerrini

@StefanoGuerrini StefanoGuerrini commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #31 (feat/crawl-client-v2): it adds a test harness only, so the feature PR stays focused.

Summary

This PR adds acceptance/crawl/, a first-install acceptance test for the Crawl client. It installs the release artifact the way a new user does. Then it follows the Crawl quickstart in README.md and checks every Crawl feature against the live API. It writes one markdown report.

What a run does

acceptance/crawl/run.sh runs these steps:

  1. It takes the source from this checkout, or from GitHub at REF when you set it.
  2. In its own container, it builds the wheel with python -m build.
  3. It installs that wheel in a clean, digest-pinned container. That container gets the artifact, the checks and the key by name, and no source, caches or tokens.
  4. It runs the Crawl quickstart from README.md as written. The report lists each change the cost rules need.
  5. It runs checks 1 to 21, plus a mypy --strict check that reads the package as typed. Each check writes one JSON result with status PASS, FAIL or SKIP.
  6. It stops every crawl it started, also when a check fails or on Ctrl-C.
  7. It writes acceptance/crawl/out/crawl-first-install-report.md, then makes sure the key is in no file under out/.

The checks cover create, idempotency, wait, get, list, results, content, download, stop, error codes, naming, documentation copy, failed crawls and documented defaults. Some checks point the client at a small local server, so they give the same result on every run. acceptance/crawl/README.md lists every check with its expected result.

Input Default
CRAWL_URL https://www.scrapingcourse.com/ecommerce/
CRAWL_INCLUDE /product/
CRAWL_API_BASE https://api.zenrows.com/v1
REF unset: this checkout
Images node:20, python:3.12-slim, golang:1.25, pinned by digest in versions.env
OUT_DIR moves the output

Cost

A run starts five crawls, each with depth 1 and max_pages 10 or less. On the default CRAWL_URL, a run fetches about 10 pages, and at most 50. Check 9 can start a sixth crawl, which adds at most 10 pages. At most 2 crawls run at the same time, under the account limit of 3 active jobs. Do not run two copies against one account at the same time.

Packaging

pyproject.toml excludes /acceptance from the sdist, so the harness never ships in the package.

How to try it

Run it from the repository root. It needs bash, git and Docker.

export ZENROWS_API_KEY="$(cat path/to/key-file)"
bash acceptance/crawl/run.sh              # every check
bash acceptance/crawl/run.sh --no-key     # offline checks only; the live checks are SKIP
bash acceptance/crawl/run.sh --report     # rebuild the report from out/
REF=v1.2.3 bash acceptance/crawl/run.sh   # test a tag, branch or SHA from GitHub

Without ZENROWS_API_KEY, run.sh stops and asks for it. The report shows a verdict, a feature matrix and one finding per FAIL. The script never prints the key.

Squash-merge with the title as the message: the branch commits carry trailer lines that should not land.

acceptance/crawl/ builds the release artifact from this checkout, installs
it in a clean container the way a new user would, and runs every Crawl
feature against the live API: README example, create, wait, results,
content, download, stop, errors, naming and docs. One JSON result per
check and a markdown report land in acceptance/crawl/out/ (git-ignored).
It is kept out of the published package.
The naming check now covers content_id, the argument of the content
operation, so a client that names it differently fails check 17. The MCP
surface calls crawl_content with content_id.
Check 9 read content from the crawl check 20 stops; a stop that lands before any page is fetched left nothing to read. It now falls back to a one-item html crawl that runs to the end.
@StefanoGuerrini
StefanoGuerrini marked this pull request as ready for review October 12, 2026 14:03
@StefanoGuerrini
StefanoGuerrini merged commit 36acfdf into feat/crawl-client-v2 Oct 12, 2026
1 check passed
@StefanoGuerrini
StefanoGuerrini deleted the test/crawl-acceptance branch October 12, 2026 14:27
@StefanoGuerrini

Copy link
Copy Markdown
Contributor Author

Merged into the feature branch with a merge commit. It ships to main with #31.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant