[samgozman]

Three fixes from a review of refetch on real sites.

Cloudflare's 403 challenge goes to headless Chrome

Cloudflare sends its managed challenge as HTTP 403 with cf-mitigated: challenge. refetch checked for 403/429 before it checked for a challenge, so it filed the page as refused and never ran headless Chrome. Now:

• a challenge counts before a refusal
• both clients read cf-mitigated

Files are an error, not bytes

A PDF, an image or another file used to come back as its raw bytes, cut to max_chars. Now:

• it fails with not a web page or text (application/pdf)
• the hand-off tells Claude to open the file with a tool that reads that type
• a body starting with %PDF- counts as a PDF even when the server calls it text

render: true forces headless Chrome

render could only turn Chrome off. With render: true (CLI: refetch get --force-render), headless Chrome reads the page even when the HTTP clients got it or were refused. This is for pages whose content scripts fill in, which detection can miss. If Chrome can't read the page, the result is the HTTP client's text, with a note saying why.

One change in behavior: a call that passed render: true used to behave as if it had been left out. Now it always runs Chrome.

Checks

• Unit tests:
  • a 403 challenge page, and a 403 with only the header, under both impersonation modes
  • a PDF, plus a NotText table
  • cf-mitigated read by both real clients (through the proxy test's backend)
  • the render modes, including falling back to the HTTP page
• e2e: a new /challenge-403 fixture. With refetch get:
  • main: refused
  • this branch: challenge, then headless Chrome ok
• A real PDF from w3.org fails with the new error.
• make test, make lint (macOS and GOOS=linux), make cover-check (89.0%; 87% of the changed lines), claude plugin validate . --strict.
• I did not run the full e2e/run.py (about 35 Claude sessions).

🤖 Generated with Claude Code

BEGIN_COMMIT_OVERRIDE
feat: fetch_page render true forces headless Chrome, even for a page the HTTP clients read (#6)
fix: send a Cloudflare challenge that arrives as HTTP 403 to headless Chrome (#6)
fix: refuse PDFs, images and other files with a clear error instead of their bytes (#6)
END_COMMIT_OVERRIDE

[codecov]

Codecov Report

❌ Patch coverage is 87.27273% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.75%. Comparing base (0cf778e) to head (925ebd5).

Files with missing lines	Patch %	Lines
cmd/refetch/get.go	62.50%	3 Missing ⚠️
fetch/clients.go	57.14%	3 Missing ⚠️
fetch/fetch.go	90.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main       #6      +/-   ##
==========================================
+ Coverage   89.72%   89.75%   +0.02%     
==========================================
  Files          18       18              
  Lines        1655     1689      +34     
==========================================
+ Hits         1485     1516      +31     
- Misses        169      172       +3     
  Partials        1        1              

Files with missing lines	Coverage Δ
cmd/refetch/main.go	79.00% <100.00%> (+0.42%)	⬆️
cmd/refetch/mcp.go	88.39% <100.00%> (+1.93%)	⬆️
fetch/text.go	96.77% <100.00%> (+1.31%)	⬆️
fetch/fetch.go	93.60% <90.00%> (+0.11%)	⬆️
cmd/refetch/get.go	90.24% <62.50%> (-6.82%)	⬇️
fetch/clients.go	86.27% <57.14%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.