[samgozman]

fetch_page and refetch get now return the page as Markdown by default. Before, a page came back as one flat run of text: links lost their URLs, tables lost their columns, and code lost its whitespace. Claude could see a search result or a "next page" link but couldn't follow it. format: "text" (CLI: --format text) gives the old plain text.

What the Markdown keeps

• Headings, lists, blockquotes and rules.
• Links
  • Made absolute, resolved against the address after redirects (or <base href>).
  • Search engines' click-through links (DuckDuckGo's uddg, Google's /url?q=) point at their real targets.
  • Tracking parameters (utm_*, fbclid, gclid and so on) are dropped.
  • A target repeated right after itself shows only its text. A search result links its title, URL and snippet to the same place, so the URL appears once instead of three times.
  • Only http, https and mailto links keep a target. javascript: and data: links show only their text.
• Tables
  • A data table becomes a pipe table, without rows and columns that are empty throughout (spacer rows, vote arrows).
  • A layout table, one with nested tables, lists or headings in its cells, becomes its cells' text.
• Code blocks: whitespace and language kept, fenced longer than any backtick run inside them.
• Elements laid out like a browser does: an element with a block inside it (a custom element such as <react-app>, or a span around a div) is a block. Without this, GitHub's releases page came out as one line. A link stays one line, which keeps a card as one link.
• Dropped: scripts, styles, forms, images, and hidden elements (hidden, aria-hidden, display:none).
• Unchanged: the detection of JavaScript-only and empty pages still measures plain text.
• Saved pages: save: true writes .md, or .txt with format: "text".

Real pages (Markdown vs plain text size)

Page	Markdown	Text	Notes
DuckDuckGo results	3.8 KB	3.7 KB	each result: a title with its real target, the URL, the snippet
GitHub releases	11 KB	5.5 KB	release notes with headings and commit links
Hacker News	15 KB	4.1 KB	the story list as a table, with every link
Wikipedia (Go)	not measured	not measured	the infobox as a two-column table, code samples fenced
go.dev tutorial	10 KB	7.5 KB	shell and Go code blocks inside the numbered steps

Links cost space. A link-heavy page is two to four times its plain-text size, so it reaches the default max_chars (20,000) sooner.

Sessions (e2e/run.py, Sonnet)

There's a new case, links: an index page that refuses plain clients and names its notes pages only by link text. The secret word is on the page behind one link, under a path that can't be guessed.

Setup	main	this branch
alone	7/8. links fails: Claude guessed paths with 2 more fetch_page calls and Bash, and gave no answer	8/8
context-mode	n/a	8/8

The other seven cases (ok, 403, challenge, challenge-403, challenge-js, js, slow) pass in every run.

Checks

• make test
• make lint, on macOS and with GOOS=linux
• make cover-check: 89.7%
• claude plugin validate . --strict

Review fixes (6dfc0fa)

A code review found one critical issue: crafted markup could make the output grow with the square of the page. A 280 KB table with one 4,000-cell row gave 225 MB of Markdown. Deep quotes and lists grew the same way, because each level prefixed every line again.

• Fix: tables wider than 64 columns or with more than 20,000 cells are rendered as their cells' text, and quotes and lists stop adding prefixes after six levels.
• Result: at 2 MB, the largest page refetch reads, every shape I probed (16 of them) converts in under 0.2 s to at most three times its size. A regression test checks this.

Smaller fixes from the same review:

• Links in layout tables lost their target.
• Code blocks lost blank lines.
• Hidden table rows and cells were shown.
• A code block's Copy button got into its code.
• Dropping a tracking parameter re-sorted the rest of the query.
• A saved page that isn't HTML is now saved as .txt.

One known limit: when headless Chrome reads a page after the HTTP fetch failed, links resolve against the URL that was asked for, not the address Chrome ended up on after redirects. --dump-dom doesn't report that address.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 93.78698% with 21 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.47%. Comparing base (b5996a6) to head (2bca6b1).

Files with missing lines	Patch %	Lines
fetch/markdown.go	93.48%	20 Missing ⚠️
cmd/refetch/mcp.go	94.11%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main       #7      +/-   ##
==========================================
+ Coverage   89.75%   90.47%   +0.71%     
==========================================
  Files          18       19       +1     
  Lines        1689     2015     +326     
==========================================
+ Hits         1516     1823     +307     
- Misses        172      191      +19     
  Partials        1        1              

Files with missing lines	Coverage Δ
cmd/refetch/get.go	95.45% <100.00%> (+5.21%)	⬆️
fetch/clients.go	86.27% <100.00%> (ø)
fetch/fetch.go	93.60% <100.00%> (ø)
fetch/text.go	97.14% <100.00%> (+0.36%)	⬆️
cmd/refetch/mcp.go	88.70% <94.11%> (+0.31%)	⬆️
fetch/markdown.go	93.48% <93.48%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.