Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [0.7.0] - 2026-07-06

### Added
- `--raw` (CLI) / `raw` (MCP): return the unprocessed body verbatim — no Readability, no content-type gate; `MARKFETCH_MAX_BYTES` still applies.

### Fixed
- Cloudflare-class CDNs 403 valid Chrome headers over HTTP/2 but pass them over HTTP/1.1 — markfetch now defaults to HTTP/1.1.
- Empty-heading pruning truncated headings containing an inline `#` (`## Step 1 # download` became `# download`).

## [0.6.0] - 2026-05-14

### Added
Expand Down
18 changes: 11 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
[![node](https://img.shields.io/node/v/markfetch.svg?color=10b981)](https://nodejs.org/)
[![license](https://img.shields.io/npm/l/markfetch.svg?color=10b981)](https://github.com/vasylenko/markfetch/blob/main/LICENSE)

The built-in fetch tools that ship with AI coding agents return raw HTML, broken markdown, or `403` from Cloudflare more often than you'd like. `markfetch` sends **HTTP/2 with a coherent Chrome header set** so bot-detection systems see a real browser, then runs the response through the **same Reader View pipeline your browser uses** (Mozilla's Readability → turndown). The output is markdown indistinguishable from a human running "Save as Markdown" — on sites that would block a naive curl.
The built-in fetch tools that ship with AI coding agents return raw HTML, broken markdown, or `403` from Cloudflare more often than you'd like. `markfetch` sends **a coherent Chrome header set** so bot-detection systems see a real browser, then runs the response through the **same Reader View pipeline your browser uses** (Mozilla's Readability → turndown). The output is markdown indistinguishable from a human running "Save as Markdown" — on sites that would block a naive curl.

One command, two surfaces:

Expand Down Expand Up @@ -61,13 +61,13 @@ gemini mcp add -s user markfetch npx -y markfetch
| CloudFlare `/markdown` | ✓ | ✓ | – | paid |
| **`markfetch`** | **✓** | **✓** | **✓ (8 codes)** | **✓** |

- **Real-browser HTTP/2 + Chrome fingerprint.** ALPN-negotiated h2, `User-Agent`, `Sec-CH-UA-*`, `Sec-Fetch-*`, `Accept-*`. A Chrome UA with no client hints is a *stronger* automation signal than curl `markfetch` sends the full coherent set, derived from the UA at startup so an override stays internally consistent.
- **Real-browser request fingerprint.** `User-Agent`, `Sec-CH-UA-*`, `Sec-Fetch-*`, `Accept-*` — a coherent Chrome header set. A Chrome UA with no client hints is a *stronger* automation signal than curl, so `markfetch` sends the full set, derived from the UA at startup so an override stays internally consistent. HTTP/1.1 over TLS: some CDNs fingerprint undici's HTTP/2 connection and 403 it.

- **Reader-View-quality extraction.** [linkedom](https://github.com/WebReflection/linkedom) → [@mozilla/readability](https://github.com/mozilla/readability) → [turndown](https://github.com/mixmark-io/turndown) with GFM tables, strikethrough, and task lists. Code fences preserve `language-X` hints. Sphinx-style bare `<pre>` blocks render as code, not escaped prose. Intraword underscores stay un-escaped — no more `list\_tools`.

- **One tool, one shape (MCP).** `fetch_markdown(url, savePath?)` returns markdown in `content[0].text`. No `structuredContent`, no frontmatter, no metadata fields. Several major MCP clients (Claude Code CLI, VS Code/Copilot) forward only `structuredContent` to the model and drop `content[]` when both are present — `markfetch` deliberately stays on the channel your LLM can actually read.
- **One tool, one shape (MCP).** `fetch_markdown(url, savePath?, raw?)` returns markdown in `content[0].text`. No `structuredContent`, no frontmatter, no metadata fields. Several major MCP clients (Claude Code CLI, VS Code/Copilot) forward only `structuredContent` to the model and drop `content[]` when both are present — `markfetch` deliberately stays on the channel your LLM can actually read.

- **`savePath` / `-o` escape valve.** Pass an absolute path (MCP `savePath`) or `-o <path>` (CLI) and the markdown lands on disk instead of the response channel. Use it when your client's inline tool-result cap would truncate large responses, or to redirect output from a shell pipeline. The file is only ever the markdown of the URL — fetch errors return a `[code]` string and never touch the disk.
- **`savePath` / `-o` escape valve.** Pass an absolute path (MCP `savePath`) or `-o <path>` (CLI) and the output lands on disk instead of the response channel. Use it when your client's inline tool-result cap would truncate large responses, or to redirect output from a shell pipeline. The file is only ever the fetched output (extracted markdown, or the raw body with `--raw`) — fetch errors return a `[code]` string and never touch the disk.

- **Whole document or honest failure.** No pagination, no truncation. If the document doesn't fit in `MARKFETCH_MAX_BYTES`, you get `too_large` — never a half-truth.

Expand All @@ -88,6 +88,9 @@ npx -y markfetch https://example.com/article -o article.md

# Pipe into another tool
npx -y markfetch https://example.com/article | pandoc -o article.pdf

# Fetch JSON / APIs / page source verbatim
npx -y markfetch --raw https://api.github.com/repos/vasylenko/markfetch
```

For repeat use, install once:
Expand All @@ -102,7 +105,8 @@ Flags:

| Flag | Purpose |
|---|---|
| `-o, --output <path>` | Save markdown to a file (absolute or relative path). Default is stdout. |
| `-o, --output <path>` | Save the output to a file (absolute or relative path). Default is stdout. |
| `--raw` | Return the unprocessed response body as UTF-8 text — skips Readability and the content-type gate. For JSON, XML, plain text, or page source (binary is not byte-preserved). |
| `-V, --version` | Print version and exit. |
| `-h, --help` | Print usage and exit. |

Expand All @@ -115,8 +119,8 @@ Errors carry one of eight deterministic codes:
| `network_error` | DNS / TCP / TLS failure, or an unexpected internal error from the fetcher. |
| `http_error` | Upstream returned a non-2xx status. |
| `timeout` | Per-request budget `MARKFETCH_TIMEOUT_MS` exceeded. |
| `unsupported_content_type` | Response was not `text/html` or `application/xhtml+xml`. |
| `extraction_failed` | Readability returned no article content (typical for pure client-rendered SPAs). |
| `unsupported_content_type` | Response was not `text/html` or `application/xhtml+xml` (not raised with `--raw` / `raw`). |
| `extraction_failed` | Readability returned no article content (typical for pure client-rendered SPAs). Not raised with `--raw` / `raw`. |
| `too_large` | Response body or extracted markdown exceeded `MARKFETCH_MAX_BYTES`. |
| `save_failed` | `savePath` was given but `writeFile` failed (parent directory missing, permission denied, etc.). |
| `save_forbidden` | `savePath` resolves outside the allowed write roots — see [Write sandbox](#write-sandbox). MCP-only; the CLI has no sandbox. |
Expand Down
9 changes: 7 additions & 2 deletions docs/SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,15 @@ Text processing pipeline:

```
URL
→ undici.fetch HTTP/2 via ALPN; full Chrome header set; Sec-CH-UA-* derived from UA
→ undici.fetch HTTP/1.1; full Chrome header set; Sec-CH-UA-* derived from UA
→ linkedom.parseHTML
→ @mozilla/readability
→ turndown + GFM HTML → markdown
→ caller markdown body, or "Saved N bytes to /path" confirmation
```

`raw` mode (`--raw` / MCP `raw`) returns the body straight from `undici.fetch`, skipping the parse → extract → convert steps.

Errors throw `MarkfetchError` uniformly from core; adapters catch once. Codes: `network_error`, `http_error`, `timeout`, `unsupported_content_type`, `extraction_failed`, `too_large`, `save_failed`; plus `save_forbidden`, emitted by the MCP adapter only (before `fetchMarkdown` runs — see "Asymmetric write sandbox" under Core Decisions). CLI emits `[code] message` to stderr and exits 1; MCP emits `{ isError: true, content: [{ text: "[code] message" }] }`.

## Core Decisions
Expand All @@ -23,7 +25,9 @@ Errors throw `MarkfetchError` uniformly from core; adapters catch once. Codes: `

- **Core throws, adapters translate.** Seven of the eight error codes surface from `core.ts` — five are thrown explicitly as `MarkfetchError`; `network_error`, `timeout`, and (sometimes) `http_error` are translated by `classifyError` from underlying-API errors (undici TypeErrors, AbortSignal timeouts). The eighth code, `save_forbidden`, is the exception — it's emitted by the MCP adapter before `fetchMarkdown` is invoked (see "Asymmetric write sandbox" below). New core codes need an `ErrorCode` union member + a throw site; adapters don't change.

- **HTTP/2 + coherent Chrome fingerprint.** Wire protocol, headers, and UA must agree — a Chrome UA over HTTP/1.1 or without `Sec-CH-UA-*` is *more* suspicious than curl. `Sec-CH-UA-*` is derived from `MARKFETCH_USER_AGENT` at startup so override-coherence is mechanical.
- **HTTP/1.1 + coherent Chrome header fingerprint.** Headers and UA must agree — a Chrome UA without `Sec-CH-UA-*` is a stronger bot signal than curl, so `Sec-CH-UA-*` is derived from `MARKFETCH_USER_AGENT` at startup and override-coherence is mechanical. HTTP/1.1 is deliberate: undici's h2 path hands `node:http2` a pre-connected socket whose first-flight frames some CDNs (Cloudflare, seen on `openai.com`) score as a bot and 403; the identical request over h1.1 passes. h2 buys nothing for single-shot GETs.

- **`raw` passthrough.** Returns the fetched body verbatim — no Readability, no content-type gate; the fetch layer and `MARKFETCH_MAX_BYTES` cap stay. Same `fetchMarkdown` code path via a `raw` flag — no second entry point. Body is UTF-8 text; binary is not byte-preserved.

- **Single-channel MCP response.** `content[0].text` only. Several major MCP clients (Claude Code CLI, VS Code/Copilot) forward only `structuredContent` to the model and drop `content[]` when both are present — a single-channel response keeps the markdown reachable from those clients.

Expand All @@ -40,6 +44,7 @@ Errors throw `MarkfetchError` uniformly from core; adapters catch once. Codes: `
- **Authentication.** `MARKFETCH_AUTH_HEADER` env var (simple), or Chrome-cookie import for sites where the user is already logged in (frictionless, platform-specific, security-sensitive). Trigger: first useful internal / paywalled doc.
- **JS rendering fallback for SPAs.** Playwright / headless Chrome as a companion package (`markfetch-heavy`) so the lean package stays lean. Trigger: enough useful sites returning `extraction_failed`.
- **CloudFlare `/markdown` fallback.** Gated by `CF_AUTH_TOKEN`; fall back when Readability fails. Trigger: extraction failure rate stays high after Readability tuning.
- **Browser-grade TLS + HTTP/2 impersonation.** Stricter CDN tiers fingerprint Node's TLS (JA3/JA4) and 403 every protocol. Apify's `impit` (BoringSSL-based) ships a Chrome-matching TLS + h2 fingerprint, near-drop-in for `undici.fetch`. Trigger: target sites 403 the current h1.1 + Chrome-header approach.
- **Cookie reuse across redirects within a single fetch.** Currently none. Trigger: a target serves content only after a session-cookie redirect.
- **Proxy support** (`MARKFETCH_PROXY_URL`) and **`Accept-Language` control** (`MARKFETCH_ACCEPT_LANGUAGE`). Trigger: corporate proxy / locale-specific content.
- **Single-binary distribution.** Bun's `build --compile`, Node SEA, or similar. Trigger: `npx` first-run latency feedback, or an offline / airgapped need.
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "markfetch",
"version": "0.6.0",
"version": "0.7.0",
"description": "Fetch a URL, return clean markdown. MCP server and CLI for AI agents.",
"license": "MIT",
"author": {
Expand Down
11 changes: 8 additions & 3 deletions src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,13 @@ program
.argument("<url>", "absolute http(s) URL to fetch")
.option(
"-o, --output <path>",
"save markdown to file (absolute or relative path); default is stdout",
"save output to a file (absolute or relative path); default is stdout",
)
.action(async (url: string, options: { output?: string }) => {
.option(
"--raw",
"return the unprocessed response body (skip Readability and the content-type gate)",
)
.action(async (url: string, options: { output?: string; raw?: boolean }) => {
// CLI resolves relative output paths against cwd before calling core;
// core requires an absolute path so the contract is unambiguous regardless
// of which adapter invokes it. Tilde expansion is intentionally NOT done
Expand All @@ -43,9 +47,10 @@ program
const { markdown, bytes, savedTo } = await fetchMarkdown({
url,
savePath,
raw: options.raw,
});
if (savedTo === undefined) {
// Raw markdown body — no added newline, matches MCP content[0].text.
// Verbatim output — no added newline, matches MCP content[0].text.
process.stdout.write(markdown);
} else {
// Confirmation message — the only stdout newline the CLI ever adds.
Expand Down
Loading
Loading