Default to HTTP/1.1 transport + add --raw passthrough#6
Merged
Conversation
undici's h2 path hands a pre-connected socket to node:http2, whose first-flight frame pattern some CDNs (Cloudflare, observed on openai.com) score as a bot and 403 — even against a valid Chrome header set, while the identical request over HTTP/1.1 is let through. h2 buys nothing for single-shot GETs and every h2 server speaks h1.1, so pin allowH2:false.
Match CHANGELOG, README, and SPEC to the transport default. Correct the SPEC decision that called Chrome-over-h1.1 more suspicious than curl — observed evidence is the opposite for these edges. Record impit (browser-grade impersonation) as the escalation path for stricter CDN tiers.
vasylenko
force-pushed
the
fix/http1-fallback-on-cloudflare-block
branch
from
2f01203 to
1f3d062
Compare
vasylenko
changed the title
Return the fetched body verbatim, skipping Readability and the content-type gate while keeping the fetch layer and the MARKFETCH_MAX_BYTES cap. Non-HTML responses (JSON, XML, plain text, source) come back as-is instead of unsupported_content_type. Threaded as a `raw` flag through fetchMarkdown (fetchHtml → fetchBody, content-type gate now conditional); CLI `--raw`, MCP `raw` boolean.
CHANGELOG entry, README (flags table, usage example, error-code note), and SPEC (Core Decision + pipeline note).
vasylenko
changed the title
- extractMarkdown helper + `content = raw ? body : extractMarkdown(...)` replaces the let/if-else and a duplicated comment - reuse enforceTooLarge for the output cap; compute body byte-length once - correct the dispatcher comment: undici v8 defaults allowH2:true, so allowH2:false is a deliberate override (deleting it re-enables h2), not a restated default - test raw+savePath writes the raw body to disk; drop a subsumed assertion; fix the T6 invariant comment for raw
- "save markdown" → "save output" (raw+save writes the raw body) in CLI help, MCP tool description + savePath describe, and README - README MCP signature gains raw?; extraction_failed row notes it's not raised with raw - note that raw returns UTF-8 text (binary not byte-preserved) in MCP describe, README --raw row, and SPEC
Drop the "earlier revisions assumed" temporal framing; keep the why (the Chrome-over-h1.1 intuition doesn't hold for these edges).
The dispatcher comment defended its own necessity and lectured on h2 mechanics; fetchBody/extractMarkdown/fetchMarkdown comments restated the adjacent code. Keep the why only: the undici default, the CDN bot-scoring, the raw-mode error contract.
undici h2 internals belong in SPEC and the code comment, not in a changelog entry. The "naive expectation" parenthetical restated evidence the decision text already carries.
The [^\n]+\s* pairs in the empty-heading prune and the /\n+$/ pre-trim
backtrack quadratically on adversarial whitespace (Sonar S8786). Match
blank spans line-by-line and trim by index instead. Requiring \n before
the next heading also fixes real truncation: a heading with an inline
"#" ("## Step 1 # download") lost everything before it. Snapshot 10
pins that.
Descriptions load into every MCP client's context; platform path examples, schema-rejection mechanics, and restated behavior were tax, not signal.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent changes to the fetch layer, shipped together.
1. Default to HTTP/1.1 (fix)
WHY — Some CDNs (Cloudflare, seen on
openai.com/products/release-notes/) 403 a valid Chrome header set over HTTP/2 but pass the identical request over HTTP/1.1 — undici's h2 handshake pattern trips their bot-scoring. h2 buys a single-shot GET nothing anyway.HOW — undici v8 defaults to
allowH2: true; pin the global dispatcher toallowH2: false. Corrected the SPEC decision that called Chrome-over-h1.1 "more suspicious than curl"; recordedimpitin SPEC as the escalation path if stricter CDN tiers need genuine h2.2.
--raw/rawpassthrough (feature)WHY — Sometimes the useful output is the unprocessed body — JSON, XML, plain text, page source — not extracted article markdown.
HOW — A
rawflag onfetchMarkdownreturns the body verbatim, skipping Readability and the content-type gate; theMARKFETCH_MAX_BYTEScap stays. CLI--raw, MCPrawboolean. Body is UTF-8 text (binary not byte-preserved).fetchHtmlbecamefetchBody(url, raw)— no second entry point.