Fix responses lost to decompression, add alpn_protocols to request()

[liquidsec]

Responses were being discarded over undecodable bodies

A response that declared Content-Encoding but carried no body was thrown away entirely. gzip and brotli correctly report there's no stream to read, and the ? turned that into a transport error, so the caller couldn't tell the host apart from an unreachable one. We ask for gzip, deflate, br on every request, so any bodyless response carrying a Content-Encoding hit it: bodyless redirects, HEAD responses (which echo the entity headers of the GET they mirror), and 304s (which carry the headers a 200 would). The last two are required server behavior, not broken edges. request_batch and request_batch_stream were affected too, since they share parse_response.

max_body_size made it worse twice over. It cut the compressed bytes mid-stream and the resulting decode failure discarded the whole response, so the cap was unusable with compressed bodies. And it only ever bounded what came off the wire, never the inflated result, so a 291KB response could expand to 300MB resident, 28x past the 10MiB default.

decompress now takes the cap and reads incrementally: an empty body is empty whatever the header claims, output is bounded at every layer, and a stream that breaks partway keeps what inflated.

More broadly, a body that won't decode no longer costs you the response. Whatever the reason, the status line and headers arrived cleanly and the raw bytes are what the server actually sent, so they come back undecoded with a note in the debug log. Discarding the response looks identical to an unreachable host from the caller's side, which throws away much more than a body we can't read.

Content-Encoding lists and the x-gzip alias

Only an exact match on the whole header value was recognized, so gzip, br and x-gzip fell through and handed back a still-compressed body as if it were content. Those are now parsed as the ordered list they are and undone in reverse. A coding we can't undo returns the body untouched, since peeling the layers beneath it would only produce nonsense.

alpn_protocols was unreachable from request()

RequestConfig.alpn_protocols already existed and the direct-connection path already honored it, but the pooled path hardcoded an h2-first offer and never consulted the config, and the pyo3 signature didn't accept the parameter at all. So there was no way to keep a request off HTTP/2 from Python.

That matters when a server puts a connection-specific header in an HTTP/2 response. RFC 9113 8.2.2 forbids it, hyper kills the stream with PROTOCOL_ERROR, and the response is lost even though the same server answers cleanly over HTTP/1.1. The only workaround was passing resolve_ip to divert onto the direct path, which is a DNS-pinning parameter doing protocol selection.

The h2-first default is unchanged, so this is additive. Different offers already can't share a pooled connection, since TlsKey includes alpn_protocols.

Behavior changes worth a look

• max_body_size now caps the decompressed body. A caller setting a small cap on a large compressed body used to receive the whole thing inflated and now gets it truncated to the cap, which is what the parameter says it does.
• A body that doesn't match its declared encoding used to raise and now returns undecoded. Callers with except RuntimeError fallbacks around request() will take a different branch.

Also

Version bumped to 0.10.0, and compiled extension modules are gitignored. A build left at the repo root shadows the installed package, because pytest puts the rootdir on sys.path, so a stale one silently gets tested instead of what you built.

[en0f]

Comments:

1. deflate with a zlib wrapper now seems to return the compressed bytes as the body, silently.

decode_one only tries raw DEFLATE. The zlib-wrapped form is what IIS and several CDNs actually send for Content-Encoding: deflate, and blasthttp advertises deflate in its default Accept-Encoding. Raw inflate chokes on the 78 9c header, buf is empty, so decode_one errors, and the new parse_response fallback hands back the compressed bytes at status 200.

on pr78: status 200, content = b'x\x9c\xb3\xc9(...' (== zlib.compress(payload))
on dev: RuntimeError: deflate decompression failed: corrupt deflate stream

Before, you got an error. Now r.content is binary garbage with nothing to distinguish it from a real body. Response has no attribute that says whether decoding succeeded, so anything matching, hashing, or regexing bodies silently matches on compressed bytes. Fix is to try flate2::read::ZlibDecoder and fall back to DeflateDecoder.

2. Repeated Content-Encoding header lines. Only the first is decoded.

parse_response uses headers().get("content-encoding"), which returns the first value only. This is the same stacked-encoding case the PR set out to fix, just spelled differently:

Content-Encoding: gzip
Content-Encoding: gzip
body = gzip(gzip(payload))
→ status 200, content = gzip(payload) # one layer peeled, still compressed

get_all("content-encoding") joined with "," feeds straight into the new list parser.

3. alpn_protocols on request() is broken alongside resolve_ip / request_target

Those two options route to dispatch_direct, which discards the negotiated protocol (let (stream, cert_info, _alpn, peer_ip)) and unconditionally does http1::handshake. So forcing h2 negotiates h2 and then speaks HTTP/1.1 over it:

python
request("https://one.one.one.one/", resolve_ip="1.1.1.1", alpn_protocols=["h2"])
→ RuntimeError: dispatch_direct request failed: connection closed before message completed
same request without alpn_protocols → 403, fine

The new docstring at src/python.rs:864 says the default offers ["h2", "http/1.1"]. That's the pooled path only; connect_stream (hyper.rs:1001) defaults to http/1.1 alone, so the documented semantics of the new parameter are wrong for both direct-dispatch modes. Either reject a non-http/1.1 ALPN list on that path with a clear error, or dispatch h2 when it's negotiated.

4. The new comment claims a bound that doesn't exist

hyper.rs:1306 — "read_body stops at max_body, so a body that hit the cap is one we cut ourselves, mid-stream." read_body does body.collect().await and truncates after buffering the whole thing. With max_body_size=1000 against a 40 MB response, the client read all 40,012,223 bytes into memory and then returned 985. cut_by_cap happens to still be labelled correctly, but in the decompress doc comment, that a cap is needed because a small response can inflate into an arbitrarily large allocation — applies just as much to the wire read, which is unbounded.

[en0f]

Quick comments:

1.) Duplicated Content-Encoding line, body encoded once, now comes back undecoded

Joining get_all("content-encoding") is spec-correct (RFC 9110 5.3), but it regresses the more common shape of the doubled header: a proxy that re-adds Content-Encoding: gzip in front of a backend that already set it, without re-compressing. Two lines now mean gzip,gzip; the outer peel succeeds, the inner one fails on plain HTML, and decompress returns the original bytes per Decoded::raw, so the caller gets compressed bytes plus a decode_error instead of the body.

I confirmed this against both commits with a local server sending the header twice and a singly-gzipped body:

base 8e2d395: content == PAYLOAD → True
head e466b35: content == still-gzipped body → True, decode_error: gzip decompression failed: invalid gzip header

The PR's own doubled-mismatch test encodes this as intended behavior (gzip + br, only gzip applied → raw + flagged), so it's a deliberate choice, but for the identical-coding case it trades a body that used to read fine for one that doesn't. Worth considering keeping the deepest successful peel (flagged) rather than the original bytes when at least one layer came off; that keeps the genuinely double-compressed case working and stops the misconfig case from losing the body.

2.) Also, please double check the CLA Assistant check that is failing.

[liquidsec]

Cla failure is transient github issue

made an adjustment for the double gzip case