Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion architecture/client.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

## The internal terminal

The bottom of the middleware chain (the "terminal") is internal. It calls `self._httpx2_client.send(request)`, maps `httpx2` errors to `httpware` errors, and raises a `StatusError` subclass on 4xx/5xx. The error-mapping table (what `httpx2` exception maps to which `httpware` exception) lives at the terminal in `src/httpware/client.py`; status-keyed exceptions are looked up via the `STATUS_TO_EXCEPTION` table in `src/httpware/errors.py`. The same terminal lifecycle holds in both worlds: `Client.send` / `AsyncClient.send` enter the middleware chain first, and it is the internal terminal — `Client._terminal` / `AsyncClient._terminal` — that calls `httpx2.Client.send` / `httpx2.AsyncClient.send`.
The bottom of the middleware chain (the "terminal") is internal. It calls `self._httpx2_client.send(request)`, maps `httpx2` errors to `httpware` errors, and raises a `StatusError` subclass on 4xx/5xx. The error-mapping table (what `httpx2` exception maps to which `httpware` exception) lives in `src/httpware/_internal/exception_mapping.py` and fires at the terminal in `src/httpware/client.py`; status-keyed exceptions are looked up via the `STATUS_TO_EXCEPTION` table in `src/httpware/errors.py`. The same terminal lifecycle holds in both worlds: `Client.send` / `AsyncClient.send` enter the middleware chain first, and it is the internal terminal — `Client._terminal` / `AsyncClient._terminal` — that calls `httpx2.Client.send` / `httpx2.AsyncClient.send`.

## Sync/async parity

Expand Down
2 changes: 1 addition & 1 deletion architecture/errors.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ exc.response.request.url # URL of the failed request

`__repr__` and the `str()` summary redact URL userinfo (`user:pass@`) and mask the values of known-sensitive query and fragment parameters (e.g. `token`, `api_key`, `secret`) to avoid leaking credentials in tracebacks.

The error-mapping table (what `httpx2` exception maps to which `httpware` exception) lives at the terminal in `src/httpware/client.py`. Status-keyed exceptions are looked up via the `STATUS_TO_EXCEPTION` table in `src/httpware/errors.py`. Unknown 4xx falls back to `ClientStatusError`; unknown 5xx falls back to `ServerStatusError`.
The error-mapping table (what `httpx2` exception maps to which `httpware` exception) lives in `src/httpware/_internal/exception_mapping.py` and fires at the terminal in `src/httpware/client.py`. Status-keyed exceptions are looked up via the `STATUS_TO_EXCEPTION` table in `src/httpware/errors.py`. Unknown 4xx falls back to `ClientStatusError`; unknown 5xx falls back to `ServerStatusError`.

`TimeoutError` inherits from both `httpware.ClientError` and `builtins.TimeoutError` so `except builtins.TimeoutError` (the form `asyncio.wait_for` uses) also catches httpware-raised timeouts.

Expand Down
9 changes: 5 additions & 4 deletions architecture/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,9 @@ src/httpware/
│ └── _backoff.py # full-jitter helper (shared)
├── decoders/ # shared (ResponseDecoder + adapters)
└── _internal/
├── exception_mapping.py # map_httpx2_exception (shared)
├── import_checker.py # is_*_installed flags
├── observability.py # _emit_event
└── status.py # _raise_on_status_error, _is_streaming_body_*, STREAMING_BODY_MARKER
├── body_cap.py # max_response_body_bytes: validate, read-capped (sync+async)
├── exception_mapping.py # map_httpx2_exception + context-manager wrappers (shared)
├── import_checker.py # is_*_installed flags
├── observability.py # _emit_event
└── status.py # _raise_on_status_error, _is_streaming_body_*, STREAMING_BODY_MARKER
```
155 changes: 155 additions & 0 deletions planning/changes/2026-07-13.01-body-cap-module-extraction.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
---
summary: Extract the body-cap subsystem and the httpx2-exception-mapper context managers out of client.py into dedicated _internal/ modules.
---

# Design: Extract the body-cap subsystem into `_internal/body_cap.py`

## Summary

`client.py:38-212` holds ~180 lines of pure, self-contained code with zero
`self`/`Client`/`AsyncClient` coupling: the `max_response_body_bytes`
body-capping subsystem (`_validate_max_response_body_bytes`,
`_parse_content_length`, `_CapExceeded`, `_accumulate_capped`,
`_safe_extensions`, `_buffered_headers`, `_response_has_body`,
`_read_capped`, `_read_capped_async`) and two httpx2-exception-mapper context
managers (`_httpx2_exception_mapper`, `_httpx2_exception_mapper_sync`). This
change moves the body-cap subsystem into a new `_internal/body_cap.py`, and
folds the two context managers into the existing `_internal/exception_mapping.py`
next to the `map_httpx2_exception` function they wrap. No behavior change.

## Motivation

- `client.py` is 2,154 lines, by far the largest module in the package.
Everything genuinely coupled to `Client`/`AsyncClient` (`_terminal`,
`_prepare_request`, `stream()`) has to stay there; ~200 lines of this file
currently don't.
- The package already has a home for exactly this shape of code:
`_internal/status.py` holds `_raise_on_status_error` and the
streaming-body predicates, `_internal/exception_mapping.py` holds
`map_httpx2_exception` — both pure, self-contained, shared by both clients.
The body-cap subsystem and the exception-mapper context managers are the
same shape and never received the same treatment.
- Same-day precedent: `2026-06-23.01-retry-policy-extraction.md` and
`2026-06-23.02-decoder-resolver-extraction.md` pulled comparable
self-contained logic out of their hosting files into dedicated modules.
`2026-06-23.03-response-body-cap.md` landed the body-cap subsystem directly
in `client.py` the same day and never got the follow-up move.
- **Depth:** the body-cap subsystem's own tests (`tests/test_capped_read.py`,
`tests/test_capped_read_props.py`) already exercise it as a standalone
unit, importing the functions directly rather than through `Client`. The
seam already exists in the tests; the source hasn't caught up.
- **Deletion test:** delete `_internal/body_cap.py` after this change and the
~150 lines of cap-accumulation/header-rebuild logic reappear somewhere —
it's real, load-bearing complexity, not a pass-through. It earns the move.

## Design

### 1. `_internal/body_cap.py` — new module

Moves verbatim (no logic change):

- `_MAX_RESPONSE_BODY_BYTES_INVALID`, `_validate_max_response_body_bytes`
- `_parse_content_length`
- `_CapExceeded`, `_accumulate_capped`
- `_safe_extensions`
- `_WIRE_BODY_HEADERS`, `_BODILESS_STATUS`, `_buffered_headers`
- `_response_has_body`
- `_read_capped`, `_read_capped_async`

Needs its own imports: `httpx2`, `HTTPStatus` (from `http`), `Mapping` (from
`collections.abc`), and `ResponseTooLargeError` (from `httpware.errors`) —
all currently imported by `client.py` for this code's sake and dropped from
`client.py`'s import block once it moves (`Mapping` and
`ResponseTooLargeError` are used nowhere else in `client.py`; `HTTPStatus`
stays imported in `client.py` because `stream()` uses it independently at
`client.py:1152` and `:2146`).

`_build_default_decoders` (`client.py:172-190`) stays in `client.py`.
`architecture/decoders.md` documents its location explicitly ("`decoders=None`
resolves via `client.py:_build_default_decoders()`"); it is the same shape of
pure, self-contained code but moving it isn't part of this change's motivation
and would mean editing a Seam B contract that this change has no reason to
touch.

### 2. `_internal/exception_mapping.py` — gains two context managers

`_httpx2_exception_mapper` (async) and `_httpx2_exception_mapper_sync` (sync)
move in next to `map_httpx2_exception`, which they already wrap. Same
rationale as body-cap — pure, self-contained, sibling concern already has a
module — bundled into this change rather than a second near-identical PR.

### 3. `client.py` shrinks to three imported names

Only `_validate_max_response_body_bytes` (called once per `__init__`),
`_read_capped`, and `_read_capped_async` (called from `_terminal` and
`stream()`, both worlds) are actually referenced from inside `Client`/
`AsyncClient`. Every other moved name (`_parse_content_length`,
`_CapExceeded`, `_accumulate_capped`, `_safe_extensions`, `_buffered_headers`,
`_response_has_body`) is internal to `body_cap.py` and never imported by
`client.py`. `_httpx2_exception_mapper`/`_httpx2_exception_mapper_sync` are
imported by name, same call sites as today (`client.py:283`, `:1151`,
`:1255`, `:2145`).

```python
from httpware._internal.body_cap import _read_capped, _read_capped_async, _validate_max_response_body_bytes
from httpware._internal.exception_mapping import _httpx2_exception_mapper, _httpx2_exception_mapper_sync, map_httpx2_exception
```

Nine functions/one class collapse behind a three-name import in `client.py`
for body-cap, plus the two context managers for exception mapping.

### 4. `architecture/overview.md` — module-layout table

Add a `body_cap.py` row under `_internal/`; update `exception_mapping.py`'s
description to mention the context managers:

```text
└── _internal/
├── body_cap.py # max_response_body_bytes: validate, read-capped (sync+async)
├── exception_mapping.py # map_httpx2_exception + context-manager wrappers (shared)
├── import_checker.py # is_*_installed flags
├── observability.py # _emit_event
└── status.py # _raise_on_status_error, _is_streaming_body_*, STREAMING_BODY_MARKER
```

## Non-goals

- No behavior change. Cap validation, accumulation, header rebuild,
`ResponseTooLargeError` reasons, and exception-mapping order stay
byte-identical.
- Not moving `_build_default_decoders` (see Design §1).
- Not touching the public `max_response_body_bytes` parameter, its
validation error message, or `ResponseTooLargeError`'s shape.
- Not addressing the larger `Client`/`AsyncClient` sync/async duplication
(the per-verb methods, `_prepare_request`, `_request_with_body`) — a
separate, much larger candidate surfaced in the same architecture review,
deliberately deferred pending its own design call.

## Testing

- **Parity net:** rename `tests/test_capped_read.py` →
`tests/test_body_cap.py` and `tests/test_capped_read_props.py` →
`tests/test_body_cap_props.py`, updating their imports from
`httpware.client` to `httpware._internal.body_cap`. All existing
assertions stay unchanged — byte-identical behavior is the bar.
- Move `test_parse_content_length` (currently `tests/test_client_stream.py:446-447`,
testing a `body_cap.py` function directly rather than `stream()` behavior)
into `tests/test_body_cap.py`, updating its import.
- No new tests — this is a pure move with an existing, adequate net
(`test_body_cap.py`, `test_body_cap_props.py`, `test_client_body_cap.py`
for cap wiring through the client, `test_error_mapping_terminal.py` for
exception-mapping-at-terminal behavior).
- `just lint && just test` green; 100% coverage maintained.

## Risk

- **Import cycle** (unlikely × medium): `_internal/body_cap.py` imports
`httpware.errors.ResponseTooLargeError`. *Mitigation:* `_internal/status.py`
already imports from `httpware.errors` today with no cycle — same
direction, same precedent.
- **Missed call site during the move** (unlikely × low): a stale import left
in `client.py`, or a name imported but now unused. *Mitigation:* `just
lint` catches unused imports; `just test` exercises every moved function
through its existing suite.
- **Test-file rename loses git history readability** (certain × low):
`git mv` preserves blame across the rename; low impact either way.
146 changes: 146 additions & 0 deletions src/httpware/_internal/body_cap.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
"""Response-body cap enforcement: validate, read-capped (sync + async)."""

import typing
from collections.abc import Mapping
from http import HTTPStatus

import httpx2

from httpware.errors import ResponseTooLargeError


_MAX_RESPONSE_BODY_BYTES_INVALID = "max_response_body_bytes must be >= 1"


def _validate_max_response_body_bytes(cap: int | None) -> None:
"""Reject a non-None cap below 1. None means unbounded (the default)."""
if cap is not None and cap < 1:
raise ValueError(_MAX_RESPONSE_BODY_BYTES_INVALID)


def _parse_content_length(raw: str | None) -> int | None:
"""Return a non-negative int Content-Length, or None for missing/garbage. Never raises."""
if raw is None:
return None
try:
value = int(raw)
except ValueError:
return None
return value if value >= 0 else None


class _CapExceeded(Exception): # noqa: N818 — internal control-flow signal, not a user-facing error
"""Internal signal: decoded bytes crossed the cap mid-read. Carries bytes read so far."""

def __init__(self, *, read: int) -> None:
self.read = read
super().__init__(f"decoded body exceeded cap after {read} bytes")


def _accumulate_capped(chunks: typing.Iterable[bytes], cap: int) -> bytes:
"""Concatenate `chunks`, raising `_CapExceeded` the moment the running total exceeds `cap`.

Counts decoded bytes (the in-memory footprint). Grown in a single bytearray
so there is no transient list-plus-join double allocation.
"""
buf = bytearray()
for chunk in chunks:
buf += chunk
if len(buf) > cap:
raise _CapExceeded(read=len(buf))
return bytes(buf)


def _safe_extensions(extensions: Mapping[str, typing.Any]) -> dict[str, typing.Any]:
"""Copy response extensions, dropping the now-stale `network_stream`.

The rebuilt buffered Response never touches its network stream, so carrying a
consumed/closed one wholesale is sloppy. `http_version`/`reason_phrase` and
any other keys are preserved.
"""
return {key: value for key, value in extensions.items() if key != "network_stream"}


# Headers describing the wire encoding of the body. The accumulator yields the
# DECODED body, so these no longer apply; httpx2 recomputes content-length from
# the buffered content. Carrying content-encoding forward makes httpx2 try to
# re-decode already-decoded bytes and raise.
_WIRE_BODY_HEADERS = ("content-encoding", "content-length", "transfer-encoding")
_BODILESS_STATUS = frozenset({HTTPStatus.NO_CONTENT, HTTPStatus.NOT_MODIFIED}) # 204, 304


def _buffered_headers(headers: httpx2.Headers) -> httpx2.Headers:
"""Copy `headers`, stripping wire-encoding headers stale after decoding+buffering."""
out = httpx2.Headers(headers)
for name in _WIRE_BODY_HEADERS:
if name in out:
del out[name]
return out


def _response_has_body(method: str, status_code: int) -> bool:
"""Whether a response carries a message body (RFC 9110 §6.4.1).

HEAD responses and 204/304 never have a body regardless of a declared
Content-Length, so they must never trip the cap.
"""
return method.upper() != "HEAD" and status_code not in _BODILESS_STATUS


def _read_capped(response: httpx2.Response, cap: int, request: httpx2.Request) -> httpx2.Response:
"""Buffer a streaming sync `response` under `cap` decoded bytes; return a buffered Response.

Raises `ResponseTooLargeError` (reason="declared") if the declared
Content-Length already exceeds `cap` — before any byte is read — and
(reason="streamed") if the decoded body crosses `cap` mid-read. Does not
close `response`; the caller owns the stream lifecycle.
"""
if not _response_has_body(request.method, response.status_code):
response.read() # empty body; preserve the original response (and its headers)
return response
content_length = _parse_content_length(response.headers.get("content-length"))
if content_length is not None and content_length > cap:
raise ResponseTooLargeError(
status_code=response.status_code, limit=cap, content_length=content_length, reason="declared"
)
try:
content = _accumulate_capped(response.iter_bytes(), cap)
except _CapExceeded:
raise ResponseTooLargeError(
status_code=response.status_code, limit=cap, content_length=content_length, reason="streamed"
) from None
return httpx2.Response(
status_code=response.status_code,
headers=_buffered_headers(response.headers),
content=content,
request=request,
extensions=_safe_extensions(response.extensions),
history=response.history,
)


async def _read_capped_async(response: httpx2.Response, cap: int, request: httpx2.Request) -> httpx2.Response:
"""Async mirror of `_read_capped` (counts decoded bytes from `aiter_bytes`)."""
if not _response_has_body(request.method, response.status_code):
await response.aread() # empty body; preserve the original response (and its headers)
return response
content_length = _parse_content_length(response.headers.get("content-length"))
if content_length is not None and content_length > cap:
raise ResponseTooLargeError(
status_code=response.status_code, limit=cap, content_length=content_length, reason="declared"
)
buf = bytearray()
async for chunk in response.aiter_bytes():
buf += chunk
if len(buf) > cap:
raise ResponseTooLargeError(
status_code=response.status_code, limit=cap, content_length=content_length, reason="streamed"
)
return httpx2.Response(
status_code=response.status_code,
headers=_buffered_headers(response.headers),
content=bytes(buf),
request=request,
extensions=_safe_extensions(response.extensions),
history=response.history,
)
36 changes: 31 additions & 5 deletions src/httpware/_internal/exception_mapping.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,15 @@
"""httpx2 -> httpware exception mapping.
"""httpx2 -> httpware exception mapping + context-manager wrappers (shared).

Pure function used by both Client._terminal and AsyncClient._terminal,
and by both stream() methods. Clause ordering: TimeoutException ->
InvalidURL/CookieConflict -> NetworkError -> HTTPError (subclass before
parent so the right type wins).
map_httpx2_exception is a pure function used by both Client._terminal and
AsyncClient._terminal, and by both stream() methods. Clause ordering:
TimeoutException -> InvalidURL/CookieConflict -> NetworkError -> HTTPError
(subclass before parent so the right type wins). The two context managers
below wrap it for use as `with`/`async with` blocks around the httpx2 call.
"""

import contextlib
from collections.abc import AsyncIterator, Iterator

import httpx2

from httpware.errors import NetworkError, TimeoutError, TransportError # noqa: A004
Expand All @@ -26,3 +30,25 @@ def map_httpx2_exception(exc: BaseException) -> NetworkError | TimeoutError | Tr
if isinstance(exc, httpx2.HTTPError):
return TransportError(str(exc))
return TransportError(str(exc)) # pragma: no cover — defensive default; httpx2.HTTPError is the root


@contextlib.asynccontextmanager
async def _httpx2_exception_mapper() -> AsyncIterator[None]:
"""Map httpx2 exceptions to httpware exceptions. Shared by AsyncClient._terminal and stream()."""
try:
yield
except httpx2.HTTPError as exc:
raise map_httpx2_exception(exc) from exc
except (httpx2.InvalidURL, httpx2.CookieConflict) as exc:
raise map_httpx2_exception(exc) from exc


@contextlib.contextmanager
def _httpx2_exception_mapper_sync() -> Iterator[None]:
"""Map httpx2 exceptions to httpware exceptions. Sync sibling of _httpx2_exception_mapper."""
try:
yield
except httpx2.HTTPError as exc:
raise map_httpx2_exception(exc) from exc
except (httpx2.InvalidURL, httpx2.CookieConflict) as exc:
raise map_httpx2_exception(exc) from exc
Loading