Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 4 additions & 30 deletions BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,42 +13,16 @@ on day one.

## Must land before the next release (0.6.0)

Items 1–4 (Anthropic transport classification, the Anthropic `http_client` type and
ownership, `phase` / `request_sent` on body errors, and the API reference pages) landed on
`failure-telemetry`. The numbers below are kept so existing references stay valid.
Nothing open. Items 1–6 landed: 1–4 (Anthropic transport classification, the Anthropic
`http_client` type and ownership, `phase` / `request_sent` on body errors, the API reference
pages) in PR 11; 5 (a failed turn's tokens are `None`, not `0`) and 6 (Dependabot PR 10 and
the `main` merge) after it. The numbers below are kept so existing references stay valid.

Left over from item 2, optional: `pyproject.toml` still allows `anthropic>=1.0`, which
includes SDKs built on httpx rather than httpx2. `AnthropicClient` reads the expected
client class from the installed SDK, so either works, but the type hints and docs name
`httpx2.Client`. Raise the floor to the first httpx2-based SDK once that version is known.

### 5. Decide: failed turns report `0` tokens instead of "not reported" [decide before next release]

**Status:** not implemented. `TurnRecord.input_tokens` / `output_tokens` are plain
`int = 0` (`summary.py`). `_Run._add_turn` (`loop.py`) fills `0` for a call that raised
before answering. The project rule is that `None` means "not reported", never zero.

**Concrete trigger:** set `SummaryOptions(timeline=True)` and let one model call fail. The
failed `TurnRecord` reports 0 input and 0 output tokens. That reads as "measured, and
free". A consumer summing the timeline can't tell a failed call from a cheap one without
also checking `error_type`.

**What implementing it takes:**
- Make both fields `int | None`, set them to `None` when `response is None`, and update
anything that sums them.
- Add a CHANGELOG "Changed" entry.

`TurnRecord` is in `__all__`, so this breaks code doing arithmetic on the fields. **Why it
is a release decision:** 0.6.0 is already a minor bump that changes `LoopObserver`. Making
this change in the same release costs nothing extra. Making it later costs another
breaking release. Either do it now, or record that it waits.

### 6. Release housekeeping for PR 11 [blocks next release]

- Merge Dependabot PR 10 (`actions/upload-artifact` 7.0.2 and `download-artifact` 8.0.2 in
`release.yml`; CI green) before tagging. Otherwise the release runs on the older
actions.

---

## High priority
Expand Down
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,9 @@ while `0.x`, a minor bump may break the public API (`agentloop.__all__`).
- A non-JSON body now reports its real status in `ResponseInvalid.status_code` and the message.
- `OpenAICompatClient` sends its URL and headers per request instead of configuring them on the
`httpx.Client`; code that replaced the private `_http` attribute still works.
- `TurnRecord.input_tokens` and `output_tokens` are `int | None`, and `None` (not `0`) for a
call that raised before answering: there was no usage to read, and `0` read as "measured,
and free". Code that sums the timeline must skip `None`.

### Fixed
- `TurnRecord.attempts` reported `1` for a call that failed after retries.
Expand Down
3 changes: 2 additions & 1 deletion docs/api/summary.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,8 @@ These options report conditions but do not stop or otherwise change the conversa

Metadata for one model call: duration, stop reason, token use, cost, attempts, tools, tool
errors, and whether a truncated call was discarded. When a call fails before returning,
`error_type` contains the exception class name. `attempt_log` has one `AttemptRecord` per
`error_type` contains the exception class name, and `input_tokens` / `output_tokens` are
`None` (not reported), never `0`. `attempt_log` has one `AttemptRecord` per
attempt (empty when the first attempt succeeded), and `attempts` is the real count even for a
call that failed.

Expand Down
2 changes: 1 addition & 1 deletion docs/guides/summaries.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ These features are off by default. Configure them together with `SummaryOptions`
- **`timeline`**: `summary.timeline` becomes a tuple of `TurnRecord`s, one per model call:
duration, stop reason, tokens, cost, attempts, the tools it led to and which of them
errored, and whether a truncated turn was discarded. A call that raised appears with its
`error_type`. The timeline also excludes conversation content.
`error_type`, and with `None` tokens: it was never answered, so nothing was measured. The timeline also excludes conversation content.

## How reporting stays separate from execution

Expand Down
4 changes: 2 additions & 2 deletions src/agentloop/loop.py
Original file line number Diff line number Diff line change
Expand Up @@ -334,8 +334,8 @@ def _add_turn(
model_seconds=model_seconds,
stop_reason=str(response.stop_reason) if response is not None else None,
error_type=type(error).__name__ if error is not None else None,
input_tokens=usage.input_tokens if usage else 0,
output_tokens=usage.output_tokens if usage else 0,
input_tokens=usage.input_tokens if usage else None,
output_tokens=usage.output_tokens if usage else None,
cost_usd=usage.cost_usd if usage else None,
attempts=(
response.attempts
Expand Down
5 changes: 3 additions & 2 deletions src/agentloop/summary.py
Original file line number Diff line number Diff line change
Expand Up @@ -120,8 +120,9 @@ class TurnRecord(BaseModel):
# None when the call raised before answering; `error_type` then says why.
stop_reason: str | None = None
error_type: str | None = None
input_tokens: int = 0
output_tokens: int = 0
# None, not 0, for a call that raised before answering: there was no usage to read.
input_tokens: int | None = None
output_tokens: int | None = None
cost_usd: float | None = None
attempts: int = 1
# Per-attempt detail; empty when the first attempt succeeded.
Expand Down
2 changes: 2 additions & 0 deletions tests/test_summary.py
Original file line number Diff line number Diff line change
Expand Up @@ -802,6 +802,8 @@ def test_a_failed_call_reports_its_real_attempts_and_whether_it_was_sent() -> No
assert summary.timeline is not None
assert summary.timeline[0].attempts == 1
assert summary.timeline[0].attempt_log[0].phase is FailurePhase.TLS
# Not answered, so not measured: None, never a free-looking 0.
assert (summary.timeline[0].input_tokens, summary.timeline[0].output_tokens) == (None, None)


def test_a_retried_failure_is_counted_and_the_turn_shows_every_attempt(
Expand Down
Loading