From 18836da7a783d37a4d3416a2b10da8c3abe50e42 Mon Sep 17 00:00:00 2001 From: aaron Date: Sun, 18 Oct 2026 16:05:55 -0700 Subject: [PATCH] Implement the high-priority backlog items 7-12 - Truncated tool calls never lower max_tokens, and stop with LoopBudgetExceeded(kind="max_tokens") / TOKEN_LIMIT once the budget cannot grow, instead of re-billing the same cut-off. - An empty turn cut off by max_tokens is a truncation the loop answers with a bigger budget, not a retryable ResponseInvalid. - Unregistered (model-invented) tool names are reported as "" to summaries, metrics, spans and the event trace. - ToolSpec.timeout_seconds bounds a single tool call. - OTelObserver takes redact / allow_unredacted like EventTraceObserver. - Refresh stale docs and add the hosted-tools guide. Co-Authored-By: Claude Sonnet 5.5 --- BACKLOG.md | 235 ++---------------------------- CHANGELOG.md | 19 +++ docs/api/agent.md | 5 +- docs/api/errors.md | 4 + docs/api/loop.md | 9 +- docs/api/provider.md | 10 ++ docs/api/providers/anthropic.md | 11 ++ docs/api/providers/modal.md | 3 + docs/api/providers/openrouter.md | 4 + docs/api/summary.md | 4 +- docs/api/tooling.md | 1 + docs/api/tracing.md | 9 +- docs/api/transcript.md | 15 +- docs/guides/custom-providers.md | 15 ++ docs/guides/hosted-tools.md | 80 ++++++++++ docs/guides/providers.md | 15 ++ docs/guides/retries-and-errors.md | 4 +- docs/guides/summaries.md | 1 + docs/guides/tracing.md | 3 +- docs/guides/transcript-storage.md | 11 ++ docs/index.md | 1 + src/agentloop/agent.py | 9 +- src/agentloop/errors.py | 4 +- src/agentloop/events.py | 11 +- src/agentloop/loop.py | 46 ++++-- src/agentloop/observer.py | 5 + src/agentloop/provider.py | 17 ++- src/agentloop/summary.py | 2 + src/agentloop/tooling.py | 71 ++++++++- src/agentloop/tracing.py | 76 ++++++++-- tests/test_layering.py | 1 + tests/test_loop.py | 84 ++++++++++- tests/test_tooling.py | 68 +++++++++ tests/test_tracing.py | 75 +++++++++- 34 files changed, 651 insertions(+), 277 deletions(-) create mode 100644 docs/guides/hosted-tools.md diff --git a/BACKLOG.md b/BACKLOG.md index 6eccfe7..135a684 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -53,225 +53,22 @@ breaking release. Either do it now, or record that it waits. ## High priority -### 7. A truncated tool call can repeat at full cost until the iteration limit - -**Status:** bug. `loop.py` `_Run.drive`, after a turn ends with `_is_truncated_mid_call`: - -```python -self.max_tokens = min(self.max_tokens * 2, MAX_TOKENS_CEILING) -``` - -- **At the ceiling, nothing changes and it retries anyway.** Suppose a model keeps writing - a tool call larger than 64,000 output tokens (a big file write). Every iteration then - discards a full 64k-token turn and asks again with the same budget, up to - `max_iterations` (default 50). That is up to 3.2M billed output tokens for an answer - that can never fit. -- **Above the ceiling, the budget shrinks.** A caller who passes `max_tokens=100_000` is - cut to 64,000 on the first truncation. That makes the next truncation more likely. - -**What implementing it takes:** -- Never lower a budget: `max(self.max_tokens, min(self.max_tokens * 2, CEILING))`. -- When a truncation happens at a budget that can no longer grow, stop instead of retrying - blind. One option is a non-retryable error, or a dedicated `ConversationStatus`, that - says "the tool call does not fit in max_tokens". -- Add a test with `FakeClient` that scripts repeated truncations at the ceiling. - -### 8. An empty turn is retried as if it were transient - -**Status:** design gap. `ModelResponse.content` requires `min_length=1`, so an empty -assistant turn becomes `ResponseInvalid`. That error is `retryable = True`, so -`RetryingClient` re-sends the identical request. - -Two real cases where a retry is wasted money: - -- **Completions, `finish_reason: "length"` with empty content.** A reasoning model spends - the whole budget on hidden reasoning. With no readable reasoning there is no - `ThinkingPart`, so the turn is empty. It is retried `max_attempts` times at the same - `max_tokens`, every attempt billed, and then the conversation fails. The loop's - `max_tokens` doubling (item 7) never runs, because the loop never sees a response. -- **Anthropic, an empty `end_turn` right after tool results.** This is a known Claude - behaviour, and re-sending the same request does not reliably change it. - -**What implementing it takes:** treat "empty and truncated" as truncation, not -invalidity. Give the loop a way to see it, either a distinct non-retryable error or -`ModelResponse` allowing empty content when `stop_reason` is `MAX_TOKENS`, so it can raise -the budget or report `TRUNCATED`. Decide separately whether an empty `END_TURN` should -end the conversation with empty text or be retried once with a nudge. - -### 9. Tool names the model made up reach summaries, metrics and span attribute keys - -**Status:** breaks the "content-free" and "low-cardinality metrics" invariants (CLAUDE.md). -`SummaryCollector.on_tool_end`, `RepeatTracker`, `MetricsObserver` (`tool` attribute) and -`OTelObserver.on_conversation_end` (`agentloop.tool.calls.{name}` attribute **keys**) all -take the name from `call.tool_name`. Those names are whatever the model wrote, including -calls to tools that are not registered. `_execute` answers those with "Unknown tool", but -the name has already been counted. - -**Concrete trigger:** a model invents a new tool name each run (common with small -open-weight models on Modal or OpenRouter). Each name becomes a new metric series and a -new span attribute key, and model-written text ends up in a "content-free" summary. - -**What implementing it takes:** in `_run_one` / the collector, report unregistered names -under one fixed bucket such as `""`. The executor still answers the model -with the real name. Apply the same mapping in `RepeatTracker`'s reported name, the metrics -and the root span. Add a test asserting that a hallucinated name never appears in a -summary or metric. - -### 10. No timeout on individual tool-handler execution - -**Status:** not implemented. `run_conversation`'s `deadline_seconds` is checked only -*between* model calls (`loop.py`). Neither `tooling.run_tools`'s single-call path nor its -`ThreadPoolExecutor` path (`future.result()`, no `timeout=`) limits how long a tool -handler may run. - -**Concrete trigger:** register a tool whose handler makes a network call with no timeout of -its own, to an endpoint that never responds. The whole agent loop hangs forever, even with -`deadline_seconds` set. The deadline only fires before the *next* model call, and that -call never comes. - -**What implementing it takes:** a per-tool or per-call timeout, probably a `ToolSpec` field -that `run_tools` passes to `future.result(timeout=...)`, with an equivalent guard on the -single-call path. It needs a decision on what happens to a handler thread still running -after its timeout fires: Python cannot kill a thread. - -### 11. `OTelObserver` has no `redact`, unlike `EventTraceObserver` - -**Status:** not implemented. `EventTraceObserver(capture_content=True)` refuses to start -without a `redact` function, or an explicit `allow_unredacted=True`, and redacts every -captured string before clipping it. `OTelObserver(capture_content=True)` takes no `redact`. -System instructions, tool arguments and results, and error and retry messages go to the -exporter as written. `ERROR_DETAIL_CHARS` (200) limits their size, not their content. -`Agent(trace=True)` cannot enable capture at all, since `_with_tracing` does not pass -`capture_content` through. - -**Concrete trigger:** capture is on to debug a tool, and a tool result or a provider's 400 -body quotes an API key or a customer record. The event trace would redact it; the span -exports it. - -**What implementing it takes:** -- Add `redact: Callable[[str], str] | None` and `allow_unredacted` to `OTelObserver`, with - the event trace's rules: capture without either raises `ConfigurationError`; redact - before clipping; a redactor that raises drops the attribute instead of writing it. -- Decide whether `Agent(trace=...)` should accept capture options or keep capture off by - design. -- Optionally move the redaction helpers in `events.py` to a shared module, so the two - observers cannot drift. - -### 12. Documentation still describing 0.1 behaviour (hosted tools, Responses, `Origin`) - -**Status:** partly done. Already written: the `docs/api/agentloop.md` exports, -`docs/api/config.md` / `getting-started/configuration.md` for `OPENAI_API`, and the -`provider.md` coverage of `PAUSED` / `HostedTool` / `hosted_tools` / `origin`. Also -`transcript.md`'s `HostedToolPart` / `citations` / `Source` / `Citation` / -`Message.origin`, and the agentloop-tools README. - -**Stale pages (wrong today):** - -- `docs/api/transcript.md`: - - `final_text` is still described as "Concatenates". It now uses `join_text`, which - puts a space between separate blocks, but not across Anthropic's citation splits or - before closing punctuation. - - Missing: `add_assistant_message(parts, origin=None)`, `extend_last_assistant`, and the - `check_invariants` rule that hosted parts appear only in assistant messages. -- `docs/api/provider.md`: `supported_hosted_tools(client)`, and the optional - `hosted_tools` attribute a client may expose. -- `docs/api/agent.md`: a `ToolSpec` named like a hosted tool raises `ConfigurationError`. -- `docs/api/loop.md`: the `PAUSED` continuation. It sends no user message, joins the same - assistant turn, and counts as an iteration. -- `docs/api/providers/openrouter.md`, `modal.md`: `api="completions"` and empty - `hosted_tools` for both. OpenRouter replays `reasoning_details` from `Origin`. -- `docs/api/providers/anthropic.md`: - - `HOSTED_TOOLS` and `API = "messages"`. - - `WEB_SEARCH_BASIC` (`web_search_20250305`) is the default. `WEB_SEARCH_FILTERED` - (`web_search_20260209`) is sent only when `dynamic_filtering=True`. - - `hosted_tool_to_wire`. - - `pause_turn` maps to `PAUSED`. - - Assistant turns are replayed from `origin.raw`. -- `docs/api/tracing.md`: the `agentloop.hosted_tool.calls` span attribute (a count only). -- `docs/api/errors.md`, `docs/guides/retries-and-errors.md`: an OpenAI 429 whose - `type`/`code` is `insufficient_quota` / `credit_balance_exhausted` is - `InsufficientCredits` (not retried). -- `docs/guides/custom-providers.md`: `StopReason.PAUSED`. Also how a custom client opts in - to hosted tools (the `hosted_tools` attribute) and to exact replay - (`ModelResponse.origin`). - -**New content needed:** - -- `docs/guides/providers.md`: Chat Completions vs Responses. - - What differs, and which providers speak which. - - Why Responses runs **stateless**: `store: false`, the full transcript resent, item ids - stripped on replay. - - How to switch: `OPENAI_API`. -- `docs/guides/conversations.md` / `transcript-storage.md`: what `Message.origin` is, why - it can be large (encrypted search results), and that JSONL keeps it so a resumed - conversation replays exactly. -- A **hosted tools / web search** guide: - - `native_web_search(...)` from agentloop-tools: its options and each provider's handling - (OpenAI `search_context_size`; Anthropic `dynamic_filtering`, below). - - The option policy: unknown and restriction options raise; tuning options are logged. - - Approval is per request, not per search. - - Native and client-side search are separate choices. - - **Sources vs citations: the two evidence readers mean different things.** Say so - plainly, with this table: - - | | `sources(transcript)` | `citations(transcript)` | - |---|---|---| - | What it is | Every page the provider's search returned (`Source(url, title)`), deduplicated | A claim in the model's text tied to a page (`Citation`) | - | Comes from | `HostedToolPart.sources` (search result blocks / `web_search_call.action.sources`) | `TextPart.citations` | - | Anthropic, basic search (default) | Filled | Filled, with `cited_text` (the quoted source text) | - | Anthropic, `dynamic_filtering=True` | Filled | **Always empty** (see below) | - | OpenAI Responses | Filled (needs the `include` agentloop sends) | Filled, with `start`/`end` (a span of the answer), no `cited_text` | - | Use it for | "What did it look at?" A reading list, an audit trail | "Where did this sentence come from?" Footnotes, fact-checking | - - A source is not evidence that the answer used the page; a citation is. Neither one is - readable page text: Anthropic encrypts result content, and OpenAI returns only URLs. - - **`dynamic_filtering` (Anthropic only; default `False`).** - - *What it does:* `True` sends `web_search_20260209`. Claude writes and runs a Python - program, in an auto-provisioned code-execution sandbox, that calls the search, - filters the results and prints what it keeps. Claude reads only that output. - `False` sends `web_search_20250305`, and results go straight into the model's - context. - - *Why citations disappear:* a citation must point at a search-result block in the - model's context, and with filtering the results go to the program instead. - Measured live in September 2026 (`claude-opus-5`, 2 questions × 3 variants): - filtering on gave 0 citations every time; basic search gave 2–3; - `web_search_20260209` with `allowed_callers: ["direct"]` gave 2–5. So the cause is - the code path, not the tool version. - - *Examples to show:* one response of each. Use the frozen captures in - `agentloop-tools/experiments/examples/citations/`, for example - `shannon_A_dynamic.json` next to `shannon_B_basic.json`. `experiments/` is - gitignored, so on a fresh clone rerun `experiments/citations_experiment.py` first - (6 requests, a few cents). - - *Cost:* it did NOT save tokens on single simple searches (13,300 vs 12,827 input - tokens). Recommend it only for search-heavy requests that don't need citations. - - *Constraints:* Claude 4.6+ only; not ZDR-eligible; the response carries extra - `server_tool_use` / `code_execution_tool_result` blocks, surfaced as a - `HostedToolPart` of kind `code_execution`. - - *OpenAI:* has no equivalent. `dynamic_filtering=True` logs a warning and is ignored. - - **Domain policy (`agentloop_tools.web.DomainPolicy`).** It prevents and reports, and - deliberately does nothing else. - - *Modes:* `DomainPolicy.block([...])` or `DomainPolicy.allow([...])`, not both. - - *Matching:* a rule covers its host and every subdomain, but not a longer label - (`europa.eu` covers `eur-lex.europa.eu`, not `noteuropa.eu`). A rule may add a path - prefix in which `*` matches within the path. - - *Prevent:* `policy.native_web_search(**options)` sends the list to the provider. - Passing a domain option there as well is an error. - - *Report:* `policy.audit(transcript, since=n)` returns `Violation(url, where, - message_index, rule)`. `where` is `"source"`, `"citation"` or `"text"`. To audit one - turn, pass the `len(transcript.messages)` taken before `converse`. - - *Text scanning:* full URLs in both modes, with balanced parentheses kept. Bare host - mentions are checked in block mode only. - - *The hard limit:* with native search, no library can stop the model READING a page. - The policy prevents what the provider can and reports the rest. Only client-side - search (item 22) filters before the model sees anything. - - *Live evidence (September 2026):* - - `block(["wikipedia.org"])` held on both providers (0 leaked sources out of 10 - and 31). - - Asked for Wikipedia's URL, Claude wrote 4 Wikipedia links from memory, which the - audit reported as `"text"` violations. - - `allow(["python.org"])` on OpenAI returned 27 sources, all on the list. - -**Verify while writing:** each example runs. +Items 7–12 (truncated-call retry cost, empty truncated turns, made-up tool names in telemetry, +tool-handler timeouts, `OTelObserver` redaction, and the stale 0.1-era docs) landed on +`failure-telemetry`. The numbers stay so references remain valid. + +Left over: + +- **Empty `END_TURN` (item 8).** An empty non-truncated turn is still `ResponseInvalid` and is + retried verbatim. Decide whether it should end the conversation with empty text or be retried + once with a nudge. +- **Hung tool threads (item 10).** A timed-out handler's thread is abandoned, not killed. A tool + that holds a lock or connection keeps it. `ToolSpec.timeout_seconds` defaults to `None`. +- **Shared redaction helpers (item 11).** `events.py` and `tracing.py` each implement + redact-then-clip; move them to one module so they cannot drift. +- **Hosted-tools guide examples (item 12).** The `agentloop-tools` parts of + `docs/guides/hosted-tools.md` (`native_web_search`, `DomainPolicy`) were written from this + backlog and not run here. --- diff --git a/CHANGELOG.md b/CHANGELOG.md index cadbd3e..7721589 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,8 +36,27 @@ while `0.x`, a minor bump may break the public API (`agentloop.__all__`). `agentloop.provider.errors` counter (`error_type`, `phase`, `request_sent` of `true`, `false` or `unknown`). Tracing: `agentloop.attempts`, `gen_ai.response.id`, and `agentloop.error.*` on the model and root spans. +- `ToolSpec.timeout_seconds` (and `@tool(timeout_seconds=...)`): a call that runs longer gets + an error result and the loop moves on. The handler's thread is abandoned, not killed. + Default `None` (no limit). +- `OTelObserver(redact=..., allow_unredacted=...)`, with the event trace's rules. +- `ConversationStatus.TOKEN_LIMIT` and `LoopBudgetExceeded(kind="max_tokens")`. ### Changed +- `OTelObserver(capture_content=True)` now needs `redact` or `allow_unredacted=True`, else it + raises `ConfigurationError`. Captured text is redacted, then clipped; a redactor that raises + drops the attribute. Exceptions are recorded as an `exception` event with a redacted message + and no stack trace. +- A turn truncated mid-tool-call no longer lowers a `max_tokens` above the 64,000 ceiling, and + when the budget cannot grow the run stops with `LoopBudgetExceeded(kind="max_tokens")` + (status `token_limit`) instead of re-billing the same cut-off up to `max_iterations`. +- `ModelResponse.content` may be empty when `stop_reason` is `MAX_TOKENS`. The loop treats such + a turn (a reasoning model that spent the budget thinking) as a truncation and raises the + budget, instead of the adapter raising a retryable `ResponseInvalid`. Other empty turns are + still `ResponseInvalid`. +- Calls to tools that are not registered are reported to observers, summaries, metrics, spans + and the event trace under `""`, not the model's invented name. The model still + receives the real name in its "Unknown tool" result. - TLS verification, DNS and proxy failures are no longer retried: they fail the same way every time. They raise `ProviderUnreachable`, still a `ProviderUnavailable`, with `retryable = False`. - A 3xx is reported as `UnexpectedRedirect` (with the target host only), not as a retryable diff --git a/docs/api/agent.md b/docs/api/agent.md index 27b7f53..6167497 100644 --- a/docs/api/agent.md +++ b/docs/api/agent.md @@ -164,8 +164,9 @@ kept as internal state. `client` is borrowed: an agent constructed directly never closes it. `tools` accepts local `ToolSpec` objects and provider-run `HostedTool` objects. Values produced by `@tool` can be -passed directly. Duplicate local tool names and unsupported hosted tools fail during -construction. `deps` is the one runtime dependency bundle passed to local tool handlers, +passed directly. Duplicate local tool names, unsupported hosted tools, and a local `ToolSpec` named like a +hosted tool kind (for example `web_search`) fail during construction; native and client-side +tools are separate, explicit choices. `deps` is the one runtime dependency bundle passed to local tool handlers, defaulting to `Deps()`. `trace=True` combines an `OTelObserver` with the supplied observer. `metrics=True` adds a `MetricsObserver`. `summary_options` configures the optional timeline, repeated-call detection, and limit warnings described in diff --git a/docs/api/errors.md b/docs/api/errors.md index 5410fb6..15de427 100644 --- a/docs/api/errors.md +++ b/docs/api/errors.md @@ -65,6 +65,10 @@ and `retry_wait_seconds`. Only `RateLimited`, `ProviderUnavailable`, `ProviderTi retryable: they are faults in the environment, and a retry fails the same way. A refused or reset connection and a read error stay retryable. +`InsufficientCredits` covers OpenRouter's 402 and an OpenAI 429 whose error `type`/`code` is +`insufficient_quota` or `credit_balance_exhausted`. The latter is a 429 only in name: it is +not retried, since waiting does not add credit. + An `AgentLoopError` raised from an active conversation also carries its `ConversationSummary` as `.summary`. Errors raised before a conversation starts may have `.summary = None`. diff --git a/docs/api/loop.md b/docs/api/loop.md index f7946b3..272f554 100644 --- a/docs/api/loop.md +++ b/docs/api/loop.md @@ -34,8 +34,13 @@ run_conversation( Mutates and returns `transcript`. Each iteration sends a provider-independent `ModelRequest`, appends the response, executes requested tools, and appends their results. A -response truncated mid-tool-call is discarded and retried with a doubled token allowance up -to the ceiling. +response truncated mid-tool-call, or truncated before producing anything readable, is discarded +and retried with a doubled token allowance up to the 64,000 ceiling. A budget is never lowered, +and when it cannot grow the run raises `LoopBudgetExceeded` with `kind="max_tokens"`. + +A provider that pauses a turn (`StopReason.PAUSED`, for example Anthropic's `pause_turn` while +a hosted tool runs) is continued without sending a user message: the continuation joins the +same assistant message, and each continuation counts as an iteration. The deadline is checked between turns. A provider call, its retries, or a started tool batch may finish after it. `None` disables the deadline. Exhaustion raises `LoopBudgetExceeded`; diff --git a/docs/api/provider.md b/docs/api/provider.md index 9d44d8a..573f4b1 100644 --- a/docs/api/provider.md +++ b/docs/api/provider.md @@ -83,6 +83,16 @@ complete(request: ModelRequest) -> ModelResponse See [custom providers](../guides/custom-providers.md) for an implementation. +## `supported_hosted_tools` + +```text +supported_hosted_tools(client) -> frozenset[str] +``` + +The hosted tool kinds a client can run: its optional `hosted_tools` attribute, or an empty set +when it has none. A client that says nothing supports nothing, so a custom provider needs no +new attribute to stay an `LLMClient`. + ## `HostedTool` A provider-run capability declared on each request. `kind` names the capability, such as diff --git a/docs/api/providers/anthropic.md b/docs/api/providers/anthropic.md index 5f3919c..4229306 100644 --- a/docs/api/providers/anthropic.md +++ b/docs/api/providers/anthropic.md @@ -9,6 +9,12 @@ Anthropic Messages API integration. The SDK is imported only when needed and req - `DEFAULT_MODEL = "claude-opus-5"` - `DEFAULT_TIMEOUT_SECONDS = 600.0` - `CONTEXT_OVERFLOW_STOP_REASON = "model_context_window_exceeded"` +- `API = "messages"` +- `HOSTED_TOOLS = frozenset({"web_search"})`: the hosted kinds `AnthropicClient` runs. +- `WEB_SEARCH_BASIC = "web_search_20250305"`: the default web search tool. +- `WEB_SEARCH_FILTERED = "web_search_20260209"`: sent only when the hosted tool has + `dynamic_filtering=True` (Claude 4.6+). Claude then runs a program that filters results, so + the response carries no citations; see the [hosted tools guide](../../guides/hosted-tools.md). ## Translation helpers @@ -18,6 +24,8 @@ These public helpers convert between shared Agentloop types and the Anthropic AP - `message_to_wire(message) -> dict[str, Any]`: converts one provider-independent message to one Anthropic message. - `request_to_wire(request, *, model) -> dict[str, Any]`: builds a Messages API payload. +- `hosted_tool_to_wire(tool) -> dict[str, Any]`: maps a `HostedTool` to the Anthropic tool + declaration. - `usage_from_wire(raw) -> Usage`: normalizes Anthropic usage fields. - `response_from_wire(message) -> ModelResponse`: normalizes a response and may raise `ContextLengthExceeded` or `ResponseInvalid`, both with `phase=BODY` and @@ -57,4 +65,7 @@ close. The SDK follows redirects by default; pass a client with `follow_redirect have a 3xx reported. - `__enter__()` returns the client; `__exit__()` closes it. +`pause_turn` maps to `StopReason.PAUSED`. Assistant turns are replayed from `Origin.raw`, so +encrypted search results and signatures go back exactly as received. + When `api_key=None`, credential resolution is delegated to the Anthropic SDK. diff --git a/docs/api/providers/modal.md b/docs/api/providers/modal.md index 219c533..5703551 100644 --- a/docs/api/providers/modal.md +++ b/docs/api/providers/modal.md @@ -36,5 +36,8 @@ client = build_modal_client( ) ``` +It speaks Chat Completions (`api="completions"`) and has no hosted tools +(`hosted_tools` is empty). + The returned client exposes `name`, `model`, `complete(request)`, `close()`, and context manager methods. Construct through `build_client` or wrap with `RetryingClient` for retries. diff --git a/docs/api/providers/openrouter.md b/docs/api/providers/openrouter.md index b78804f..fd76dd5 100644 --- a/docs/api/providers/openrouter.md +++ b/docs/api/providers/openrouter.md @@ -34,6 +34,10 @@ client = build_openrouter_client( ) ``` +It speaks Chat Completions (`api="completions"`) and has no hosted tools +(`hosted_tools` is empty). For reasoning models it replays `reasoning_details` from the +message's `Origin`, unchanged and in order. + The returned client exposes `name`, `model`, `complete(request)`, `close()`, and context manager methods. Add `RetryingClient` explicitly or construct through `build_client` for automatic retries. diff --git a/docs/api/summary.md b/docs/api/summary.md index ed2aae7..2fe5bc6 100644 --- a/docs/api/summary.md +++ b/docs/api/summary.md @@ -6,10 +6,10 @@ prompts, model responses, tool arguments, tool results, or exception messages. ## `ConversationStatus` Describes how a conversation ended: `completed`, `refused`, `truncated`, -`iteration_limit`, `deadline_exceeded`, `provider_error`, `error`, or `cancelled`. +`iteration_limit`, `deadline_exceeded`, `token_limit`, `provider_error`, `error`, or `cancelled`. Use `status.succeeded` to check whether the call returned normally and -`status.budget_exhausted` to check whether it reached an iteration or time limit. +`status.budget_exhausted` to check whether it reached an iteration, time or output-token limit. ## `ConversationSummary` diff --git a/docs/api/tooling.md b/docs/api/tooling.md index df327f1..4cf18d9 100644 --- a/docs/api/tooling.md +++ b/docs/api/tooling.md @@ -33,6 +33,7 @@ A frozen Pydantic model containing the model-visible definition and handler. | `description` | non-empty `str` | Model-visible instructions | | `parameters` | `dict[str, Any]` | Bare handwritten JSON Schema | | `handler` | `Handler` | Callable whose first argument is the deps bundle | +| `timeout_seconds` | `float \| None` | Per-call limit; `None` waits forever. On expiry the model gets an error result and the handler thread is abandoned (Python cannot kill it), so the handler should still set its own network timeouts | ```python def ping(deps: ToolDeps) -> str: diff --git a/docs/api/tracing.md b/docs/api/tracing.md index 439c746..3c7bb30 100644 --- a/docs/api/tracing.md +++ b/docs/api/tracing.md @@ -17,6 +17,8 @@ OTelObserver( model, agent_name="agentloop", capture_content=False, + redact=None, + allow_unredacted=False, ) ``` @@ -28,12 +30,17 @@ span for each local tool call. It implements all observer callbacks: - `on_response()` records response metadata and usage, then ends the model span. - `on_retry()` adds an event to the current model span. - `on_model_error()` records the phase and send state of a failed call on its model span. +- `on_response()` also sets `agentloop.hosted_tool.calls`, a count of provider-run tool calls + (never the queries). - `on_tool_start()` starts a tool span keyed by call ID. - `on_tool_end()` records result metadata and ends the matching span. - `on_text_delta()` is currently a no-op reserved for streaming. With `capture_content=False`, prompts, system instructions, arguments, results, and error -messages are excluded. With it enabled those values may reach the exporter. +messages are excluded. Enabling it requires `redact` (a `str -> str` function) or +`allow_unredacted=True`, otherwise `ConfigurationError`. Every captured string is redacted +and then clipped; a `redact` that raises drops that attribute. `Agent(trace=True)` never +captures content: build an `OTelObserver` yourself to opt in. One observer instance supports one active conversation. Overlapping `conversation()` contexts raise `RuntimeError`. Tool callbacks are synchronized because tools can run in diff --git a/docs/api/transcript.md b/docs/api/transcript.md index 69ada02..f61c878 100644 --- a/docs/api/transcript.md +++ b/docs/api/transcript.md @@ -72,14 +72,19 @@ conversation accurately. Mutable Pydantic model with public `messages: list[Message]`, defaulting to an empty list. - `add_user_message(text) -> None`: appends one user `TextPart` message. -- `add_assistant_message(parts) -> None`: appends an assistant message. +- `add_assistant_message(parts, origin=None) -> None`: appends an assistant message, keeping + the provider's `Origin` for exact replay. +- `extend_last_assistant(parts, origin=None) -> None`: continues the last assistant message + (a turn the provider paused), merging `origin.raw` when both origins match. Raises + `TranscriptError` when the last message is not from the assistant. - `add_tool_results(results) -> None`: verifies that every pending call is answered exactly once, orders results by call order, and appends a user-role result message. Raises `TranscriptError` on mismatch. - `pending_tool_calls() -> tuple[ToolCallPart, ...]`: calls in the final assistant message, or an empty tuple. - `check_invariants() -> None`: validates call/result adjacency, completeness, and unique - call IDs; raises `TranscriptError` if the transcript is unsafe to send. + call IDs, and that hosted-tool parts appear only in assistant messages; raises + `TranscriptError` if the transcript is unsafe to send. ```python transcript = Transcript() @@ -113,5 +118,7 @@ errors. It intentionally does not call `check_invariants()`. final_text(transcript) -> str ``` -Concatenates all text parts in the last assistant message. Returns `""` when there is no -assistant message or it contains no text. +Joins the text parts of the last assistant message with `join_text`: a space goes between +separate blocks ("I'll search." then "The answer is..."), but not across Anthropic's mid-sentence +citation splits or before closing punctuation. Returns `""` when there is no assistant +message or it contains no text. diff --git a/docs/guides/custom-providers.md b/docs/guides/custom-providers.md index b03c8b3..916ed84 100644 --- a/docs/guides/custom-providers.md +++ b/docs/guides/custom-providers.md @@ -37,4 +37,19 @@ Custom providers should convert vendor failures into the [`ProviderError` hierarchy](../api/errors.md), preserve unknown stop reasons as `StopReason.UNKNOWN`, and keep vendor-specific types out of the public method signature. +## Optional capabilities + +- **Pause.** Return `StopReason.PAUSED` when the provider stopped partway through a turn it is + running itself. The loop sends the turn back unchanged, with no user message, and joins the + continuation to the same assistant message. +- **Hosted tools.** Expose a `hosted_tools` attribute (a `frozenset` of kinds such as + `"web_search"`) to accept `HostedTool` declarations. Without it the client supports none, + and `Agent` rejects them at construction. See `supported_hosted_tools`. +- **Exact replay.** Set `ModelResponse.origin` to an `Origin(provider=..., api=..., model=..., + raw=...)` holding your wire form of the turn. Replay it when `origin.matches(provider, api)` + and fall back to the canonical parts otherwise. +- **Empty turns.** `ModelResponse.content` may be empty only when `stop_reason` is + `StopReason.MAX_TOKENS`; the loop answers that with a bigger budget. Any other empty turn + should raise `ResponseInvalid`. + See the [`provider` API](../api/provider.md). diff --git a/docs/guides/hosted-tools.md b/docs/guides/hosted-tools.md new file mode 100644 index 0000000..89b8ac8 --- /dev/null +++ b/docs/guides/hosted-tools.md @@ -0,0 +1,80 @@ +# Hosted tools and web search + +A `HostedTool` is a capability the provider runs for you, such as web search. Declare it with +`Agent(tools=[...])` beside your local `ToolSpec`s. `native_web_search(...)` in agentloop-tools +builds one. Native and client-side search are separate, explicit choices: a local `ToolSpec` +named like a hosted kind raises `ConfigurationError`. + +```python +from agentloop import Agent, HostedTool + +agent = Agent([HostedTool(kind="web_search")], client=client) +``` + +Support depends on the client (`supported_hosted_tools(client)`). Anthropic and OpenAI over +Responses support `web_search`; Chat Completions providers do not. Approval is per request, not +per search: `before_tool` never sees a hosted call. + +**Options.** Unknown options and restriction options a provider cannot honour raise; tuning +options it cannot honour are logged and ignored. OpenAI takes `search_context_size`. + +## Sources and citations + +The two evidence readers mean different things. + +| | `sources(transcript)` | `citations(transcript)` | +|---|---|---| +| What it is | Every page the provider's search returned (`Source(url, title)`), deduplicated | A claim in the model's text tied to a page (`Citation`) | +| Comes from | `HostedToolPart.sources` | `TextPart.citations` | +| Anthropic, basic search (default) | Filled | Filled, with `cited_text` | +| Anthropic, `dynamic_filtering=True` | Filled | **Always empty** | +| OpenAI Responses | Filled | Filled, with `start`/`end` (a span of the answer), no `cited_text` | +| Use it for | "What did it look at?" A reading list, an audit trail | "Where did this sentence come from?" Footnotes, fact-checking | + +A source is not evidence that the answer used the page; a citation is. Neither is readable page +text: Anthropic encrypts result content, and OpenAI returns only URLs. + +## `dynamic_filtering` (Anthropic only, default `False`) + +`True` sends `web_search_20260209`. Claude writes and runs a Python program in a code-execution +sandbox that calls the search, filters the results and prints what it keeps; Claude reads only +that output. `False` sends `web_search_20250305` and results go straight into the model's +context. + +- **Citations disappear** because a citation must point at a search-result block in the model's + context, and with filtering the results go to the program. Measured live in September 2026 + (`claude-opus-5`, 2 questions x 3 variants): filtering on gave 0 citations every time, basic + search 2-3, and `web_search_20260209` with `allowed_callers: ["direct"]` 2-5. The cause is the + code path, not the tool version. Frozen captures are in + `agentloop-tools/experiments/examples/citations/` (`shannon_A_dynamic.json` beside + `shannon_B_basic.json`); `experiments/` is gitignored, so rerun + `experiments/citations_experiment.py` on a fresh clone (6 requests, a few cents). +- **Cost:** it did not save tokens on simple searches (13,300 vs 12,827 input tokens). Use it for + search-heavy requests that do not need citations. +- **Constraints:** Claude 4.6+ only; not ZDR-eligible; the response carries extra + `server_tool_use` / `code_execution_tool_result` blocks, surfaced as a `HostedToolPart` of + kind `code_execution`. +- **OpenAI** has no equivalent: `dynamic_filtering=True` logs a warning and is ignored. + +## Domain policy (`agentloop_tools.web.DomainPolicy`) + +It prevents and reports, and does nothing else. + +- **Modes:** `DomainPolicy.block([...])` or `DomainPolicy.allow([...])`, not both. +- **Matching:** a rule covers its host and every subdomain, not a longer label (`europa.eu` + covers `eur-lex.europa.eu`, not `noteuropa.eu`). A rule may add a path prefix where `*` + matches within the path. +- **Prevent:** `policy.native_web_search(**options)` sends the list to the provider; passing a + domain option there as well is an error. +- **Report:** `policy.audit(transcript, since=n)` returns `Violation(url, where, message_index, + rule)`, with `where` of `"source"`, `"citation"` or `"text"`. To audit one turn, pass the + `len(transcript.messages)` taken before `converse`. +- **Text scanning:** full URLs in both modes, balanced parentheses kept; bare host mentions in + block mode only. +- **The hard limit:** with native search no library can stop the model reading a page. The + policy prevents what the provider can and reports the rest. Only client-side search filters + before the model sees anything. +- **Live evidence (September 2026):** `block(["wikipedia.org"])` held on both providers (0 leaked + sources out of 10 and 31). Asked for Wikipedia's URL, Claude wrote 4 Wikipedia links from + memory, which the audit reported as `"text"` violations. `allow(["python.org"])` on OpenAI + returned 27 sources, all on the list. diff --git a/docs/guides/providers.md b/docs/guides/providers.md index 2272c05..3d9f69e 100644 --- a/docs/guides/providers.md +++ b/docs/guides/providers.md @@ -73,3 +73,18 @@ A TLS, DNS or proxy failure raises `ProviderUnreachable` immediately, without re See the [provider reference](../api/providers/index.md) or [implement a custom provider](custom-providers.md). + +## Chat Completions or Responses + +OpenAI can be reached over two APIs, chosen with `OPENAI_API` (`responses`, the default, or +`completions`; `Settings.openai_api`). + +| | Chat Completions | Responses | +|---|---|---| +| Spoken by | OpenAI, OpenRouter, Modal/vLLM | OpenAI | +| Hosted tools (web search) | Not supported | Supported | +| Reasoning state | Provider-specific (OpenRouter `reasoning_details`) | Reasoning items replayed | + +Responses runs **stateless**: every request sends `store: false` and the full transcript, and +item ids are stripped on replay, so nothing depends on the provider remembering a conversation. +OpenRouter and Modal always use Chat Completions. diff --git a/docs/guides/retries-and-errors.md b/docs/guides/retries-and-errors.md index 5c41f25..cee0d47 100644 --- a/docs/guides/retries-and-errors.md +++ b/docs/guides/retries-and-errors.md @@ -17,7 +17,9 @@ except AgentLoopError as error: ``` `RateLimited`, `ProviderUnavailable`, `ProviderTimeout`, and `ResponseInvalid` are retryable. -Authentication, bad-request, context-length, missing-model, and credit errors are not. +Authentication, bad-request, context-length, missing-model, and credit errors are not. That +includes an OpenAI 429 whose `type`/`code` is `insufficient_quota` or +`credit_balance_exhausted`: it raises `InsufficientCredits` and is not retried. `ProviderUnreachable` (TLS verification, DNS, a refusing proxy) and `UnexpectedRedirect` are not retried, because retrying cannot fix them. Both are faults in the environment. A refused diff --git a/docs/guides/summaries.md b/docs/guides/summaries.md index 6d7b58f..c8ba3f9 100644 --- a/docs/guides/summaries.md +++ b/docs/guides/summaries.md @@ -41,6 +41,7 @@ Branch on `summary.status` (a `ConversationStatus`), never on an exception messa | `truncated` | The final answer hit `max_tokens` | returned | | `iteration_limit` | Out of iterations | `LoopBudgetExceeded` | | `deadline_exceeded` | Out of wall-clock time | `LoopBudgetExceeded` | +| `token_limit` | A turn cannot fit in `max_tokens` and the budget cannot grow | `LoopBudgetExceeded` | | `provider_error` | A `ProviderError`, after any retries | raised | | `error` | Anything else raised | raised | | `cancelled` | `KeyboardInterrupt` and similar | raised | diff --git a/docs/guides/tracing.md b/docs/guides/tracing.md index cdbaf1f..39348a9 100644 --- a/docs/guides/tracing.md +++ b/docs/guides/tracing.md @@ -35,7 +35,8 @@ OpenTelemetry trace. Retries appear as events on the related model-request span. Prompts, tool arguments, tool results, and provider error messages are not captured by default. Construct an -`OTelObserver(..., capture_content=True)` only when your data-handling policy permits that +`OTelObserver(..., capture_content=True, redact=scrub)` (a redactor is required, as for the event +trace) only when your data-handling policy permits that content to leave the process. One `OTelObserver` can trace only one conversation at a time. The conversation span records its status, identifiers, iterations, limits, timing, retries, diff --git a/docs/guides/transcript-storage.md b/docs/guides/transcript-storage.md index 66916c7..8d55364 100644 --- a/docs/guides/transcript-storage.md +++ b/docs/guides/transcript-storage.md @@ -22,3 +22,14 @@ lines are ignored. Invalid or truncated JSON raises instead of silently discardi mid-turn. Call `check_invariants()` before sending a loaded transcript to a provider. See the [`transcript` API](../api/transcript.md). + +## `Message.origin` + +An assistant `Message` may carry an `Origin`: the provider's own wire form of that turn +(`provider`, `api`, `model`, `raw`). Adapters replay it instead of rebuilding the turn from the +canonical parts, which keeps things the canonical form cannot express, such as Anthropic's +encrypted search results and signed thinking, or OpenRouter's `reasoning_details`. + +It can be large. JSONL keeps it so a resumed conversation replays exactly. Drop it +(`message.model_copy(update={"origin": None})`) if you accept a replay rebuilt from canonical +parts, or switch providers: an origin from a different provider or API is ignored on replay. diff --git a/docs/index.md b/docs/index.md index 901d556..a8c955f 100644 --- a/docs/index.md +++ b/docs/index.md @@ -16,6 +16,7 @@ leaving application policy in your code. - [Tools and approval](guides/tools.md) - [Conversations and transcripts](guides/conversations.md) - [Providers](guides/providers.md) +- [Hosted tools and web search](guides/hosted-tools.md) - [Implement a custom provider](guides/custom-providers.md) - [Conversation summaries](guides/summaries.md) - [Event traces](guides/event-trace.md) diff --git a/src/agentloop/agent.py b/src/agentloop/agent.py index f639e8e..8b80593 100644 --- a/src/agentloop/agent.py +++ b/src/agentloop/agent.py @@ -74,6 +74,7 @@ def tool( *, name: str | None = None, description: str | None = None, + timeout_seconds: float | None = None, ) -> Callable[[Callable[..., Any]], ToolSpec]: """Turn a plain function into a ToolSpec. @@ -84,6 +85,9 @@ def tool( The decorated function does not take `deps`. If it wants one, it declares a keyword-only `deps` parameter; either way `deps` is stripped from the signature the model is shown. + + `timeout_seconds` bounds one call: past it the model is told the tool timed out and + the loop moves on. The handler's thread is abandoned, not killed (see `ToolSpec`). """ def decorate(fn: Callable[..., Any]) -> ToolSpec: @@ -99,6 +103,7 @@ def decorate(fn: Callable[..., Any]) -> ToolSpec: description=resolved_description, parameters=parameters, handler=_make_handler(fn), + timeout_seconds=timeout_seconds, ) return decorate @@ -167,7 +172,9 @@ def __init__( `trace=True` builds an `OTelObserver` (from `client.name`/`client.model` and `name`) and merges it with `observer`. Spans go nowhere until you separately call `agentloop.tracing.configure_tracing(endpoint)`; see its - docstring. An `OTelObserver` also holds a non-reentrant lock: one + docstring. It never captures content: that needs a `redact` function, so build + your own `OTelObserver(capture_content=True, redact=...)` and pass it as + `observer` instead. An `OTelObserver` also holds a non-reentrant lock: one `Agent(..., trace=True)` instance can run only one conversation at a time. Calling `run()`/`converse()` again while one is already in flight (e.g. from a second concurrent request sharing this instance) raises diff --git a/src/agentloop/errors.py b/src/agentloop/errors.py index fbef363..4e5e4d7 100644 --- a/src/agentloop/errors.py +++ b/src/agentloop/errors.py @@ -55,7 +55,7 @@ class ToolError(AgentLoopError): class LoopBudgetExceeded(AgentLoopError): - """The agent loop ran out of iterations or wall-clock time. + """The agent loop ran out of iterations, wall-clock time or output budget. `kind`, `limit` and `actual` say which budget and by how much, so a handler never parses the message. They are None on an instance built without them. @@ -65,7 +65,7 @@ def __init__( self, message: str, *, - kind: Literal["iterations", "deadline"] | None = None, + kind: Literal["iterations", "deadline", "max_tokens"] | None = None, limit: float | None = None, actual: float | None = None, ) -> None: diff --git a/src/agentloop/events.py b/src/agentloop/events.py index 2acf1a2..3f9ddd0 100644 --- a/src/agentloop/events.py +++ b/src/agentloop/events.py @@ -26,7 +26,7 @@ from typing import Any from agentloop.errors import ConfigurationError, ProviderError -from agentloop.observer import BudgetKind, NullObserver +from agentloop.observer import UNREGISTERED_TOOL, BudgetKind, NullObserver from agentloop.provider import ModelRequest, ModelResponse from agentloop.summary import ConversationContext, ConversationSummary from agentloop.transcript import Message, ToolCallPart, ToolResultPart @@ -73,6 +73,7 @@ def __init__( self._sent_messages = 0 # request messages already written (capture only) self._system_sent = False self._tool_started: dict[str, float] = {} + self._known_tools: frozenset[str] = frozenset() # from the last request # --- plumbing --- @@ -163,9 +164,13 @@ def on_request(self, request: ModelRequest) -> None: fields["new_messages"] = [self._message(m) for m in fresh] if send_system and request.system is not None: fields["system"] = self._text(request.system) + with self._lock: + self._known_tools = frozenset(tool.name for tool in request.tools) self._emit("request", **fields) def on_response(self, response: ModelResponse) -> None: + with self._lock: + known = self._known_tools usage = response.usage fields: dict[str, Any] = { "stop_reason": str(response.stop_reason), @@ -182,7 +187,9 @@ def on_response(self, response: ModelResponse) -> None: "attempts": response.attempts, "retry_wait_seconds": response.retry_wait_seconds or None, "tool_calls": [ - part.tool_name for part in response.content if isinstance(part, ToolCallPart) + part.tool_name if part.tool_name in known else UNREGISTERED_TOOL + for part in response.content + if isinstance(part, ToolCallPart) ], } if self._capture: diff --git a/src/agentloop/loop.py b/src/agentloop/loop.py index 88d0074..6882c32 100644 --- a/src/agentloop/loop.py +++ b/src/agentloop/loop.py @@ -33,7 +33,7 @@ TurnRecord, new_conversation_id, ) -from agentloop.tooling import Registry, ToolDeps, run_tools +from agentloop.tooling import Registry, ToolDeps, reported_tool_name, run_tools from agentloop.transcript import ToolCallPart, ToolResultPart, Transcript logger = logging.getLogger(__name__) @@ -44,14 +44,17 @@ DEFAULT_DEADLINE_SECONDS = 600.0 -def _is_truncated_mid_call(response: ModelResponse) -> bool: - """A turn cut off while the model was still writing a tool call's arguments. +def _is_unusable_truncation(response: ModelResponse) -> bool: + """A turn cut off before it said anything the loop can keep. - Unrecoverable: the arguments are partial, so it cannot be executed, and it - cannot be skipped either (that leaves an unanswered tool call). + Either the model was still writing a tool call's arguments (partial, so it cannot + be executed, and cannot be skipped either: that leaves an unanswered tool call), + or it produced nothing readable at all (a reasoning model that spent the whole + budget thinking). A bigger budget is the only thing that can help. """ - return response.stop_reason is StopReason.MAX_TOKENS and any( - isinstance(part, ToolCallPart) for part in response.content + return response.stop_reason is StopReason.MAX_TOKENS and ( + not response.content + or any(isinstance(part, ToolCallPart) for part in response.content) ) @@ -62,6 +65,8 @@ def _status_for(error: BaseException) -> ConversationStatus: return ConversationStatus.ITERATION_LIMIT if error.kind == "deadline": return ConversationStatus.DEADLINE_EXCEEDED + if error.kind == "max_tokens": + return ConversationStatus.TOKEN_LIMIT return ConversationStatus.ERROR # a budget error raised without saying which if isinstance(error, ProviderError): return ConversationStatus.PROVIDER_ERROR @@ -348,8 +353,10 @@ def _add_turn( else error.attempt_log if isinstance(error, ProviderError) else () ), discarded=discarded, - tools=tuple(r.tool_name for r in results), - tool_errors=tuple(r.tool_name for r in results if r.is_error), + tools=tuple(reported_tool_name(self.registry, r.tool_name) for r in results), + tool_errors=tuple( + reported_tool_name(self.registry, r.tool_name) for r in results if r.is_error + ), tool_seconds=tool_seconds, ) ) @@ -398,14 +405,25 @@ def drive(self) -> ConversationStatus: self.model_seconds += spent watcher.on_response(response) - if _is_truncated_mid_call(response): - self.max_tokens = min(self.max_tokens * 2, MAX_TOKENS_CEILING) + if _is_unusable_truncation(response): + self._add_turn(spent, response=response, discarded=True) + # Never lower a budget the caller chose above the ceiling. + grown = max(self.max_tokens, min(self.max_tokens * 2, MAX_TOKENS_CEILING)) + if grown == self.max_tokens: + # Retrying the identical request would bill another full turn + # for the same cut-off, until the iteration limit. + raise LoopBudgetExceeded( + f"the model's turn does not fit in max_tokens={self.max_tokens}", + kind="max_tokens", + limit=self.max_tokens, + actual=response.usage.output_tokens, + ) + self.max_tokens = grown logger.warning( - "turn truncated mid-tool-call; discarding it and retrying " + "turn truncated before it was usable; discarding it and retrying " "with max_tokens=%d", self.max_tokens, ) - self._add_turn(spent, response=response, discarded=True) continue # F1: do NOT append this turn if self.continuing: @@ -430,7 +448,7 @@ def drive(self) -> ConversationStatus: calls = [part for part in response.content if isinstance(part, ToolCallPart)] for tool_name, count in self.tracker.observe(calls): - watcher.on_stall(tool_name, count) + watcher.on_stall(reported_tool_name(self.registry, tool_name), count) tools_started = time.monotonic() results = run_tools(response.content, self.registry, self.deps, watcher) diff --git a/src/agentloop/observer.py b/src/agentloop/observer.py index 7c2bbae..8a9db1f 100644 --- a/src/agentloop/observer.py +++ b/src/agentloop/observer.py @@ -24,6 +24,11 @@ type BudgetKind = Literal["iterations", "deadline"] +# What telemetry reports for a call to a tool that is not registered. The model wrote +# that name, so it is unbounded and not ours to export (see CLAUDE.md invariants). The +# model itself still gets the real name back in its "Unknown tool" result. +UNREGISTERED_TOOL = "" + class LoopObserver(Protocol): """Every interesting thing that happens. Implementations may no-op.""" diff --git a/src/agentloop/provider.py b/src/agentloop/provider.py index fb7bd7c..253fb9d 100644 --- a/src/agentloop/provider.py +++ b/src/agentloop/provider.py @@ -9,7 +9,7 @@ from enum import StrEnum from typing import Any, Literal, Protocol -from pydantic import BaseModel, ConfigDict, Field +from pydantic import BaseModel, ConfigDict, Field, model_validator from agentloop.errors import FailurePhase, ProviderError from agentloop.transcript import ContentPart, Message, Origin @@ -142,13 +142,16 @@ def from_error(cls, attempt: int, error: ProviderError, *, wait: float = 0.0) -> class ModelResponse(BaseModel): """One model turn, normalized. - `content` must be non-empty: an adapter that would return an empty turn - raises ValidationError instead, which it converts to ResponseInvalid. + `content` must be non-empty, with one exception: a turn cut off by `max_tokens` + before it produced anything readable (a reasoning model that spent the whole + budget thinking). That is a truncation the loop can answer with a bigger budget, + not an invalid response worth re-sending unchanged. Any other empty turn raises + ValidationError, which an adapter converts to ResponseInvalid. """ model_config = ConfigDict(frozen=True, protected_namespaces=()) - content: tuple[ContentPart, ...] = Field(min_length=1) + content: tuple[ContentPart, ...] stop_reason: StopReason usage: Usage model: str @@ -165,6 +168,12 @@ class ModelResponse(BaseModel): # The adapter's own wire form of this turn, for exact replay. See Origin. origin: Origin | None = None + @model_validator(mode="after") + def _empty_only_when_truncated(self) -> ModelResponse: + if not self.content and self.stop_reason is not StopReason.MAX_TOKENS: + raise ValueError("content must not be empty unless the turn hit max_tokens") + return self + class LLMClient(Protocol): """The seam. One method, no hidden state, no vendor types in the signature.""" diff --git a/src/agentloop/summary.py b/src/agentloop/summary.py index 743e945..489d47c 100644 --- a/src/agentloop/summary.py +++ b/src/agentloop/summary.py @@ -32,6 +32,7 @@ class ConversationStatus(StrEnum): TRUNCATED = "truncated" # the final answer hit max_tokens ITERATION_LIMIT = "iteration_limit" # LoopBudgetExceeded: out of iterations DEADLINE_EXCEEDED = "deadline_exceeded" # LoopBudgetExceeded: out of wall clock + TOKEN_LIMIT = "token_limit" # LoopBudgetExceeded: a tool call that cannot fit max_tokens PROVIDER_ERROR = "provider_error" # a ProviderError, after any retries ERROR = "error" # anything else raised CANCELLED = "cancelled" # KeyboardInterrupt and friends @@ -50,6 +51,7 @@ def budget_exhausted(self) -> bool: return self in { ConversationStatus.ITERATION_LIMIT, ConversationStatus.DEADLINE_EXCEEDED, + ConversationStatus.TOKEN_LIMIT, } diff --git a/src/agentloop/tooling.py b/src/agentloop/tooling.py index 61dcc60..76e1f5e 100644 --- a/src/agentloop/tooling.py +++ b/src/agentloop/tooling.py @@ -12,8 +12,10 @@ from __future__ import annotations +import contextvars import inspect import logging +import threading from collections.abc import Callable, Sequence from concurrent.futures import ThreadPoolExecutor from typing import Any, Protocol @@ -21,7 +23,7 @@ from pydantic import BaseModel, ConfigDict, Field, SkipValidation from agentloop.errors import ToolError -from agentloop.observer import LoopObserver, guarded +from agentloop.observer import UNREGISTERED_TOOL, LoopObserver, guarded from agentloop.provider import ToolSchema from agentloop.transcript import ContentPart, ToolCallPart, ToolResultPart @@ -67,6 +69,9 @@ class ToolSpec(BaseModel): # of the hand-written prompt contract. parameters: SkipValidation[dict[str, Any]] handler: Handler + # Seconds one call may run before the model is told it timed out. None (the + # default) waits forever. Python cannot kill a thread: see `_call_with_timeout`. + timeout_seconds: float | None = Field(default=None, gt=0) class Registry: @@ -120,6 +125,41 @@ def schemas(self) -> tuple[ToolSchema, ...]: # --- the executor ------------------------------------------------------------ +class _ToolTimeout(Exception): + """A handler ran past its `timeout_seconds`. Internal: `_execute` turns it into a result.""" + + +def _call_with_timeout( + handler: Handler, args: tuple[Any, ...], kwargs: dict[str, Any], timeout: float +) -> str: + """Run `handler` in its own thread and stop waiting after `timeout` seconds. + + The thread is a daemon and is NOT killed (Python cannot): a handler that never + returns keeps running, and keeps whatever it holds, until it finishes or the process + exits. A pool thread would be worse -- the executor's shutdown joins it, so the hang + would come back one level up. A handler with a timeout should still set its own + socket timeouts; this only guarantees the LOOP moves on. + """ + outcome: list[tuple[bool, Any]] = [] + context = contextvars.copy_context() # keep the caller's trace context + + def target() -> None: + try: + outcome.append((True, context.run(handler, *args, **kwargs))) + except BaseException as exc: # noqa: BLE001 - handed back to the caller's thread + outcome.append((False, exc)) + + thread = threading.Thread(target=target, name="agentloop-tool", daemon=True) + thread.start() + thread.join(timeout) + if not outcome: + raise _ToolTimeout + ok, value = outcome[0] + if not ok: + raise value + return value # type: ignore[no-any-return] + + def _execute(call: ToolCallPart, registry: Registry, deps: ToolDeps) -> ToolResultPart: """Run one tool call. NEVER raises: every failure becomes an error result.""" @@ -152,8 +192,19 @@ def fail(message: str) -> ToolResultPart: return fail(f"Invalid arguments for {call.tool_name}: {exc}") try: - result = spec.handler(*bound.args, **bound.kwargs) + if spec.timeout_seconds is None: + result = spec.handler(*bound.args, **bound.kwargs) + else: + result = _call_with_timeout( + spec.handler, bound.args, bound.kwargs, spec.timeout_seconds + ) content = str(result) # inside the guard: __str__ is someone else's code too + except _ToolTimeout: + logger.warning("tool %s timed out after %ss", call.tool_name, spec.timeout_seconds) + return fail( + f"Error: {call.tool_name} timed out after {spec.timeout_seconds:g}s " + "and was abandoned; its outcome is unknown." + ) except ToolError as exc: return fail(f"Error: {exc}") except Exception as exc: # noqa: BLE001 - totality is the invariant: every call gets exactly one result @@ -165,12 +216,24 @@ def fail(message: str) -> ToolResultPart: def _run_one( call: ToolCallPart, registry: Registry, deps: ToolDeps, observer: LoopObserver ) -> ToolResultPart: - observer.on_tool_start(call) + # Observers see a fixed bucket for an unregistered name; the transcript keeps the real one. + seen_call = call + if registry.get(call.tool_name) is None: + seen_call = call.model_copy(update={"tool_name": UNREGISTERED_TOOL}) + observer.on_tool_start(seen_call) result = _execute(call, registry, deps) - observer.on_tool_end(call, result) + seen_result = ( + result if seen_call is call else result.model_copy(update={"tool_name": UNREGISTERED_TOOL}) + ) + observer.on_tool_end(seen_call, seen_result) return result +def reported_tool_name(registry: Registry, name: str) -> str: + """The name telemetry may report for a call to `name`.""" + return name if registry.get(name) is not None else UNREGISTERED_TOOL + + def run_tools( content: Sequence[ContentPart], registry: Registry, diff --git a/src/agentloop/tracing.py b/src/agentloop/tracing.py index a512fd2..7c2c4e0 100644 --- a/src/agentloop/tracing.py +++ b/src/agentloop/tracing.py @@ -16,7 +16,7 @@ import logging import threading -from collections.abc import Iterator +from collections.abc import Callable, Iterator from contextlib import contextmanager from typing import TYPE_CHECKING, Any @@ -24,7 +24,7 @@ from opentelemetry.context import Context from opentelemetry.trace import Span, Status, StatusCode -from agentloop.errors import ProviderError +from agentloop.errors import ConfigurationError, ProviderError from agentloop.observer import BudgetKind from agentloop.provider import ModelRequest, ModelResponse from agentloop.summary import ConversationContext, ConversationSummary @@ -63,6 +63,11 @@ class OTelObserver: With `capture_content` off, nothing the model, a tool or a provider wrote reaches a span: that includes error messages, which quote arguments and code. + + Turning it on needs a `redact` function (or an explicit `allow_unredacted=True`), + the same rule as `EventTraceObserver`. Every captured string goes through `redact` + first, and error details are clipped to `ERROR_DETAIL_CHARS` after it. A `redact` + that raises drops that one attribute rather than writing it unredacted. """ def __init__( @@ -72,13 +77,22 @@ def __init__( model: str, agent_name: str = DEFAULT_AGENT_NAME, capture_content: bool = False, + redact: Callable[[str], str] | None = None, + allow_unredacted: bool = False, ) -> None: + if capture_content and redact is None and not allow_unredacted: + raise ConfigurationError( + "capture_content=True needs a `redact` function, so captured text is " + "scrubbed before it is exported. Pass allow_unredacted=True to capture " + "it as-is." + ) self._provider = provider self._model = model # The APPLICATION's name, because it is what a trace reader is looking for. self._agent_name = agent_name # Off by default: a coding agent's prompts contain the user's source code. self._capture = capture_content + self._redact = redact self._root: Span | None = None self._model_span: Span | None = None self._tool_spans: dict[str, Span] = {} @@ -91,6 +105,37 @@ def __init__( self._context: Context | None = None self._running = threading.Lock() + def _text(self, value: str, limit: int | None = None) -> str | None: + """`value` redacted, then clipped; None when it must not be written. + + The caller drops the attribute on None. Redact comes first so a secret cut in + half by the clip is never exported. + """ + if self._redact is not None: + try: + value = self._redact(value) + except Exception: + logger.exception("redact raised; dropping the attribute") + return None + return value if limit is None else value[:limit] + + def _set_text(self, span: Span, key: str, value: str) -> None: + text = self._text(value) + if text is not None: + span.set_attribute(key, text) + + def _error_detail(self, exc: BaseException) -> str | None: + return self._text(str(exc), ERROR_DETAIL_CHARS) if self._capture else None + + def _record_exception(self, span: Span, exc: BaseException) -> None: + """An `exception` event with a redacted message and NO stack trace: the SDK's + own `record_exception` writes both unredacted.""" + attributes: dict[str, Any] = {"exception.type": type(exc).__name__} + detail = self._error_detail(exc) + if detail is not None: + attributes["exception.message"] = detail + span.add_event("exception", attributes) + @contextmanager def conversation(self) -> Iterator[None]: if not self._running.acquire(blocking=False): @@ -108,7 +153,7 @@ def conversation(self) -> Iterator[None]: }, # Both default to writing the exception MESSAGE onto the span, # which may quote content; the except below does it by policy. - record_exception=self._capture, + record_exception=False, set_status_on_exception=False, ) as root: self._root = root @@ -122,10 +167,9 @@ def conversation(self) -> Iterator[None]: # bug here should be logged, not swap out what the caller sees. try: root.set_attribute("error.type", type(exc).__name__) - detail = ( - str(exc)[:ERROR_DETAIL_CHARS] if self._capture else None - ) - root.set_status(Status(StatusCode.ERROR, detail)) + if self._capture: + self._record_exception(root, exc) + root.set_status(Status(StatusCode.ERROR, self._error_detail(exc))) self._end_unfinished(exc) except Exception: logger.exception( @@ -150,7 +194,7 @@ def _end_unfinished(self, exc: BaseException) -> None: try: span.set_attribute("error.type", type(exc).__name__) if self._capture: - span.record_exception(exc) + self._record_exception(span, exc) span.set_status(Status(StatusCode.ERROR)) finally: span.end() @@ -173,7 +217,7 @@ def on_request(self, request: ModelRequest) -> None: if request.temperature is not None: span.set_attribute("gen_ai.request.temperature", request.temperature) if self._capture and request.system: - span.set_attribute("gen_ai.system_instructions", request.system) + self._set_text(span, "gen_ai.system_instructions", request.system) self._model_span = span def on_response(self, response: ModelResponse) -> None: @@ -233,7 +277,9 @@ def on_retry(self, attempt: int, error: Exception) -> None: attributes.update(_failure_attributes(error, prefix="agentloop.retry")) if self._capture: # A provider's 400 can quote the request back, so this is content. - attributes["agentloop.retry.message"] = str(error)[:ERROR_DETAIL_CHARS] + detail = self._error_detail(error) + if detail is not None: + attributes["agentloop.retry.message"] = detail self._model_span.add_event("retry", attributes) def on_model_error(self, error: Exception) -> None: @@ -260,7 +306,9 @@ def on_tool_start(self, call: ToolCallPart) -> None: # tests/test_layering.py stays shorter for it. import json - attributes["gen_ai.tool.call.arguments"] = json.dumps(call.arguments) + arguments = self._text(json.dumps(call.arguments)) + if arguments is not None: + attributes["gen_ai.tool.call.arguments"] = arguments span = _tracer.start_span( f"execute_tool {call.tool_name}", context=self._context, attributes=attributes ) @@ -279,10 +327,12 @@ def on_tool_end(self, call: ToolCallPart, result: ToolResultPart) -> None: "agentloop.tool.result_bytes", len(result.content.encode("utf-8")) ) if self._capture: - span.set_attribute("gen_ai.tool.call.result", result.content) + self._set_text(span, "gen_ai.tool.call.result", result.content) if result.is_error: # The error text quotes arguments and file contents: content, not metadata. - detail = result.content[:ERROR_DETAIL_CHARS] if self._capture else None + detail = ( + self._text(result.content, ERROR_DETAIL_CHARS) if self._capture else None + ) span.set_status(Status(StatusCode.ERROR, detail)) finally: span.end() diff --git a/tests/test_layering.py b/tests/test_layering.py index ff49808..ae15963 100644 --- a/tests/test_layering.py +++ b/tests/test_layering.py @@ -39,6 +39,7 @@ "collections", # Counter, abc "concurrent", # ThreadPoolExecutor, in the tool executor "contextlib", # nullcontext / contextmanager, observers and tracing + "contextvars", # a timed-out tool runs in a thread that keeps the caller's context "enum", # StopReason "hashlib", # stall detection compares tool calls by hash, in memory only "inspect", # signature().bind(), tool executor and @tool diff --git a/tests/test_loop.py b/tests/test_loop.py index c365aaf..783496e 100644 --- a/tests/test_loop.py +++ b/tests/test_loop.py @@ -6,10 +6,16 @@ from pydantic import BaseModel, ConfigDict from agentloop.errors import LoopBudgetExceeded -from agentloop.loop import ListObserver, run_conversation +from agentloop.loop import ( + MAX_TOKENS_CEILING, + ListObserver, + run_conversation, + run_conversation_result, +) from agentloop.provider import HostedTool, ModelRequest, ModelResponse, StopReason, Usage from agentloop.providers._openai_compat import Quirks, response_from_wire from agentloop.providers.fake import FakeClient +from agentloop.summary import ConversationStatus, SummaryOptions from agentloop.tooling import ApprovalHook, Registry, ToolSpec, allow_all from agentloop.transcript import ( ContentPart, @@ -112,6 +118,82 @@ def test_a_turn_truncated_mid_call_is_discarded() -> None: assert client.requests[1].max_tokens > client.requests[0].max_tokens +def test_a_truncation_that_cannot_grow_stops_instead_of_retrying() -> None: + # Already at the ceiling: the identical request would bill the same cut-off again. + truncated = resp(echo_call("c1", "hel"), stop=StopReason.MAX_TOKENS) + client = FakeClient(truncated, truncated, truncated) + with pytest.raises(LoopBudgetExceeded) as caught: + run(client, max_tokens=MAX_TOKENS_CEILING) + assert caught.value.kind == "max_tokens" + assert caught.value.summary is not None + assert caught.value.summary.status is ConversationStatus.TOKEN_LIMIT + assert caught.value.summary.status.budget_exhausted + assert len(client.requests) == 1 + + +def test_repeated_truncations_stop_once_the_budget_reaches_the_ceiling() -> None: + truncated = resp(echo_call("c1", "hel"), stop=StopReason.MAX_TOKENS) + client = FakeClient(*[truncated] * 10) + with pytest.raises(LoopBudgetExceeded): + run(client, max_tokens=16_000) + # 16k -> 32k -> 64k, then it cannot grow: three calls, not max_iterations. + assert [r.max_tokens for r in client.requests] == [16_000, 32_000, 64_000] + + +def test_a_budget_above_the_ceiling_is_never_lowered() -> None: + client = FakeClient( + resp(echo_call("c1", "hel"), stop=StopReason.MAX_TOKENS), + resp(TextPart(text="never reached")), + ) + with pytest.raises(LoopBudgetExceeded): + run(client, max_tokens=100_000) + assert [r.max_tokens for r in client.requests] == [100_000] + + +def test_an_empty_truncated_turn_raises_the_budget() -> None: + # A reasoning model that spent the whole budget thinking: nothing readable at all. + client = FakeClient( + resp(stop=StopReason.MAX_TOKENS), + resp(TextPart(text="recovered")), + ) + transcript = run(client) + assert [m.role for m in transcript.messages] == ["user", "assistant"] + assert client.requests[1].max_tokens > client.requests[0].max_tokens + + +def test_an_empty_turn_that_was_not_truncated_is_still_invalid() -> None: + with pytest.raises(ValueError, match="empty"): + resp(stop=StopReason.END_TURN) + + +def test_a_made_up_tool_name_is_reported_under_one_bucket() -> None: + watcher = ListObserver() + client = FakeClient( + resp(ToolCallPart(id="c1", tool_name="invented_xyz", arguments={}), + stop=StopReason.TOOL_USE), + resp(TextPart(text="done")), + ) + result = run_conversation_result( + _transcript(), client, REGISTRY, ToyDeps(), observer=watcher, + summary_options=SummaryOptions(timeline=True), + ) + summary = result.summary + assert summary.tool_calls == {"": 1} + assert summary.tool_errors == {"": 1} + assert "invented_xyz" not in summary.model_dump_json() + # the model still hears the real name + reply = result.transcript.messages[2].content[0] + assert "invented_xyz" in reply.content # type: ignore[union-attr] + started = [call for kind, call in watcher.events if kind == "tool_start"] + assert [c.tool_name for c in started] == [""] # type: ignore[attr-defined] + + +def _transcript() -> Transcript: + transcript = Transcript() + transcript.add_user_message("go") + return transcript + + def test_a_truncated_text_turn_is_kept() -> None: # MAX_TOKENS with NO tool call is a clipped but usable answer — keep it transcript = run( diff --git a/tests/test_tooling.py b/tests/test_tooling.py index 32f27f5..a109b61 100644 --- a/tests/test_tooling.py +++ b/tests/test_tooling.py @@ -261,3 +261,71 @@ def test_run_tools_called_directly_survives_a_raising_observer() -> None: calls = [call("echo", text="a"), call("refuse")] results = run_tools(calls, REGISTRY, ToyDeps(), BrokenTracer()) assert [r.call_id for r in results] == ["c-echo", "c-refuse"] + + +# --- a per-tool timeout --- + + +def test_a_tool_that_outlives_its_timeout_gets_an_error_result() -> None: + registry = Registry([ToolSpec( + name="slow", description="the slow tool", parameters=schema(ms="integer"), + handler=slow, timeout_seconds=0.05, + )]) + started = time.monotonic() + [result] = run_tools( + [ToolCallPart(id="c1", tool_name="slow", arguments={"ms": 2000})], + registry, ToyDeps(), + ) + assert time.monotonic() - started < 1.0 + assert result.is_error + assert "timed out" in result.content + assert result.call_id == "c1" + + +def test_a_timed_tool_that_finishes_in_time_returns_normally() -> None: + registry = Registry([ToolSpec( + name="slow", description="the slow tool", parameters=schema(ms="integer"), + handler=slow, timeout_seconds=5, + )]) + [result] = run_tools( + [ToolCallPart(id="c1", tool_name="slow", arguments={"ms": 1})], registry, ToyDeps() + ) + assert result.content == "1" + assert not result.is_error + + +def test_a_timed_tool_keeps_its_error_contract() -> None: + registry = Registry([ + ToolSpec(name="refuse", description="d", parameters=schema(), + handler=refuse, timeout_seconds=5), + ToolSpec(name="boom", description="d", parameters=schema(), + handler=boom, timeout_seconds=5), + ]) + refused, crashed = run_tools( + [ToolCallPart(id="a", tool_name="refuse", arguments={}), + ToolCallPart(id="b", tool_name="boom", arguments={})], + registry, ToyDeps(), + ) + assert refused.content == "Error: the input was not in the expected shape" + assert crashed.content == "Error: RuntimeError: kaboom" + + +def test_one_hung_tool_does_not_hold_up_its_siblings() -> None: + registry = Registry([ + ToolSpec(name="slow", description="d", parameters=schema(ms="integer"), + handler=slow, timeout_seconds=0.05), + spec("echo", echo, text="string"), + ]) + started = time.monotonic() + hung, fine = run_tools( + [ToolCallPart(id="a", tool_name="slow", arguments={"ms": 2000}), + ToolCallPart(id="b", tool_name="echo", arguments={"text": "x"})], + registry, ToyDeps(), + ) + assert time.monotonic() - started < 1.0 # the pool's shutdown must not wait for it + assert hung.is_error and fine.content == "x" + + +def test_a_timeout_must_be_positive() -> None: + with pytest.raises(ValueError): + ToolSpec(name="t", description="d", parameters=schema(), handler=boom, timeout_seconds=0) diff --git a/tests/test_tracing.py b/tests/test_tracing.py index 111777b..6e527ae 100644 --- a/tests/test_tracing.py +++ b/tests/test_tracing.py @@ -1,4 +1,4 @@ -from collections.abc import Mapping +from collections.abc import Callable, Mapping from typing import Any import pytest @@ -10,7 +10,7 @@ from pydantic import BaseModel, ConfigDict from agentloop.agent import Agent -from agentloop.errors import ProviderUnavailable, RateLimited, ToolError +from agentloop.errors import ConfigurationError, ProviderUnavailable, RateLimited, ToolError from agentloop.loop import run_conversation from agentloop.provider import ModelRequest, ModelResponse, StopReason, Usage from agentloop.providers._openai_compat import OpenAICompatClient @@ -76,7 +76,7 @@ def exporter() -> InMemorySpanExporter: def drive(exporter: InMemorySpanExporter, *responses: ModelResponse, - capture: bool = False) -> None: + capture: bool = False, redact: Callable[[str], str] | None = None) -> None: exporter.clear() transcript = Transcript() transcript.add_user_message("go") @@ -85,7 +85,8 @@ def drive(exporter: InMemorySpanExporter, *responses: ModelResponse, # agent_name is the APPLICATION's, and this is the seam: a test, the CLI and # an eval each pass their own, and the library ships only the default. observer=OTelObserver(provider="fake", model="m", agent_name="harness", - capture_content=capture), + capture_content=capture, redact=redact, + allow_unredacted=capture and redact is None), ) @@ -498,3 +499,69 @@ def test_a_successful_call_records_its_attempts_and_request_id( drive(exporter, resp(TextPart(text="x"))) chat = attrs(next(s for s in exporter.get_finished_spans() if s.name == "chat m")) assert chat["agentloop.attempts"] == 1 + + +# --- capture needs a redactor, like the event trace --- + + +def scrub(text: str) -> str: + return text.replace("sk-real-key", "[REDACTED]") + + +def test_capture_without_redact_is_refused() -> None: + with pytest.raises(ConfigurationError, match="redact"): + OTelObserver(provider="fake", model="m", capture_content=True) + + +def test_captured_text_is_redacted_before_it_is_exported( + exporter: InMemorySpanExporter, +) -> None: + drive(exporter, resp(echo_call("API_KEY = 'sk-real-key'"), stop=StopReason.TOOL_USE), + resp(TextPart(text="done")), capture=True, redact=scrub) + tool = next(s for s in exporter.get_finished_spans() if s.name == "execute_tool echo") + assert "sk-real-key" not in str(attrs(tool)) + assert "[REDACTED]" in str(attrs(tool)["gen_ai.tool.call.arguments"]) + assert "[REDACTED]" in str(attrs(tool)["gen_ai.tool.call.result"]) + + +def test_an_error_detail_is_redacted_then_clipped(exporter: InMemorySpanExporter) -> None: + exporter.clear() + transcript = Transcript() + transcript.add_user_message("go") + with pytest.raises(ProviderUnavailable): + run_conversation( + transcript, DownClient(), REGISTRY, ToyDeps(), + observer=OTelObserver(provider="fake", model="m", capture_content=True, redact=scrub), + ) + root = next(s for s in exporter.get_finished_spans() if s.name.startswith("invoke_agent")) + assert root.status.description is not None + assert "sk-real-key" not in root.status.description + [event] = root.events + assert event.name == "exception" + assert event.attributes is not None + assert "sk-real-key" not in str(dict(event.attributes)) # and no stack trace + + +def test_a_redactor_that_raises_drops_the_attribute(exporter: InMemorySpanExporter) -> None: + def broken(text: str) -> str: + raise RuntimeError("boom") + + drive(exporter, resp(echo_call("API_KEY = 'sk-real-key'"), stop=StopReason.TOOL_USE), + resp(TextPart(text="done")), capture=True, redact=broken) + tool = next(s for s in exporter.get_finished_spans() if s.name == "execute_tool echo") + assert "gen_ai.tool.call.arguments" not in attrs(tool) + assert "gen_ai.tool.call.result" not in attrs(tool) + assert "sk-real-key" not in str(attrs(tool)) + + +# --- a made-up tool name is not exported --- + + +def test_an_unregistered_tool_name_stays_out_of_spans(exporter: InMemorySpanExporter) -> None: + drive(exporter, + resp(ToolCallPart(id="c1", tool_name="made_up_xyz", arguments={}), + stop=StopReason.TOOL_USE), + resp(TextPart(text="done"))) + spans = exporter.get_finished_spans() + assert "made_up_xyz" not in str([(s.name, dict(attrs(s))) for s in spans]) + assert "execute_tool " in [s.name for s in spans]