diff --git a/AGENTS.md b/AGENTS.md index 605bd74..4644750 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,24 +25,30 @@ go test -tags e2e -run 'TestE2E' -timeout 15m -v . # LIVE provider e2e (see be |---|---| | `llm.go` | `SDK` entrypoint: `New`, options (`WithProvider`, `FromEnv`, …), `Chat`, `Provider` (model cache, shared learn-once state) | | `message.go` | Canonical types: `ChatRequest`, `Message`, `ChatResult`, `Delta`, `Usage`, `ToolDef` | +| `content.go` | Content-part boundary validation (text, image, image URL, document; MIME case-insensitive, canonical lower-case on the wire), aggregate inline cap, attachment helpers | | `chat.go` | `providerClient`: retry orchestration (buffered + streaming), error classification, learn-once consumption, SSE pump wiring, `httpError` parsing | | `openai.go` / `gemini.go` / `anthropic.go` | Per-format request builders, response/stream mappers, model listing | | `responses.go` | OpenAI Responses API (`/v1/responses`) for GPT-5.6+ tools+reasoning | | `tts.go` | Text-to-speech: `Speak`/`SpeakRequest`/`SpeakResult` — OpenAI-compat `/audio/speech` + Gemini AUDIO modality, no transcoding | -| `controls.go` | Request controls: `ToolChoice`, `ResponseFormat` (incl. Anthropic forced-tool JSON emulation + `foldAnthropicJSON`), `ParallelToolCalls`, per-format mapping and boundary validation | -| `embed.go` | Embeddings: `Embed`/`EmbedRequest`/`EmbedResult` — OpenAI-compat `/embeddings` + Gemini `batchEmbedContents` | +| `controls.go` | Request controls: `ToolChoice`, `ResponseFormat` (Anthropic JSON is native `output_config.format`; the forced-tool emulation + `foldAnthropicJSON` run only under `Quirks.AnthropicToolJSON`, off for the built-in provider), `ParallelToolCalls`, per-format mapping and boundary validation | +| `observe.go` | `AttemptEvent` + `WithAttemptObserver`: per-attempt record filled by `doObserved` in the HTTP helpers and emitted from the retry ladders (`withRetryN`, `callStream`) with the delay about to be waited; key-masked errors; panics recovered | +| `meta.go` | `ResponseMeta`/`RateLimit` and `parseResponseMeta` (request-id + rate-limit headers from the successful attempt); per-request `Headers` validation (reserved auth/framing names rejected, no values in errors) and ctx carriage | +| `tools.go` | `RunTools`: buffered tool-call loop (`ToolHandler`, `ToolLoopOptions`, `ErrMaxToolTurns`); handler errors/panics/unknown tools become `IsError` results, `Parallel` keeps call order | +| `count.go` | `ChatClient.CountTokens`: Anthropic `/v1/messages/count_tokens` + Gemini `:countTokens` through the shared ladder (op `count_tokens`); OpenAI-format/Responses -> `UnsupportedError` (`ErrUnsupported`), no I/O, never an estimate | +| `embed.go` | Embeddings: `Embed`/`EmbedRequest`/`EmbedResult` — OpenAI-compat `/embeddings` (requests `encoding_format: base64`, decodes float32 LE, accepts float arrays) + Gemini `batchEmbedContents` | | `stt.go` | Speech-to-text: `Transcribe`/`TranscribeRequest`/`TranscribeResult` — OpenAI-compat `/audio/transcriptions` (multipart), 25MB input cap | | `sse.go` | SSE parser (abort-safe via `done` channel) + idle-watchdog pump | | `retry.go` | `RetryPolicy` + the single buffered ladder `withRetry` (chat, tts, stt, embed); backoff/jitter/`Retry-After`/`retrySleep` (default 8 attempts, cap 30s) | | `provider.go` | Built-in registry, quirks flags, config validation | -| `models.go` | `ListModels` orchestration (retries, caps, cache backing) | +| `models.go` | `ListModels` orchestration (shared `withRetryN` ladder: `WithRetryPolicy` attempts, default 3, Retry-After, request-timeout budget; deep-copied cache, empty lists cached) | | `auth.go`, `transport.go`, `errors.go` | Env resolution, pooled HTTP clients, typed errors | The canonical type system is OpenAI-shaped; `gemini.go`/`anthropic.go` translate both directions. Provider quirks are **explicit registry flags** — never URL sniffing. ## Invariants (breaking any of these is a defect) -1. **Canonical finish reasons** are `stop | length | tool_calls | content_filter | ""`. Unmapped provider stop reasons map to `""` on every format — provider-specific strings never leak. A turn with tool calls finishes `tool_calls` on every format (Gemini's `STOP` is mapped). +1. **Canonical finish reasons** are `stop | length | tool_calls | content_filter | ""`. Unmapped provider stop reasons map to `""` on every format — provider-specific strings never leak. A tool-call turn that ends normally (provider stop/`STOP`/end, or the format's tool signal) finishes `tool_calls` on every format (Gemini's `STOP` and OpenAI-format `stop` are mapped); a `length`, `content_filter` or unmapped reason keeps that value (or `""`) even with calls present, since the calls may be incomplete. + Refusal text surfaces in `ChatResult.Refusal` / `DeltaRefusal` (never in `Content`); a refusal on `stop`/unmapped finishes `content_filter`. 2. **API keys never appear** in error text, `String()`, or any typed error. 3. **Streaming**: retries only before the first emitted delta; a failure after partial output returns the partial `*ChatResult` + wrapped error and is **never retried**; a premature close (no completion signal — on every format, Gemini included; an unmapped finish reason still counts as one) is an error, never a silent empty or partial success; the parser goroutine is always released (abort-safe `done` protocol). A successful non-SSE response is parsed directly with the usual body cap and format mapper. It learns buffered mode for future requests, but never discards the current generation or makes a replacement request; parse/read errors remain terminal for that response. @@ -53,6 +59,8 @@ The canonical type system is OpenAI-shaped; `gemini.go`/`anthropic.go` translate 8. **`Delta.ToolIndex` is the call's position in `ChatResult.ToolCalls`** on every format — never a provider content-block or output-item index. 9. **`Usage` has one meaning everywhere**: `PromptTokens` uncached-only, cache reads/writes in their fields (a DeepSeek miss is uncached input, not a write), `CompletionTokens` includes reasoning, `CachedTokens` is a diagnostic subset never summed. 10. **The request timeout is a whole-call budget**, retries included, on every entry point (buffered, streaming, speech, transcription, embeddings). +11. **Meta fields are provider-reported or zero, never guessed**; headers come from the successful attempt only. Attempt-observer panics are recovered and discarded; events never contain the API key. +12. **Anthropic JSON mode is native `output_config.format`** unless the explicit registry quirk `Quirks.AnthropicToolJSON` is on (off for the built-in provider, set via `WithQuirks`); the quirk restores the pre-M4 forced-tool emulation byte-for-byte, including its ConfigErrors (also under any thinking mode). Never infer it from URL or model name. ## Testing conventions diff --git a/README.md b/README.md index 83dc01c..7baa955 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ Multi-provider Go SDK for LLM inference endpoints — **OpenAI, Google Gemini, D ## Install ```bash -go get github.com/BackendStack21/go-llm-sdk@v1.0.0 +go get github.com/BackendStack21/go-llm-sdk@v1.1.0 ``` Requires Go 1.25+. No dependencies beyond the standard library. @@ -72,6 +72,7 @@ res, err = chat.CallStream(ctx, req, func(d llm.Delta) error { switch d.Kind { case llm.DeltaReasoning: // thinking fragment case llm.DeltaContent: // text fragment + case llm.DeltaRefusal: // refusal-text fragment (OpenAI, Responses; Anthropic: one delta with the explanation) case llm.DeltaToolArgs: // tool-call argument fragment (d.ToolID, d.ToolName; d.ToolIndex = position in res.ToolCalls) } return nil // or an error to abort — partial result comes back with *StreamAbortedError @@ -110,12 +111,16 @@ llm.WithProvider("openrouter", llm.WithFormat(llm.FormatOpenAI), llm.WithProvider("anthropic", llm.WithHeaders(map[string]string{"anthropic-beta": "interleaved-thinking-2025-05-14"})) +// Anthropic-format gateway without output_config: keep the forced-tool JSON emulation. +llm.WithProvider("gw", llm.WithFormat(llm.FormatAnthropic), llm.WithBaseURL(gwURL), llm.WithAPIKey(key), + llm.WithQuirks(llm.Quirks{AnthropicToolJSON: true})) + // api-key gateways (e.g. Azure OpenAI): drop the Bearer header, send api-key. llm.WithProvider("azure", llm.WithFormat(llm.FormatOpenAI), llm.WithBaseURL(azureURL), llm.WithAPIKey(key), llm.WithHeaders(map[string]string{"Authorization": "", "api-key": key})) ``` -Redirects to another host are never followed (the redirect response surfaces as an `*APIError`): Go strips only `Authorization` across hosts, so `x-api-key`, `x-goog-api-key` and custom headers would otherwise leak to the target. `Provider` and `ProviderConfig` redact the key and header values under every `fmt` verb (`%v`, `%+v`, `%#v`). +Redirects to another host, and https-to-http scheme downgrades on the same host, are never followed (the redirect response surfaces as an `*APIError`): Go strips only `Authorization` across hosts, so `x-api-key`, `x-goog-api-key` and custom headers would otherwise leak to the target. `Provider` and `ProviderConfig` redact the key and header values under every `fmt` verb (`%v`, `%+v`, `%#v`). For request/response observability (logging, metrics, tracing), wrap the transport: `llm.WithTransport(myRoundTripper)` sees every HTTP exchange the SDK makes. @@ -133,6 +138,15 @@ and normalized to `image/jpeg` on the wire). Per-image (`MaxImageBytes`, 10 MiB) and per-request aggregate (`MaxRequestImageBytes`, 32 MiB) caps are enforced before any network I/O. All validation failures are typed `*ConfigError`. +Empty text never reaches Anthropic or Gemini as an empty block: empty system +text (`SystemBlock`, in-band system messages) is skipped; a user message with no +`Content` and no `Parts` is a `*ConfigError`; an assistant turn with no text, +thinking or tool calls is dropped on Anthropic and the surrounding user turns are +merged so roles still alternate. Gemini tool `Parameters` are sent as +`parametersJsonSchema` (full JSON Schema: `$ref`/`$defs`, `const`, type arrays). +Gemini model ids may carry a `models/` prefix on every endpoint (chat, stream, +embeddings, speech); ids are path-escaped. + ```go req := &llm.ChatRequest{Messages: []llm.Message{{ Role: llm.RoleUser, @@ -143,6 +157,30 @@ req := &llm.ChatRequest{Messages: []llm.Message{{ }}} ``` +### Documents and image URLs + +Besides `TextPart` and `ImagePart`, user and tool messages accept: + +- `DocumentPart(mime, data, name)`: an inline document, `application/pdf` or + `text/plain` (lower-case, no parameters). Up to `MaxDocumentBytes` (20 MiB) + each; document bytes count toward the same `MaxRequestImageBytes` aggregate + cap as inline images. +- `ImageURLPart(url)` / `ImageURLPartMIME(url, mime)` / `part.WithMIME(mime)`: + a remote `http`/`https` image the provider fetches (the SDK never fetches it; + `data:`, `file:` and every other scheme are rejected). The optional MIME must + be an image type. + +| Format | Document | Image URL | +|---|---|---| +| OpenAI chat | `file` part, PDF only | `image_url` | +| Responses | `input_file`, PDF only | `input_image` | +| Anthropic | `document` (PDF base64, text/plain text source) | `image` with `url` source (MIME hint ignored) | +| Gemini | `inlineData` | `fileData`; a MIME type is required (`*ConfigError` otherwise) | + +In tool results, documents are accepted on Anthropic and Gemini and are a +`*ConfigError` on OpenAI chat and the Responses API; image URLs ride the +post-tool user message (OpenAI chat) or `function_call_output` (Responses). + ```go type ChatRequest struct { Model string // optional; ChatClient's model wins when both set @@ -159,7 +197,7 @@ type ChatRequest struct { ParallelToolCalls *bool // with Tools: allow/forbid several calls per turn ResponseFormat *ResponseFormat // JSON object / JSON Schema structured output Seed *int // OpenAI chat completions, Gemini - Extra map[string]any // provider-specific top-level body fields (override the SDK's) + Extra map[string]any // provider-specific top-level body fields (override the SDK's; Gemini generationConfig merges key by key) } type ChatResult struct { @@ -169,11 +207,17 @@ type ChatResult struct { ThinkingBlocks []ThinkingBlock // every signed reasoning segment, in order (Anthropic, Responses) ToolCalls []ToolCall // {ID, Name, Arguments, Signature (Gemini thoughtSignature)} FinishReason string // stop | length | tool_calls | content_filter | "" + Refusal string // provider refusal text ("" when none); Content is never altered + Meta llm.ResponseMeta // provider-reported id/model/request id/rate-limit headroom (zero when unreported) Usage Usage // see "Usage" below } ``` -Finish reasons are canonical: anything a provider reports outside that vocabulary maps to `""` (unknown) rather than leaking provider-specific strings. A turn that requested tools finishes as `tool_calls` on every format (Gemini reports `STOP` for it; the SDK maps it). +`Refusal` carries the provider's refusal text (OpenAI `message.refusal`, Responses `refusal` parts, Anthropic `stop_details.explanation`; Gemini has none) and is streamed as `DeltaRefusal`. A refusal on a `stop`/unmapped finish reports `content_filter`; `length`/`tool_calls` are kept. `AssistantMessage()` does not replay it. + +`Meta` (`ResponseMeta`, also on `EmbedResult`, `SpeakResult`, `TranscribeResult`) is provider-reported or zero, never guessed: `ID` (OpenAI/Responses `id`, Anthropic `id`, Gemini `responseId`), `Model` (`model`, Gemini `modelVersion`), `RequestID` (first of `x-request-id`, `request-id`, `x-goog-request-id`, `x-amzn-requestid`) and `RateLimit` (`Reported`, `RemainingRequests`, `RemainingTokens`, `ResetRequests`, `ResetTokens`, from `x-ratelimit-*` with `anthropic-ratelimit-*` as per-field fallback; unparseable values stay zero). Headers come from the successful attempt only, buffered and streaming. ID/Model are empty on Embed/Speak/Transcribe. + +Finish reasons are canonical: anything a provider reports outside that vocabulary maps to `""` (unknown) rather than leaking provider-specific strings. A tool-call turn that ends normally (provider `stop`/`STOP`/`end_turn`, or the format's tool signal) finishes as `tool_calls` on every format (Gemini and OpenAI-format providers with a named `tool_choice` report `STOP`/`stop` for it; the SDK maps it). A truncation (`length`), filter (`content_filter`) or unmapped reason keeps that value (or `""`) even when calls are present, because the calls may be incomplete. A tool call with no arguments carries `{}` in `Arguments` on every format, buffered and streaming. Signatures (`ThinkingSignature`, `ThinkingBlocks`, `ToolCall.Signature`) are provider-specific: replay them only to the provider that produced them — never carry a conversation's signed turns across providers. @@ -207,19 +251,27 @@ req := &llm.ChatRequest{ |---|---|---|---| | `ToolChoice` auto / none / required / tool | `tool_choice` | `tool_choice` auto / none / any / tool | `toolConfig.functionCallingConfig` AUTO / NONE / ANY (+ `allowedFunctionNames`) | | `ParallelToolCalls` | `parallel_tool_calls` (only with tools) | `disable_parallel_tool_use` | — (ignored) | -| `ResponseFormat` json_object / json_schema | `response_format` / `text.format` | forced synthetic tool whose input is folded back into `Content` (buffered and streamed as `DeltaContent`) | `responseMimeType` + `responseJsonSchema` | +| `ResponseFormat` json_object / json_schema | `response_format` / `text.format` | native `output_config.format` (`json_schema`; `json_object` sends `{"type":"object"}`), merged with any thinking `effort` into one `output_config`; the reply text is returned as-is in `Content` | `responseMimeType` + `responseJsonSchema` | | `Seed` | `seed` (chat completions only) | — (ignored) | `generationConfig.seed` | | `Extra` | merged into the body | merged into the body | merged into the body | -Caveats: DeepSeek accepts `json_object` only (`Quirks.NoJSONSchema`; `json_schema` fails fast with a `ConfigError` — set the flag via `WithQuirks` for other gateways with the same limit). On Anthropic, JSON mode forces its own tool, so it cannot be combined with `Tools` and needs a top-level `"type": "object"` schema. Gemini 2.x models reject JSON mode together with function declarations; Gemini 3 accepts it. `Extra` is a shallow, top-level override: an object value replaces the SDK's whole object (e.g. `generationConfig`), and keys go to whichever endpoint the request lands on (including `/responses` for diverted OpenAI calls). It may not replace what the SDK validates (`model`, `messages`/`contents`/`input`, `system`/`systemInstruction`/`instructions`, `tools`) or `stream`. Gemini seeds must fit in 32 bits. +Caveats: DeepSeek accepts `json_object` only (`Quirks.NoJSONSchema`; `json_schema` fails fast with a `ConfigError` — set the flag via `WithQuirks` for other gateways with the same limit). On Anthropic, JSON mode is native and combines with `Tools`, `ToolChoice` and thinking; the API decides which schemas it accepts. Gateways lacking `output_config` can set `Quirks.AnthropicToolJSON`, which keeps the older forced-synthetic-tool emulation folded back into `Content` (it cannot combine with `Tools`, `ToolChoice` or thinking, and needs an object schema). Gemini 2.x models reject JSON mode together with function declarations; Gemini 3 accepts it. `Extra` is a shallow, top-level override: an object value replaces the SDK's whole object, except Gemini `generationConfig`, whose keys are set one by one over the SDK's (so `Extra["generationConfig"] = {"candidateCount": 2}` keeps `maxOutputTokens`, `topK` and the rest; an Extra key wins; nested objects like `thinkingConfig` are replaced whole). Keys go to whichever endpoint the request lands on (including `/responses` for diverted OpenAI calls). It may not replace what the SDK validates (`model`, `messages`/`contents`/`input`, `system`/`systemInstruction`/`instructions`, `tools`) or `stream`. Gemini seeds must fit in 32 bits. `TopK` (0 = provider default, negative is a `ConfigError`) maps to Anthropic `top_k` (omitted when thinking is on) and Gemini `generationConfig.topK`; OpenAI chat and Responses ignore it. `EmbedRequest.TaskType` (e.g. `RETRIEVAL_QUERY`) is sent as Gemini `taskType` on every batch item and ignored by OpenAI-format providers. + +Controls are validated before any network I/O (`*ConfigError`): `ToolChoice` needs `Tools` and a known tool name; `json_schema` needs a JSON object schema; names match `[A-Za-z0-9_-]{1,64}`; `Extra` may not set the reserved keys above. Anthropic rejects forced tool use under budget-based extended thinking, so `ToolChoice` required/tool fails fast when `ThinkingBudget` > 0 (adaptive thinking passes a forced `ToolChoice` through; native JSON mode adds no tool, so it is unaffected). + +### Per-request Headers -Controls are validated before any network I/O (`*ConfigError`): `ToolChoice` needs `Tools` and a known tool name; `json_schema` needs a JSON object schema; names match `[A-Za-z0-9_-]{1,64}`; `Extra` may not set the reserved keys above. Anthropic rejects forced tool use under extended thinking, so `ToolChoice` required/tool and JSON `ResponseFormat` fail fast with `Thinking` on, and JSON mode cannot be combined with `ToolChoice` (it forces its own tool). +`ChatRequest`, `EmbedRequest`, `SpeakRequest` and `TranscribeRequest` carry `Headers map[string]string`, merged last (after auth and the provider's `WithHeaders`) for that call only, on every retry attempt. An empty value removes a non-reserved header for the call. Reserved names (case-insensitive) are always rejected with a `*ConfigError`: `Authorization`, `Proxy-Authorization`, `X-Api-Key`, `X-Goog-Api-Key`, `Host`, `Content-Length`, `Content-Type`, `Transfer-Encoding`. `anthropic-version` and `anthropic-beta` are not reserved. Names must be RFC 7230 tokens and names are case-insensitive, so two keys that differ only in case are rejected; values may contain no ASCII control byte except HTAB (CR, LF, NUL and DEL are rejected); violations fail before any network I/O and the error never contains the value. `WithHeaders` is unchanged and not validated. + +## Observability + +`llm.WithAttemptObserver(func(llm.AttemptEvent))` reports every HTTP attempt on chat (buffered and streaming), embeddings, speech, transcription and model listing, retries and learn-once immediate retries included. `AttemptEvent` carries `Provider`, `Operation` (`chat|stream|embed|speak|transcribe|models`), `Model` (empty for models), `Attempt` (1-based per call), `Status` (0 on transport error), `Err` (nil on success; the API key is masked), `Latency` (request start to response headers), `RetryDelay` (the wait before the next attempt, 0 on the last) and `RequestID`. A stream that fails after headers reports that failure on its attempt's event. The observer runs on the request goroutine: keep it fast and never call back into the SDK. A panic in it is recovered and discarded. Per-response data lives in `ChatResult.Meta` (`ResponseMeta` with `RateLimit`), see above. ## Streaming semantics `CallStream` enforces four guarantees, each covered by regression tests: -1. **Idle watchdog** — a stream silent longer than the idle timeout (120s default; per SDK via `WithStreamIdleTimeout`, process-wide default via the race-safe `SetStreamIdleTimeout`; positive values only) fails with `ErrIdleTimeout`. Keepalive comments reset it. +1. **Idle watchdog** — a stream silent longer than the idle timeout (120s default; per SDK via `WithStreamIdleTimeout`, process-wide default via the race-safe `SetStreamIdleTimeout`; positive values only) fails with `ErrIdleTimeout`. Keepalive comments reset it; time spent in your `onDelta` callback never counts, only provider silence. A provider `{"error":...}` chunk (OpenAI-compat, Gemini, plus Anthropic/Responses error events) fails the stream with the provider message, and a bare `[DONE]` with nothing emitted is a premature-close error. The parser tolerates a leading UTF-8 BOM and `\n`, `\r\n` or bare `\r` line endings. 2. **Hard wall-clock deadline** — the whole stream is bounded by the per-request timeout (`WithRequestTimeout`, default 120s; per-client via `SetRequestTimeout`). 3. **Retries never duplicate output** — retries happen only before the first emitted delta. A failure after partial output returns the partial `*ChatResult` plus a wrapped error and is never retried. 4. **No silent truncation** — a provider that closes the stream before any completion signal (`[DONE]`, a `finish_reason`, `message_stop`, a Gemini `finishReason`, `response.completed`) yields an error: retryable before the first delta, the partial result plus a wrapped error after it. Gemini is no longer exempt. An unmapped provider finish reason still counts as completion. @@ -248,18 +300,30 @@ for { `IsError` marks a failed tool execution (Anthropic `is_error`, Gemini `{"error": …}`; OpenAI formats have no flag, so describe the failure in `Content`). Tool results may carry image `Parts`: Anthropic and the Responses API take them natively, Gemini sends them as `inlineData` in the same turn, and Chat Completions (whose tool messages are text-only) receives them in a user message right after the tool results. +`RunTools` runs this loop for you (buffered `Call` only): + +```go +res, transcript, err := llm.RunTools(ctx, chat, req, map[string]llm.ToolHandler{ + "get_weather": func(ctx context.Context, c llm.ToolCall) (string, error) { return lookup(c.Arguments) }, +}, llm.ToolLoopOptions{MaxTurns: 8, Parallel: true}) +``` + +`MaxTurns` bounds chat calls (default 8); `Parallel` runs one turn's handlers concurrently while results are appended in call order; `OnTurn` observes each turn. A handler error, panic or unknown tool becomes an `IsError` result and the loop continues. Assistant turns are replayed verbatim (thinking blocks, signatures). If the model still wants tools after `MaxTurns`, you get the last result, the transcript without the unanswered assistant turn, and `llm.ErrMaxToolTurns`. `req.Messages` is never mutated; the returned transcript is the full conversation. + +The context is checked before every chat call and before each handler. On cancellation `RunTools` stops starting handlers (those not yet started record an `IsError` result carrying the context error) and returns the last completed `*ChatResult`, the transcript so far, and the context error (`errors.Is(err, context.Canceled)` / `context.DeadlineExceeded`). + On Gemini, a tool result's `ToolName` may be omitted — the SDK recovers the function name from the assistant `ToolCall` it answers, and errors loudly if it cannot. ## Extended thinking - **OpenAI GPT-5.6+** — function tools plus reasoning cannot ride Chat Completions (`reasoning_effort` 400s). Those calls go to `POST /v1/responses` with `reasoning.effort` and `reasoning.summary=auto`; summaries land in `ReasoningContent`, every reasoning item lands in `ThinkingBlocks` (encrypted content as `Signature`) and is replayed in order; the legacy `ThinkingSignature` carries the last one. `Thinking: disabled` stays on Chat Completions with `reasoning_effort: none`. Other OpenAI models keep `reasoning_effort` on Chat Completions. -- **Anthropic** — every `thinking` and `redacted_thinking` block is parsed in both buffered and streaming modes into `ChatResult.ThinkingBlocks`, each with its own signature (redacted blocks keep their opaque `data`). For tool loops, replay them via `Message.ThinkingBlocks` (or `res.AssistantMessage()`): the SDK re-serializes them first, in order, as Anthropic's API requires. The legacy single pair (`ReasoningContent` + `ThinkingSignature`) still works for one-block turns. Unsigned thinking replay is rejected locally with `ConfigError`. With thinking on, `max_tokens` always exceeds `budget_tokens` (unset `MaxTokens` → budget + 8192; a preset larger than half an explicit `MaxTokens` is clamped to half of it — never below 1024 — so the answer keeps room; an explicit `ThinkingBudget` that cannot fit is a `ConfigError`), and `temperature` / `top_p` are omitted (extended thinking rejects them). +- **Anthropic** — every `thinking` and `redacted_thinking` block is parsed in both buffered and streaming modes into `ChatResult.ThinkingBlocks`, each with its own signature (redacted blocks keep their opaque `data`). For tool loops, replay them via `Message.ThinkingBlocks` (or `res.AssistantMessage()`): the SDK re-serializes them first, in order, as Anthropic's API requires. The legacy single pair (`ReasoningContent` + `ThinkingSignature`) still works for one-block turns. Unsigned thinking replay is rejected locally with `ConfigError`. Thinking presets use **adaptive thinking**: `thinking: {type: adaptive}` plus `output_config.effort` set to `low`/`medium`/`high`/`max` (`enabled` sends adaptive with no effort), never `budget_tokens`. Unset `MaxTokens` defaults to 16384 under adaptive thinking (an explicit value is sent as-is). A positive `ThinkingBudget` is the escape hatch for models that need the legacy form (e.g. Haiku 4.5 and older): `thinking: {type: enabled, budget_tokens: N}` (N >= 1024), `max_tokens` above N (unset → N + 8192; an impossible budget is a `ConfigError`), and forced tool use is rejected. With thinking on, `temperature` / `top_p` are omitted. `Extra["output_config"]` replaces the SDK-generated object, effort and JSON format included (shallow merge). - **DeepSeek** — reasoning streams as `DeltaReasoning` fragments and lands in `ReasoningContent`. Assistant-turn replay echoes it as `reasoning_content`, which DeepSeek requires for tool loops. On a chat-completions request that carries tools, the key is echoed for **every** assistant turn, even where the provider returned no reasoning for that turn (empty value instead of a dropped key) — including turns that made no tool call themselves: DeepSeek documents that a tool-bearing request whose assistant turns omit `reasoning_content` returns 400 for *every* later request in the loop, so one elided turn would otherwise poison the rest of the session. Whether DeepSeek accepts a present-but-empty value is measured against the live API by `TestE2EDeepSeekEmptyReasoningEcho` (tag-gated), not assumed by the suite. - **zai / GLM** — same reasoning field shape and `DeltaReasoning` streaming; GLM maps thinking `medium` → `reasoning_effort` `high` (no medium level) and `max` → `max`. GLM does **not** get the empty echo: no z.ai documentation confirms the same replay requirement, so `Quirks.EchoReasoningWithTools` is off and `zai` bodies are byte-identical to the pre-fix shape. A custom provider registered under its own id inherits **no** quirks — set `WithQuirks` explicitly if it needs the echo. - **Gemini** — a prompt blocked by safety (`promptFeedback.blockReason`, no candidates) completes with `content_filter` instead of erroring. `thought: true` parts map to reasoning deltas; `thinkingConfig` is derived from `Thinking` / `ThinkingBudget`. `thoughtSignature`s are captured per function call (`ToolCall.Signature`) and on text parts (`ThinkingSignature`) and replayed on the same parts — Gemini 3 rejects a function-call turn whose signature is missing. - **OpenAI-format gateways (OpenRouter, LiteLLM, vLLM)** — responses are accepted in every documented reasoning shape: `reasoning_content` (LiteLLM standardized), the `reasoning` string alias (OpenRouter), and the typed `reasoning_details` array (OpenRouter; `reasoning.text`/`reasoning.summary` fold in order, `reasoning.encrypted` entries are skipped). Precedence: `reasoning_content`, then `reasoning`, then the `reasoning_details` fold. Gateways that require an explicit opt-in get it via `Quirks.IncludeReasoning`, which sends the OpenRouter-documented legacy `include_reasoning: true` (equivalent to the canonical `reasoning: {}`) next to `reasoning_effort`. The flag is off by default — strict OpenAI-format endpoints reject unknown parameters. -`ThinkingBudget`, when positive, overrides the selected non-disabled thinking preset (Anthropic enforces its 1024-token minimum). Canonical `max` selects the highest portable preset: OpenAI `high`, Gemini 24576, Anthropic 16384; GLM retains its native `max`. +`ThinkingBudget`, when positive, overrides the selected non-disabled thinking preset (Anthropic switches to the legacy `budget_tokens` form, 1024-token minimum). Canonical `max` selects the highest portable preset: OpenAI `high`, Gemini 24576, Anthropic effort `max`; GLM retains its native `max`. ## Learn-once fallbacks @@ -277,7 +341,7 @@ Every engagement is observable: `WithLearnObserver(func(LearnEvent))` registers ## Retry policy -8 attempts, exponential backoff capped at 30s with ±20% jitter (both configurable per SDK with `WithRetryPolicy(llm.RetryPolicy{MaxAttempts, MaxBackoff})`; `MaxAttempts: 1` disables retries), `Retry-After` (seconds or HTTP-date) honored and capped at 120s, context cancellation honored between and during attempts. Retryable statuses include 408/429/5xx plus Cloudflare 520–524 and Anthropic 529. Persistent 429s surface as `*llm.RateLimitError{Attempts, RetryAfter}` — including when the retry sleep is cut short by a deadline, so the caller never loses the retry signal. A 429 whose body is billing exhaustion (`insufficient_quota`, `exceeded your current quota`, `insufficient balance`, `no resource package`) is not retryable and fails on the first attempt. `RateLimitError` unwraps to `*APIError` for `errors.As` access to `Status`/`Retryable`. +8 attempts, exponential backoff capped at 30s with ±20% jitter (both configurable per SDK with `WithRetryPolicy(llm.RetryPolicy{MaxAttempts, MaxBackoff})`; `MaxAttempts: 1` disables retries), `Retry-After` (seconds or HTTP-date) honored and capped at 120s, context cancellation honored between and during attempts. Retryable statuses include 408/429/5xx plus Cloudflare 520–524 and Anthropic 529. Persistent 429s surface as `*llm.RateLimitError{Attempts, RetryAfter}` — including when the retry sleep is cut short by a deadline, so the caller never loses the retry signal. When a backoff or `Retry-After` wait would outlive the context deadline, the SDK does not sleep: it returns the last `*APIError` / transport error immediately, also matching `context.DeadlineExceeded` under `errors.Is`. Oversized response bodies (50MB; 8MB for model lists) are terminal and never retried. A 429 whose body is billing exhaustion (`insufficient_quota`, `exceeded your current quota`, `insufficient balance`, `no resource package`) is not retryable and fails on the first attempt. `RateLimitError` unwraps to `*APIError` for `errors.As` access to `Status`/`Retryable`. ## Timeouts & cancellation @@ -341,11 +405,20 @@ res, err := sdk.Transcribe(ctx, "openai", "whisper-1", llm.TranscribeRequest{ // res.Language, res.DurationSec — provider-reported, zero when omitted ``` -Requests carry the same retry ladder and error taxonomy as chat. Oversized audio (>25MB), empty `Audio`/`Model`, and control characters in `Filename`/`MIMEType` (header injection) fail fast with a `ConfigError` before any network I/O. A 2xx body that is not JSON surfaces as a typed `*APIError`. Fields the provider does not report stay zero — the SDK never guesses. +Requests carry the same retry ladder and error taxonomy as chat. Oversized audio (>25MB), empty `Audio`/`Model`, and control characters in `Filename`/`MIMEType` (header injection) fail fast with a `ConfigError` before any network I/O. `Format` `"text"`, `"srt"` and `"vtt"` are not JSON: the raw response body is returned as `Text` (other fields zero). For the JSON formats (default, `json`, `verbose_json`) a 2xx body that is not JSON surfaces as a typed `*APIError`. Fields the provider does not report stay zero — the SDK never guesses. + +## Counting tokens + +`ChatClient.CountTokens(ctx, req)` returns the input token count of a request without generating: Anthropic `POST /v1/messages/count_tokens` and Gemini `:countTokens`. The request is validated like `Call` (`Extra` and sampling parameters are ignored). OpenAI-format providers, including the Responses API, have no such endpoint: the call returns `*UnsupportedError` (`errors.Is(err, llm.ErrUnsupported)`) without any network I/O, and the SDK never estimates. Same retry ladder, whole-call timeout, per-request `Headers` and attempt observer (operation `count_tokens`) as chat; a 2xx response without a valid count is a terminal `*APIError`. + +```go +n, err := chat.CountTokens(ctx, &llm.ChatRequest{Messages: msgs, Tools: tools}) +if errors.Is(err, llm.ErrUnsupported) { /* fall back to your own estimate */ } +``` ## Embeddings -`Embed` turns texts into vectors via OpenAI-compatible `POST {base}/embeddings` or Gemini `batchEmbedContents`; Anthropic (no endpoint) returns a `ConfigError`. +`Embed` turns texts into vectors via OpenAI-compatible `POST {base}/embeddings` or Gemini `batchEmbedContents`; Anthropic (no endpoint) returns a `ConfigError`. OpenAI-compatible requests ask for `encoding_format: base64` (little-endian float32, decoded by the SDK) so a 2048-input batch fits the response size cap; providers that ignore it and return float arrays are still accepted. ```go res, err := sdk.Embed(ctx, "openai", "text-embedding-3-small", llm.EmbedRequest{ @@ -363,7 +436,7 @@ Vectors come back in input order (the SDK sorts by the provider's `index` — or ## Model discovery -`ListModels` hits each provider's models endpoint (Anthropic paginates with `after_id`, Gemini with `pageToken`), caches per SDK for 5 minutes (`WithModelCacheTTL(0)` disables, `ForceRefresh()` bypasses), and retries transient failures 3× (a malformed body is not retried). Concurrent cache misses share one upstream fetch. A listing longer than 100 pages returns `ErrModelListTruncated` — never a silently truncated list. Fields the provider does not report stay zero — the SDK never guesses. +`ListModels` hits each provider's models endpoint (Anthropic paginates with `after_id`, Gemini with `pageToken`), caches per SDK for 5 minutes (`WithModelCacheTTL(0)` disables, `ForceRefresh()` bypasses), and retries transient failures through the shared ladder: 3 attempts unless `WithRetryPolicy` sets `MaxAttempts`, honoring `Retry-After`, under a whole-call budget of 30s unless `WithRequestTimeout` is set (a malformed body is not retried). OpenAI-format listings accept `{data:[…]}`, `{models:[…]}` or a bare array (Together AI). Anthropic `has_more` without a `last_id` is an error, never a partial list. Results are deep-copied on every read (mutating `Capabilities` cannot corrupt the cache) and an empty successful listing is cached like any other. Concurrent cache misses share one upstream fetch. A listing longer than 100 pages returns `ErrModelListTruncated` — never a silently truncated list. Fields the provider does not report stay zero — the SDK never guesses. ## Testing @@ -387,6 +460,24 @@ Coverage sits at **99.0%** of statements (unit suite, no e2e), including the str See [AGENTS.md](AGENTS.md) for the architecture map, invariants, testing conventions, and the odek migration path. +## Upgrading to v1.1 (from v1.0) + +v1.1 adds API only (`Refusal`, `ResponseMeta`, per-request `Headers`, `WithAttemptObserver`, `DocumentPart`, `ImageURLPart`, `RunTools`, `CountTokens`, `TopK`, `EmbedRequest.TaskType`, `Quirks.AnthropicToolJSON`); nothing exported is removed or renamed. Positional (unkeyed) literals of structs that gained fields need field names. Behaviors that changed on purpose: + +| Area | v1.0 | v1.1 | What to do | +|---|---|---|---| +| Anthropic thinking | Presets sent `budget_tokens` (5000/1024/8192/16384); unset `MaxTokens` → budget + 8192; presets clamped to half of `MaxTokens` | Presets send adaptive thinking + `output_config.effort`; unset `MaxTokens` → 16384; `MaxTokens` as-is; `temperature`/`top_p` omitted; forced `ToolChoice` allowed | Set `ThinkingBudget` > 0 for models needing the legacy `budget_tokens` form (Haiku 4.5 and older) | +| Anthropic JSON mode | Forced synthetic tool folded into `Content`; errors with `Tools`, `ToolChoice`, thinking | Native `output_config.format` (reply text as-is); those combinations are allowed | Gateways lacking `output_config` set `Quirks.AnthropicToolJSON` for the old behaviour (byte-identical, errors included) | +| OpenAI-compatible embeddings | Float arrays requested | `encoding_format:"base64"` requested (float arrays still accepted) | — | +| Streaming | A mid-stream `{"error":…}` chunk or a bare `[DONE]` with no output ended as an empty success | Provider error / premature-close error | None; this surfaces real failures | +| Finish reasons | OpenAI `stop` with tool calls stayed `stop`; unknown Responses `incomplete` reasons leaked as `length` | `tool_calls`; `""` | Loops checking `len(res.ToolCalls) > 0` are unaffected | +| Empty content | Empty text blocks sent (provider 400) | Never sent; an empty Anthropic assistant turn is dropped and adjacent user turns merged | — | +| Gemini tools | `parameters` (OpenAPI subset) | `parametersJsonSchema` (full JSON Schema) | — | +| STT `text`/`srt`/`vtt` | JSON-decoded (failed after billing) | Raw body returned as `Text` | — | +| Redirects | Same-host https-to-http followed | Refused (credential leak), surfaces as `*APIError` | Point `WithBaseURL` at the final scheme and host | +| Oversized bodies / long backoff | Retried up to 8×; a wait past the deadline returned bare `DeadlineExceeded` | Terminal; the last provider error is returned immediately | — | +| `ListModels` | Ignored `WithRetryPolicy`, request timeout, `Retry-After` | Honours all three | — | + ## Upgrading to v1.0 (from v0.7) v1.0 removes or renames nothing exported. Existing code compiles unchanged unless it builds `ToolCall`, `Message`, `ChatRequest`, `ChatResult`, `ProviderConfig` or `Quirks` with positional (unkeyed) struct literals; those structs gained fields, so switch to field names. A few behaviors changed on purpose. Check these before upgrading: @@ -407,7 +498,7 @@ v1.0 removes or renames nothing exported. Existing code compiles unchanged unles ## Status -v1.0.0 — stable. The exported API follows semantic versioning from here: breaking changes only in a new major version. +v1.1.0 — stable. The exported API follows semantic versioning from here: breaking changes only in a new major version. ## License diff --git a/anthropic.go b/anthropic.go index e7c9bfb..15c5813 100644 --- a/anthropic.go +++ b/anthropic.go @@ -4,6 +4,7 @@ import ( "context" "encoding/base64" "encoding/json" + "errors" "fmt" neturl "net/url" "strings" @@ -31,6 +32,7 @@ type anBlock struct { Type string `json:"type"` // "text" | "tool_use" | "tool_result" Text string `json:"text,omitempty"` Source *anImageSource `json:"source,omitempty"` + Title string `json:"title,omitempty"` // document // CacheControl is set on user text blocks when Message.Cache is true. CacheControl *anCacheControl `json:"cache_control,omitempty"` // thinking / redacted_thinking (replayed assistant turns; they go @@ -51,8 +53,9 @@ type anBlock struct { type anImageSource struct { Type string `json:"type"` - MediaType string `json:"media_type"` - Data string `json:"data"` + MediaType string `json:"media_type,omitempty"` + Data string `json:"data,omitempty"` + URL string `json:"url,omitempty"` } type anMessage struct { @@ -68,22 +71,34 @@ type anTool struct { } type anThinking struct { - Type string `json:"type"` // "enabled" + Type string `json:"type"` // "enabled" (budget) or "adaptive" BudgetTokens int `json:"budget_tokens,omitempty"` } +type anOutputFormat struct { + Type string `json:"type"` + Schema json.RawMessage `json:"schema"` +} + +type anOutputConfig struct { + Effort string `json:"effort,omitempty"` + Format *anOutputFormat `json:"format,omitempty"` +} + type anRequest struct { - Model string `json:"model"` - MaxTokens int `json:"max_tokens"` - Messages []anMessage `json:"messages"` - System []anSysBlock `json:"system,omitempty"` - Tools []anTool `json:"tools,omitempty"` - Temperature *float64 `json:"temperature,omitempty"` - TopP *float64 `json:"top_p,omitempty"` - Stop []string `json:"stop_sequences,omitempty"` - Stream bool `json:"stream,omitempty"` - Thinking *anThinking `json:"thinking,omitempty"` - ToolChoice *anToolChoice `json:"tool_choice,omitempty"` + Model string `json:"model"` + MaxTokens int `json:"max_tokens"` + Messages []anMessage `json:"messages"` + System []anSysBlock `json:"system,omitempty"` + Tools []anTool `json:"tools,omitempty"` + Temperature *float64 `json:"temperature,omitempty"` + TopP *float64 `json:"top_p,omitempty"` + TopK *int `json:"top_k,omitempty"` + Stop []string `json:"stop_sequences,omitempty"` + Stream bool `json:"stream,omitempty"` + Thinking *anThinking `json:"thinking,omitempty"` + ToolChoice *anToolChoice `json:"tool_choice,omitempty"` + OutputConfig *anOutputConfig `json:"output_config,omitempty"` } const anthropicDefaultMaxTokens = 8192 @@ -91,35 +106,36 @@ const anthropicDefaultMaxTokens = 8192 // anthropicMinThinkingBudget is Anthropic's minimum budget_tokens. const anthropicMinThinkingBudget = 1024 -// anthropicThinkingBudget maps canonical thinking levels to budgets. -// Anthropic requires budget_tokens >= 1024. -func anthropicThinkingBudget(level string, explicit int) (int, bool) { - var budget int +// anthropicThinkingOn reports whether a canonical thinking level enables +// thinking ("" and "disabled" do not). +func anthropicThinkingOn(level string) bool { switch level { - case "enabled": - budget = 5000 - case "low": - budget = 1024 - case "medium": - budget = 8192 - case "high", "max": - budget = 16384 - default: // "", "disabled" - return 0, false - } - if explicit > 0 { - budget = maxInt(explicit, anthropicMinThinkingBudget) + case "enabled", "low", "medium", "high", "max": + return true } - return budget, true + return false } +// anthropicAdaptiveMaxTokens is the max_tokens default under adaptive +// thinking, leaving room for thinking plus the visible answer. +const anthropicAdaptiveMaxTokens = 16384 + // anthropicPartBlocks renders ordered content parts as text/image blocks. func anthropicPartBlocks(parts []ContentPart) []anBlock { blocks := make([]anBlock, 0, len(parts)) for _, p := range parts { - if p.Type == ContentPartImage { + switch p.Type { + case ContentPartImage: blocks = append(blocks, anBlock{Type: "image", Source: &anImageSource{Type: "base64", MediaType: wireMIME(p.MIMEType), Data: base64.StdEncoding.EncodeToString(p.Image)}}) - } else { + case ContentPartImageURL: + blocks = append(blocks, anBlock{Type: "image", Source: &anImageSource{Type: "url", URL: p.URL}}) + case ContentPartDocument: + src := &anImageSource{Type: "base64", MediaType: canonMIME(p.MIMEType), Data: base64.StdEncoding.EncodeToString(p.Data)} + if canonMIME(p.MIMEType) == "text/plain" { + src = &anImageSource{Type: "text", MediaType: "text/plain", Data: string(p.Data)} + } + blocks = append(blocks, anBlock{Type: "document", Source: src, Title: p.Name}) + default: blocks = append(blocks, anBlock{Type: "text", Text: p.Text}) } } @@ -160,8 +176,15 @@ func maxInt(a, b int) int { return b } -// buildAnthropicRequest renders the canonical request in Anthropic format. +// buildAnthropicRequest renders the canonical request in Anthropic format +// with the native (default) quirks. func buildAnthropicRequest(req *ChatRequest, model string, stream bool) ([]byte, error) { + return buildAnthropicRequestQ(req, model, stream, Quirks{}) +} + +// buildAnthropicRequestQ renders the request honouring provider quirks +// (AnthropicToolJSON selects the forced-tool JSON emulation). +func buildAnthropicRequestQ(req *ChatRequest, model string, stream bool, q Quirks) ([]byte, error) { out := anRequest{ Model: model, MaxTokens: req.MaxTokens, @@ -169,9 +192,14 @@ func buildAnthropicRequest(req *ChatRequest, model string, stream bool) ([]byte, Stream: stream, } + legacyThinking := false + // System: canonical blocks (with cache markers) + in-band system // messages become the top-level system field. for _, b := range req.System { + if b.Text == "" { // Anthropic rejects empty text blocks + continue + } sb := anSysBlock{Type: "text", Text: b.Text} if b.Cache { sb.CacheControl = &anCacheControl{Type: "ephemeral"} @@ -193,31 +221,35 @@ func buildAnthropicRequest(req *ChatRequest, model string, stream bool) ([]byte, } out.TopP = &p } - if budget, ok := anthropicThinkingBudget(req.Thinking, req.ThinkingBudget); ok { - // Anthropic requires max_tokens > budget_tokens, and the budget - // counts against max_tokens. An unset MaxTokens leaves the usual - // visible-output room on top of the budget; an explicit MaxTokens - // is a hard cap, so a preset level is clamped to half of it (the - // rest stays for the answer) and an explicit ThinkingBudget that - // cannot fit fails fast. - switch { - case out.MaxTokens <= 0: - out.MaxTokens = budget + anthropicDefaultMaxTokens - case budget >= out.MaxTokens && req.ThinkingBudget > 0: - return nil, &ConfigError{Msg: fmt.Sprintf("Anthropic ThinkingBudget %d must be below MaxTokens %d", budget, out.MaxTokens)} - case budget > out.MaxTokens/2 && req.ThinkingBudget <= 0: - // A preset larger than half an explicit cap would starve the - // visible answer: keep half for output (never below Anthropic's - // minimum budget, which must still fit under the cap). - budget = maxInt(out.MaxTokens/2, anthropicMinThinkingBudget) - if budget >= out.MaxTokens { - return nil, &ConfigError{Msg: fmt.Sprintf("Anthropic thinking needs MaxTokens above %d, got %d", anthropicMinThinkingBudget, out.MaxTokens)} + if req.TopK > 0 { + k := req.TopK + out.TopK = &k + } + if anthropicThinkingOn(req.Thinking) { + if req.ThinkingBudget > 0 { + // Legacy budget path: max_tokens must exceed budget_tokens. + budget := maxInt(req.ThinkingBudget, anthropicMinThinkingBudget) + switch { + case out.MaxTokens <= 0: + out.MaxTokens = budget + anthropicDefaultMaxTokens + case budget >= out.MaxTokens: + return nil, &ConfigError{Msg: fmt.Sprintf("Anthropic ThinkingBudget %d must be below MaxTokens %d", budget, out.MaxTokens)} + } + out.Thinking = &anThinking{Type: "enabled", BudgetTokens: budget} + legacyThinking = true + } else { + // Adaptive thinking; the preset level selects the effort. + if out.MaxTokens <= 0 { + out.MaxTokens = anthropicAdaptiveMaxTokens + } + out.Thinking = &anThinking{Type: "adaptive"} + if req.Thinking != "enabled" { + out.OutputConfig = &anOutputConfig{Effort: req.Thinking} } } - out.Thinking = &anThinking{Type: "enabled", BudgetTokens: budget} - // Extended thinking rejects modified sampling; the provider default + // Thinking rejects modified sampling; the provider default // is the only accepted value. - out.Temperature, out.TopP = nil, nil + out.Temperature, out.TopP, out.TopK = nil, nil, nil } if out.MaxTokens <= 0 { out.MaxTokens = anthropicDefaultMaxTokens @@ -237,23 +269,48 @@ func buildAnthropicRequest(req *ChatRequest, model string, stream bool) ([]byte, } out.Tools = append(out.Tools, tool) } - if err := anthropicToolControls(req, &out, out.Thinking != nil); err != nil { + if !q.AnthropicToolJSON && req.ResponseFormat.jsonMode() { + schema := req.ResponseFormat.Schema + if len(schema) == 0 { + schema = json.RawMessage(`{"type":"object"}`) + } + if out.OutputConfig == nil { + out.OutputConfig = &anOutputConfig{} + } + out.OutputConfig.Format = &anOutputFormat{Type: "json_schema", Schema: schema} + } + if err := anthropicToolControls(req, &out, legacyThinking, out.Thinking != nil, q.AnthropicToolJSON); err != nil { return nil, err } // Messages. Hard edges: // - consecutive tool messages merge into ONE user message carrying // multiple tool_result blocks; - // - empty assistant turns synthesize a placeholder text block - // (Anthropic rejects empty content arrays). + // - an assistant turn with nothing to replay is dropped (Anthropic + // rejects empty and whitespace-only text blocks); the user turns + // around it merge so role alternation still holds. + dropped := false + appendMsg := func(role string, blocks []anBlock) { + if n := len(out.Messages); dropped && role == "user" && n > 0 && out.Messages[n-1].Role == "user" { + out.Messages[n-1].Content = append(out.Messages[n-1].Content, blocks...) + } else { + out.Messages = append(out.Messages, anMessage{Role: role, Content: blocks}) + } + dropped = false + } for i := 0; i < len(req.Messages); i++ { m := req.Messages[i] switch m.Role { case RoleSystem: - out.System = append(out.System, anSysBlock{Type: "text", Text: m.Content}) + if m.Content != "" { + out.System = append(out.System, anSysBlock{Type: "text", Text: m.Content}) + } case RoleUser: blocks := make([]anBlock, 0, maxInt(1, len(m.Parts))) if len(m.Parts) == 0 { + if m.Content == "" { + return nil, &ConfigError{Msg: fmt.Sprintf("message %d: user message has no content", i)} + } blocks = append(blocks, anBlock{Type: "text", Text: m.Content}) } else { blocks = append(blocks, anthropicPartBlocks(m.Parts)...) @@ -261,7 +318,7 @@ func buildAnthropicRequest(req *ChatRequest, model string, stream bool) ([]byte, if m.Cache && len(blocks) > 0 && blocks[0].Type == "text" { blocks[0].CacheControl = &anCacheControl{Type: "ephemeral"} } - out.Messages = append(out.Messages, anMessage{Role: "user", Content: blocks}) + appendMsg("user", blocks) case RoleAssistant: blocks, err := anthropicThinkingReplay(m, i) if err != nil { @@ -289,9 +346,10 @@ func buildAnthropicRequest(req *ChatRequest, model string, stream bool) ([]byte, }) } if len(blocks) == 0 { - blocks = []anBlock{{Type: "text", Text: " "}} // placeholder + dropped = true + continue } - out.Messages = append(out.Messages, anMessage{Role: "assistant", Content: blocks}) + appendMsg("assistant", blocks) case RoleTool: // Merge the run of consecutive tool messages. var results []anBlock @@ -306,7 +364,7 @@ func buildAnthropicRequest(req *ChatRequest, model string, stream bool) ([]byte, results = append(results, tr) } i-- // outer loop increment - out.Messages = append(out.Messages, anMessage{Role: "user", Content: results}) + appendMsg("user", results) } } @@ -333,10 +391,18 @@ type anUsage struct { CacheReadTokens int `json:"cache_read_input_tokens"` } +type anStopDetails struct { + Explanation string `json:"explanation"` +} + type anResponse struct { + ID string `json:"id"` + Model string `json:"model"` Content []anRespBlock `json:"content"` StopReason string `json:"stop_reason"` - Usage anUsage `json:"usage"` + // StopDetails.Explanation is the refusal explanation, when present. + StopDetails anStopDetails `json:"stop_details"` + Usage anUsage `json:"usage"` } // usageFromAnthropic maps Messages API usage onto canonical Usage. @@ -385,7 +451,10 @@ func parseAnthropicResponse(body []byte) (*ChatResult, error) { if len(r.Content) == 0 && r.StopReason == "" { return nil, fmt.Errorf("llm: response has no content blocks") } - res := &ChatResult{FinishReason: mapAnthropicStopReason(r.StopReason)} + res := &ChatResult{FinishReason: mapAnthropicStopReason(r.StopReason), Meta: ResponseMeta{ID: r.ID, Model: r.Model}} + if r.StopReason == "refusal" { + res.Refusal = r.StopDetails.Explanation + } var content, thinking []string for _, b := range r.Content { switch b.Type { @@ -422,6 +491,8 @@ type anStreamEvent struct { Type string `json:"type"` // message_start Message struct { + ID string `json:"id"` + Model string `json:"model"` Usage anUsage `json:"usage"` } `json:"message"` // content_block_start / content_block_stop @@ -431,20 +502,18 @@ type anStreamEvent struct { // delta and usage as top-level siblings, not nested under a // "message_delta" key. Delta struct { - Type string `json:"type"` // "text_delta" | "thinking_delta" | "signature_delta" | "input_json_delta" - StopReason string `json:"stop_reason"` - Text string `json:"text"` - Thinking string `json:"thinking"` - Signature string `json:"signature"` - PartialJSON string `json:"partial_json"` + Type string `json:"type"` // "text_delta" | "thinking_delta" | "signature_delta" | "input_json_delta" + StopReason string `json:"stop_reason"` + StopDetails anStopDetails `json:"stop_details"` + Text string `json:"text"` + Thinking string `json:"thinking"` + Signature string `json:"signature"` + PartialJSON string `json:"partial_json"` } `json:"delta"` // message_delta usage (top-level sibling of delta) Usage anUsage `json:"usage"` // message_delta: cumulative counts // error - Error *struct { - Type string `json:"type"` - Message string `json:"message"` - } `json:"error"` + Error json.RawMessage `json:"error"` // see streamErrorMessage } // mapAnthropicStreamEvent folds one Anthropic SSE payload into acc. @@ -456,6 +525,7 @@ func mapAnthropicStreamEvent(data []byte, acc *streamAccum) ([]Delta, bool, erro var deltas []Delta switch ev.Type { case "message_start": + acc.noteMeta(ev.Message.ID, ev.Message.Model) acc.usage = usageFromAnthropic(ev.Message.Usage) case "content_block_start": switch ev.ContentBlock.Type { @@ -496,6 +566,7 @@ func mapAnthropicStreamEvent(data []byte, acc *streamAccum) ([]Delta, bool, erro case "input_json_delta": if acc.jsonTool != "" && ev.Index == acc.jsonBlock { acc.content.WriteString(ev.Delta.PartialJSON) + acc.jsonWritten = acc.jsonWritten || ev.Delta.PartialJSON != "" deltas = append(deltas, Delta{Kind: DeltaContent, Text: ev.Delta.PartialJSON}) break } @@ -510,6 +581,10 @@ func mapAnthropicStreamEvent(data []byte, acc *streamAccum) ([]Delta, bool, erro }) } case "message_delta": + if ev.Delta.StopReason == "refusal" && ev.Delta.StopDetails.Explanation != "" && acc.refusal.Len() == 0 { + acc.refusal.WriteString(ev.Delta.StopDetails.Explanation) + deltas = append(deltas, Delta{Kind: DeltaRefusal, Text: ev.Delta.StopDetails.Explanation}) + } if ev.Delta.StopReason != "" { acc.finishReason = mapAnthropicStopReason(ev.Delta.StopReason) acc.sawFinish = true @@ -538,8 +613,8 @@ func mapAnthropicStreamEvent(data []byte, acc *streamAccum) ([]Delta, bool, erro return deltas, true, nil case "error": msg := "provider stream error" - if ev.Error != nil && ev.Error.Message != "" { - msg = ev.Error.Message + if m, ok := streamErrorMessage(ev.Error); ok && m != "stream error" { + msg = m } return deltas, false, fmt.Errorf("llm: %s", msg) case "ping", "content_block_stop": @@ -586,9 +661,12 @@ func listModelsAnthropic(ctx context.Context, pc *providerClient) ([]Model, erro } out = append(out, mm) } - if !p.HasMore || p.LastID == "" { + if !p.HasMore { return out, nil } + if p.LastID == "" { + return nil, terminal(errors.New("llm: anthropic models: has_more without last_id (cannot page; refusing a partial listing)")) + } pageID = p.LastID } return nil, fmt.Errorf("%w (%d pages)", ErrModelListTruncated, maxModelPages) diff --git a/anthropic_test.go b/anthropic_test.go index 442a7e3..837fc36 100644 --- a/anthropic_test.go +++ b/anthropic_test.go @@ -6,6 +6,7 @@ import ( "fmt" "net/http" "net/http/httptest" + "strings" "testing" ) @@ -158,24 +159,42 @@ func TestBuildAnthropicRequest_EmptyAssistantPlaceholder(t *testing.T) { body, _ := buildAnthropicRequest(req, "claude", false) var m map[string]any json.Unmarshal(body, &m) + // Anthropic rejects empty and whitespace-only text blocks: the empty + // assistant turn is dropped and the user turns around it merge. msgs := m["messages"].([]any) - asst := msgs[1].(map[string]any) - blocks := asst["content"].([]any) - if len(blocks) != 1 || blocks[0].(map[string]any)["type"] != "text" { - t.Fatalf("empty assistant blocks = %v, want one placeholder text", blocks) + if len(msgs) != 1 || msgs[0].(map[string]any)["role"] != "user" { + t.Fatalf("messages = %v, want one merged user turn", msgs) + } + if blocks := msgs[0].(map[string]any)["content"].([]any); len(blocks) != 2 { + t.Fatalf("merged blocks = %v, want 2 text blocks", blocks) + } + if strings.Contains(string(body), `"text":" "`) { + t.Fatalf("whitespace-only block on wire: %s", body) } } func TestBuildAnthropicRequest_ThinkingLevels(t *testing.T) { - cases := map[string]int{"low": 1024, "medium": 8192, "high": 16384, "max": 16384} - for level, want := range cases { + // Presets map to adaptive thinking + output_config.effort (AC-1); "enabled" carries no effort. + cases := map[string]string{"enabled": "", "low": "low", "medium": "medium", "high": "high", "max": "max"} + for level, wantEffort := range cases { req := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, Thinking: level} body, _ := buildAnthropicRequest(req, "claude", false) var m map[string]any json.Unmarshal(body, &m) th := m["thinking"].(map[string]any) - if th["budget_tokens"].(float64) != float64(want) { - t.Errorf("%s budget = %v, want %d", level, th["budget_tokens"], want) + if th["type"] != "adaptive" { + t.Errorf("%s thinking.type = %v, want adaptive", level, th["type"]) + } + if _, ok := th["budget_tokens"]; ok { + t.Errorf("%s: adaptive thinking must not carry budget_tokens", level) + } + oc, hasOC := m["output_config"].(map[string]any) + if wantEffort == "" { + if hasOC { + t.Errorf("%s: output_config must be absent, got %v", level, oc) + } + } else if oc["effort"] != wantEffort { + t.Errorf("%s effort = %v, want %s", level, oc["effort"], wantEffort) } } // disabled → no thinking field diff --git a/anthropic_thinking_test.go b/anthropic_thinking_test.go index a2b8ecc..c52d086 100644 --- a/anthropic_thinking_test.go +++ b/anthropic_thinking_test.go @@ -20,39 +20,48 @@ func decodeAnthropicBody(t *testing.T, req *ChatRequest) map[string]any { return m } -// Anthropic requires max_tokens > thinking.budget_tokens. An unset -// MaxTokens must leave room for the budget plus visible output. -func TestBuildAnthropicRequest_ThinkingDefaultMaxTokensExceedsBudget(t *testing.T) { +// Adaptive presets default max_tokens to 16384 (AC-3); the legacy explicit +// budget path keeps budget + visible-output room (AC-2). +func TestBuildAnthropicRequest_ThinkingDefaultMaxTokens(t *testing.T) { for _, level := range []string{"enabled", "low", "medium", "high", "max"} { m := decodeAnthropicBody(t, &ChatRequest{Thinking: level, Messages: []Message{{Role: RoleUser, Content: "x"}}}) - budget := m["thinking"].(map[string]any)["budget_tokens"].(float64) - maxTok := m["max_tokens"].(float64) - if maxTok <= budget { - t.Errorf("%s: max_tokens %v must exceed budget_tokens %v", level, maxTok, budget) - } - if maxTok != budget+anthropicDefaultMaxTokens { - t.Errorf("%s: max_tokens = %v, want budget+%d", level, maxTok, anthropicDefaultMaxTokens) + if maxTok := m["max_tokens"].(float64); maxTok != 16384 { + t.Errorf("%s: max_tokens = %v, want 16384", level, maxTok) } } + m := decodeAnthropicBody(t, &ChatRequest{Thinking: "high", ThinkingBudget: 3000, Messages: []Message{{Role: RoleUser, Content: "x"}}}) + if maxTok := m["max_tokens"].(float64); maxTok != 3000+anthropicDefaultMaxTokens { + t.Errorf("budget path: max_tokens = %v, want budget+%d", maxTok, anthropicDefaultMaxTokens) + } + m = decodeAnthropicBody(t, &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}}) + if maxTok := m["max_tokens"].(float64); maxTok != anthropicDefaultMaxTokens { + t.Errorf("no thinking: max_tokens = %v, want %d", maxTok, anthropicDefaultMaxTokens) + } } -// A preset level larger than an explicit MaxTokens is clamped below it. -func TestBuildAnthropicRequest_ThinkingPresetClampedBelowMaxTokens(t *testing.T) { - m := decodeAnthropicBody(t, &ChatRequest{Thinking: "high", MaxTokens: 4000, Messages: []Message{{Role: RoleUser, Content: "x"}}}) - if m["max_tokens"].(float64) != 4000 { - t.Errorf("explicit max_tokens must be honored, got %v", m["max_tokens"]) - } - // Half the cap stays for visible output (never a one-token answer). - if b := m["thinking"].(map[string]any)["budget_tokens"].(float64); b != 2000 { - t.Errorf("budget_tokens = %v, want 2000 (half of max_tokens)", b) +// Under adaptive thinking an explicit MaxTokens is sent as-is: no clamping, +// no ConfigError (AC-3). Replaces the preset-clamp expectations. +func TestBuildAnthropicRequest_AdaptiveMaxTokensAsIs(t *testing.T) { + for _, mt := range []int{1024, 1500, 4000} { + m := decodeAnthropicBody(t, &ChatRequest{Thinking: "high", MaxTokens: mt, Messages: []Message{{Role: RoleUser, Content: "x"}}}) + if m["max_tokens"].(float64) != float64(mt) { + t.Errorf("max_tokens = %v, want %d as-is", m["max_tokens"], mt) + } + if th := m["thinking"].(map[string]any); th["type"] != "adaptive" { + t.Errorf("thinking = %v, want adaptive", th) + } } - m = decodeAnthropicBody(t, &ChatRequest{Thinking: "high", MaxTokens: 1500, Messages: []Message{{Role: RoleUser, Content: "x"}}}) - if b := m["thinking"].(map[string]any)["budget_tokens"].(float64); b != 1024 { - t.Errorf("small cap: budget_tokens = %v, want the 1024 minimum", b) +} + +// The explicit-budget path is unchanged: N >= 1024, max_tokens > N (AC-2). +func TestBuildAnthropicRequest_ExplicitBudgetPath(t *testing.T) { + m := decodeAnthropicBody(t, &ChatRequest{Thinking: "medium", ThinkingBudget: 500, Messages: []Message{{Role: RoleUser, Content: "x"}}}) + th := m["thinking"].(map[string]any) + if th["type"] != "enabled" || th["budget_tokens"].(float64) != 1024 { + t.Errorf("thinking = %v, want enabled/1024 floor", th) } - m = decodeAnthropicBody(t, &ChatRequest{Thinking: "low", MaxTokens: 1500, Messages: []Message{{Role: RoleUser, Content: "x"}}}) - if b := m["thinking"].(map[string]any)["budget_tokens"].(float64); b != 1024 { - t.Errorf("preset below cap is kept: budget_tokens = %v", b) + if _, ok := m["output_config"]; ok { + t.Errorf("budget path must not emit output_config.effort: %v", m["output_config"]) } } @@ -61,7 +70,7 @@ func TestBuildAnthropicRequest_ThinkingBudgetConflicts(t *testing.T) { cases := []ChatRequest{ {Thinking: "enabled", ThinkingBudget: 8000, MaxTokens: 8000}, {Thinking: "enabled", ThinkingBudget: 2000, MaxTokens: 1500}, - {Thinking: "low", MaxTokens: 1024}, // cannot clamp below the 1024 minimum + {Thinking: "low", ThinkingBudget: 1024, MaxTokens: 1024}, // explicit budget cannot fit under the cap } for i, req := range cases { req.Messages = []Message{{Role: RoleUser, Content: "x"}} @@ -75,14 +84,18 @@ func TestBuildAnthropicRequest_ThinkingBudgetConflicts(t *testing.T) { // Extended thinking rejects modified temperature / top_p. func TestBuildAnthropicRequest_ThinkingOmitsSampling(t *testing.T) { - m := decodeAnthropicBody(t, &ChatRequest{Thinking: "low", Temperature: 0.3, TopP: 0.5, Messages: []Message{{Role: RoleUser, Content: "x"}}}) - if _, ok := m["temperature"]; ok { - t.Error("temperature must be omitted with thinking") - } - if _, ok := m["top_p"]; ok { - t.Error("top_p must be omitted with thinking") + for _, req := range []ChatRequest{{Thinking: "low"}, {Thinking: "low", ThinkingBudget: 2000}} { // adaptive and budget + req.Temperature, req.TopP = 0.3, 0.5 + req.Messages = []Message{{Role: RoleUser, Content: "x"}} + m := decodeAnthropicBody(t, &req) + if _, ok := m["temperature"]; ok { + t.Errorf("%+v: temperature must be omitted with thinking", req.Thinking) + } + if _, ok := m["top_p"]; ok { + t.Error("top_p must be omitted with thinking") + } } - m = decodeAnthropicBody(t, &ChatRequest{Temperature: 0.3, Messages: []Message{{Role: RoleUser, Content: "x"}}}) + m := decodeAnthropicBody(t, &ChatRequest{Temperature: 0.3, Messages: []Message{{Role: RoleUser, Content: "x"}}}) if m["temperature"].(float64) != 0.3 { t.Error("temperature must survive without thinking") } diff --git a/chat.go b/chat.go index 2707974..1e07d8c 100644 --- a/chat.go +++ b/chat.go @@ -8,7 +8,6 @@ import ( "fmt" "io" "net/http" - neturl "net/url" "strings" "sync/atomic" "time" @@ -53,8 +52,9 @@ type providerClient struct { // clientOpts carries SDK-level settings into a providerClient. type clientOpts struct { retry RetryPolicy - idle time.Duration // SSE idle watchdog; 0 = process default - observer func(LearnEvent) // learn-once observer; nil = process default + idle time.Duration // SSE idle watchdog; 0 = process default + observer func(LearnEvent) // learn-once observer; nil = process default + attempt func(AttemptEvent) // per-HTTP-attempt observer; nil = off } // retryDelay picks Retry-After when present, else the policy's capped @@ -153,11 +153,11 @@ func (pc *providerClient) requestTimeout() time.Duration { // ── request assembly ───────────────────────────────────────────────────── -// buildChatRequest dispatches format-specific serialization. stream=false -// yields the buffered request; stream=true the SSE request. -func (pc *providerClient) buildChatRequest(req *ChatRequest, model string, stream bool) ([]byte, string, error) { +// validateChatRequest applies the boundary validation shared by chat and +// token counting and returns the effective model. +func validateChatRequest(req *ChatRequest, model string) (string, error) { if req == nil { - return nil, "", &ConfigError{Msg: "chat request is nil"} + return "", &ConfigError{Msg: "chat request is nil"} } // Reject unknown roles loudly: OpenAI would silently send them as user // messages and Anthropic/Gemini would silently drop them. @@ -165,26 +165,36 @@ func (pc *providerClient) buildChatRequest(req *ChatRequest, model string, strea switch m.Role { case RoleUser, RoleAssistant, RoleSystem, RoleTool: default: - return nil, "", &ConfigError{Msg: fmt.Sprintf("message %d: unknown role %q", i, string(m.Role))} + return "", &ConfigError{Msg: fmt.Sprintf("message %d: unknown role %q", i, string(m.Role))} } } if err := validateRequestContent(req.Messages); err != nil { - return nil, "", err + return "", err } if err := validateRequestControls(req); err != nil { - return nil, "", err + return "", err } if model == "" { model = req.Model } if model == "" { - return nil, "", &ConfigError{Msg: "no model set (ChatRequest.Model empty and no ChatClient model)"} + return "", &ConfigError{Msg: "no model set (ChatRequest.Model empty and no ChatClient model)"} + } + return model, nil +} + +// buildChatRequest dispatches format-specific serialization. stream=false +// yields the buffered request; stream=true the SSE request. +func (pc *providerClient) buildChatRequest(req *ChatRequest, model string, stream bool) ([]byte, string, error) { + model, err := validateChatRequest(req, model) + if err != nil { + return nil, "", err } body, url, err := pc.buildFormatBody(req, model, stream) if err != nil { return nil, "", err } - body, err = mergeExtra(body, req.Extra) + body, err = mergeExtra(body, req.Extra, pc.cfg.Format == FormatGemini) return body, url, err } @@ -192,13 +202,13 @@ func (pc *providerClient) buildChatRequest(req *ChatRequest, model string, strea func (pc *providerClient) buildFormatBody(req *ChatRequest, model string, stream bool) ([]byte, string, error) { switch pc.cfg.Format { case FormatAnthropic: - body, err := buildAnthropicRequest(req, model, stream) + body, err := buildAnthropicRequestQ(req, model, stream, pc.cfg.Quirks) return body, pc.base + "/v1/messages", err case FormatGemini: body, err := buildGeminiRequest(req, model, stream) // The model rides the URL path: escape it so a model id can never // rewrite the path or smuggle a query. - m := neturl.PathEscape(model) + m := geminiModelPath(model) if stream { return body, fmt.Sprintf("%s/v1beta/models/%s:streamGenerateContent?alt=sse", pc.base, m), err } @@ -207,7 +217,11 @@ func (pc *providerClient) buildFormatBody(req *ChatRequest, model string, stream if pc.cfg.Quirks.NoJSONSchema && req.ResponseFormat != nil && req.ResponseFormat.Type == ResponseJSONSchema { return nil, "", &ConfigError{Msg: pc.cfg.ID + " supports ResponseFormat json_object only, not json_schema"} } - if useResponsesAPI(pc.learn, pc.cfg.Format, model, req) { + useRS := useResponsesAPI(pc.learn, pc.cfg.Format, model, req) + if err := validateOpenAIFormatParts(req.Messages, useRS); err != nil { + return nil, "", err + } + if useRS { body, err := json.Marshal(buildResponsesRequest(req, model, stream)) return body, pc.base + "/responses", err } @@ -234,7 +248,7 @@ var extraReserved = map[string]bool{ // mergeExtra overlays ChatRequest.Extra onto a marshaled body: top-level // keys replace the SDK's own. "stream" is reserved because the SDK's // buffered/SSE handling depends on it. -func mergeExtra(body []byte, extra map[string]any) ([]byte, error) { +func mergeExtra(body []byte, extra map[string]any, deepGen bool) ([]byte, error) { if len(extra) == 0 { return body, nil } @@ -250,22 +264,45 @@ func mergeExtra(body []byte, extra map[string]any) ([]byte, error) { if err != nil { return nil, &ConfigError{Msg: fmt.Sprintf("ChatRequest.Extra[%q]: %v", k, err)} } + if deepGen && k == "generationConfig" { + raw = mergeGenerationConfig(m[k], raw) + } m[k] = raw } return json.Marshal(m) } +// mergeGenerationConfig sets the Extra object's keys one by one over the +// SDK's generationConfig object (nested objects are replaced whole). When +// either side is not a JSON object the Extra value replaces as before. +func mergeGenerationConfig(sdk, extra json.RawMessage) json.RawMessage { + var base, over map[string]json.RawMessage + if json.Unmarshal(sdk, &base) != nil || base == nil || json.Unmarshal(extra, &over) != nil || over == nil { + return extra + } + for k, v := range over { + base[k] = v + } + out, err := json.Marshal(base) + if err != nil { + return extra + } + return out +} + // setHeaders applies format-specific authentication headers, then the // provider's custom Headers (last, so they can override; an empty value // removes the header). Keys and header values are never logged. -func (pc *providerClient) setHeaders(h http.Header) { +func (pc *providerClient) setHeaders(h http.Header, perReq map[string]string) { pc.setAuthHeaders(h) - for k, v := range pc.cfg.Headers { - if v == "" { - h.Del(k) - continue + for _, m := range []map[string]string{pc.cfg.Headers, perReq} { + for k, v := range m { + if v == "" { + h.Del(k) + continue + } + h.Set(k, v) } - h.Set(k, v) } } @@ -358,32 +395,32 @@ func (pc *providerClient) httpError(status int, body []byte) *APIError { // post sends one HTTP request and reads the full body (capped). Non-2xx // responses return an *APIError with the Retry-After hint alongside. -func (pc *providerClient) post(ctx context.Context, client *http.Client, url string, body []byte) (data []byte, ra time.Duration, err error) { +func (pc *providerClient) post(ctx context.Context, client *http.Client, url string, body []byte) (data []byte, hdr http.Header, ra time.Duration, err error) { req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(body)) if err != nil { - return nil, 0, &ConfigError{Msg: "build request: " + err.Error()} + return nil, nil, 0, &ConfigError{Msg: "build request: " + err.Error()} } req.Header.Set("Content-Type", "application/json") req.Header.Set("Accept", "application/json") - pc.setHeaders(req.Header) + pc.setHeaders(req.Header, reqHeadersFrom(ctx)) - resp, err := client.Do(req) + resp, err := doObserved(ctx, client, req) if err != nil { - return nil, 0, err + return nil, nil, 0, err } defer func() { _ = resp.Body.Close() }() data, err = io.ReadAll(io.LimitReader(resp.Body, maxResponseSize+1)) if err != nil { - return nil, 0, err + return nil, nil, 0, err } if len(data) > maxResponseSize { - return nil, 0, fmt.Errorf("llm: response exceeds %d bytes", maxResponseSize) + return nil, nil, 0, terminal(fmt.Errorf("llm: response exceeds %d bytes", maxResponseSize)) } ra = parseRetryAfter(resp.Header.Get("Retry-After"), time.Now()) if resp.StatusCode < 200 || resp.StatusCode >= 300 { - return data, ra, pc.httpError(resp.StatusCode, data) + return data, resp.Header, ra, pc.httpError(resp.StatusCode, data) } - return data, ra, nil + return data, resp.Header, ra, nil } // ── error classification helpers ───────────────────────────────────────── @@ -460,6 +497,11 @@ func billingExhausted(e *APIError) bool { // contract as streaming — so a failing provider can never hold a caller for // timeout × attempts. func (pc *providerClient) call(ctx context.Context, req *ChatRequest, model string) (*ChatResult, error) { + ctx, herr := withReqHeaders(ctx, chatReqHeaders(req)) + if herr != nil { + return nil, herr + } + ctx = pc.withAttempts(ctx, "chat", model) ctx, cancel := context.WithTimeout(ctx, pc.requestTimeout()) defer cancel() var res *ChatResult @@ -468,11 +510,12 @@ func (pc *providerClient) call(ctx context.Context, req *ChatRequest, model stri if err != nil { return 0, terminal(err) } - data, ra, err := pc.post(ctx, pc.buffered(), url, body) + data, hdr, ra, err := pc.post(ctx, pc.buffered(), url, body) if err != nil { return ra, err } res, err = pc.parseResponse(data, url, pc.jsonTool(req)) + res.withHeaderMeta(hdr) return 0, terminal(err) }, func(apiErr *APIError) bool { return pc.learnFromChatError(req, apiErr) @@ -506,7 +549,7 @@ func (pc *providerClient) learnFromChatError(req *ChatRequest, apiErr *APIError) // jsonTool is the Anthropic JSON-mode tool for req ("" elsewhere). func (pc *providerClient) jsonTool(req *ChatRequest) string { - if pc.cfg.Format != FormatAnthropic { + if pc.cfg.Format != FormatAnthropic || !pc.cfg.Quirks.AnthropicToolJSON { return "" } return anthropicJSONToolName(req) @@ -554,6 +597,11 @@ func (pc *providerClient) callStream(ctx context.Context, req *ChatRequest, mode if pc.learn.forceBuffered.Load() { return pc.call(ctx, req, model) } + ctx, herr := withReqHeaders(ctx, chatReqHeaders(req)) + if herr != nil { + return nil, herr + } + ctx = pc.withAttempts(ctx, "stream", model) deadlineCtx, cancel := context.WithTimeout(ctx, pc.requestTimeout()) defer cancel() @@ -588,17 +636,22 @@ func (pc *providerClient) callStream(ctx context.Context, req *ChatRequest, mode // A successful HTTP response already contains this generation. // Return its buffered parse (including errors) without replacing // it with another generation or resetting the request deadline. + pc.emitAttempt(deadlineCtx, out.err, 0) return out.result, out.err case out.success(): + pc.emitAttempt(deadlineCtx, nil, 0) return out.result, nil case out.abort != nil: + pc.emitAttempt(deadlineCtx, out.abort, 0) return out.result, out.abort case out.err != nil && out.result != nil: // Partial output was delivered: never retry, surface with the // partial result (a retry would duplicate user-visible output). + pc.emitAttempt(deadlineCtx, out.err, 0) return out.result, out.err case out.learnRetry: // learn-once applied; retry without backoff lastErr = out.apiErr // every learn trigger is a provider 400 + pc.emitAttempt(deadlineCtx, out.apiErr, 0) if attempt < n-1 { continue } @@ -610,22 +663,34 @@ func (pc *providerClient) callStream(ctx context.Context, req *ChatRequest, mode rateErr, rateRA = out.apiErr, out.retryAfter } lastErr = out.apiErr - if attempt < n-1 && retrySleep(deadlineCtx, pc.retryDelay(out.retryAfter, attempt)) { - continue + if attempt < n-1 { + d := pc.retryDelay(out.retryAfter, attempt) + pc.emitAttempt(deadlineCtx, out.apiErr, d) + if waitRetry(deadlineCtx, d) { + continue + } + } else { + pc.emitAttempt(deadlineCtx, out.apiErr, 0) } if rateErr != nil { return nil, &RateLimitError{APIError: *rateErr, Attempts: attempt + 1, RetryAfter: rateRA} } return nil, out.apiErr case out.apiErr != nil: + pc.emitAttempt(deadlineCtx, out.apiErr, 0) return nil, out.apiErr case out.err != nil && attempt < n-1: lastErr = out.err - if retrySleep(deadlineCtx, pc.retryDelay(0, attempt)) { + d := pc.retryDelay(0, attempt) + pc.emitAttempt(deadlineCtx, out.err, d) + if waitRetry(deadlineCtx, d) { continue } - return nil, deadlineCtx.Err() // interrupted by deadline/cancel, not exhaustion + // Interrupted by (or about to outlive) the deadline/cancel, not + // exhaustion: keep the last error retrievable. + return nil, withBudgetCause(deadlineCtx, out.err) default: + pc.emitAttempt(deadlineCtx, out.err, 0) return nil, out.err } } @@ -663,9 +728,9 @@ func (pc *providerClient) attemptStream(ctx context.Context, url string, body [] } req.Header.Set("Content-Type", "application/json") req.Header.Set("Accept", "text/event-stream") - pc.setHeaders(req.Header) + pc.setHeaders(req.Header, reqHeadersFrom(ctx)) - resp, derr := pc.streamHTTP.Do(req) + resp, derr := doObserved(ctx, pc.streamHTTP, req) if derr != nil { return streamOutcome{err: derr} // transport error: retryable } @@ -705,6 +770,7 @@ func (pc *providerClient) attemptStream(ctx context.Context, url string, body [] return streamOutcome{bufferedResponse: true, err: fmt.Errorf("llm: response exceeds %d bytes", maxResponseSize)} } result, err := pc.parseResponse(data, url, jsonTool) + result.withHeaderMeta(resp.Header) return streamOutcome{bufferedResponse: true, result: result, err: err} } @@ -732,24 +798,27 @@ func (pc *providerClient) attemptStream(ctx context.Context, url string, body [] switch { case perr == nil, errors.Is(perr, errStreamStop): - if perr == nil && !acc.sawFinish { + // A bare [DONE] (nothing emitted, no finish) is an empty premature + // close, never a silent empty success. [DONE] after content is + // tolerated: some gateways omit finish_reason. + if !acc.sawFinish && (perr == nil || !acc.emitted) { // Every format sends an explicit completion signal ([DONE], // finish_reason, message_stop, finishReason, response.completed) // before EOF. It never arrived: the provider dropped the stream. // Never surface this as a silent (partial or empty) success. if acc.emitted { return streamOutcome{ - result: acc.result(), + result: acc.resultWith(resp.Header), err: fmt.Errorf("llm: stream closed before completion: %w", errPrematureClose), } } return streamOutcome{err: errPrematureClose} } - return streamOutcome{result: acc.result()} + return streamOutcome{result: acc.resultWith(resp.Header)} default: var ca *consumerAbort if errors.As(perr, &ca) { - return streamOutcome{result: acc.result(), abort: &StreamAbortedError{Reason: ca.err}} + return streamOutcome{result: acc.resultWith(resp.Header), abort: &StreamAbortedError{Reason: ca.err}} } if !acc.emitted { // No output delivered yet: a full retry duplicates nothing. @@ -759,7 +828,7 @@ func (pc *providerClient) attemptStream(ctx context.Context, url string, body [] } // Partial output was delivered: never retry, surface with partial. return streamOutcome{ - result: acc.result(), + result: acc.resultWith(resp.Header), err: fmt.Errorf("llm: stream failed after partial output: %w", perr), } } @@ -790,6 +859,7 @@ type thinkingAccum struct { type streamAccum struct { content strings.Builder reasoning strings.Builder + refusal strings.Builder calls []*toolCallAccum callIndex map[int]*toolCallAccum thinking []*thinkingAccum @@ -801,6 +871,9 @@ type streamAccum struct { thinkingSignature string // legacy single signature (last block's) jsonTool string // Anthropic JSON-mode synthetic tool name ("" = off) jsonBlock int // content-block index of the JSON-mode tool_use (-1 = none) + jsonWritten bool // JSON-mode tool input produced any content + lastToolKey int // OpenAI-format: provider key of the most recent tool-call delta + metaID, metaModel string // provider-reported response id / model } func newStreamAccum() *streamAccum { @@ -811,6 +884,29 @@ func newStreamAccum() *streamAccum { } } +// openAIToolKey resolves the provider key of an OpenAI-format streamed +// tool-call delta. An explicit index wins. Without one (some gateways omit +// it), a delta continues the most recent call unless it carries a +// different non-empty id, which starts a new call. +func (a *streamAccum) openAIToolKey(index *int, id string) int { + key := a.lastToolKey + switch { + case index != nil: + key = *index + case id != "" && len(a.calls) > 0: + if cur, ok := a.callIndex[key]; ok && cur.id != "" && cur.id != id { + key = 0 + for k := range a.callIndex { + if k >= key { + key = k + 1 + } + } + } + } + a.lastToolKey = key + return key +} + // call returns the accumulator for provider index idx, creating it in // first-seen order. func (a *streamAccum) call(idx int) *toolCallAccum { @@ -835,6 +931,23 @@ func (a *streamAccum) block(idx int) *thinkingAccum { return b } +// noteMeta records the first non-empty provider id/model seen. +func (a *streamAccum) noteMeta(id, model string) { + if a.metaID == "" { + a.metaID = id + } + if a.metaModel == "" { + a.metaModel = model + } +} + +// resultWith is result() plus the response-header metadata. +func (a *streamAccum) resultWith(h http.Header) *ChatResult { + res := a.result() + res.withHeaderMeta(h) + return res +} + func (a *streamAccum) result() *ChatResult { res := &ChatResult{ Content: a.content.String(), @@ -842,12 +955,61 @@ func (a *streamAccum) result() *ChatResult { ThinkingSignature: a.thinkingSignature, FinishReason: a.finishReason, Usage: a.usage, + Refusal: a.refusal.String(), + Meta: ResponseMeta{ID: a.metaID, Model: a.metaModel}, } + res.FinishReason = refusalFinish(res.FinishReason, res.Refusal) for _, b := range a.thinking { res.ThinkingBlocks = append(res.ThinkingBlocks, ThinkingBlock{Text: b.text.String(), Signature: b.sig, Redacted: b.redacted}) } + if a.jsonBlock >= 0 && !a.jsonWritten { + res.Content += "{}" // JSON-mode tool with empty input, as buffered + } for _, c := range a.calls { - res.ToolCalls = append(res.ToolCalls, ToolCall{ID: c.id, Name: c.name, Arguments: c.args.String(), Signature: c.sig}) + args := c.args.String() + if args == "" { + args = "{}" // a no-argument call is an empty object on every format + } + res.ToolCalls = append(res.ToolCalls, ToolCall{ID: c.id, Name: c.name, Arguments: args, Signature: c.sig}) } return res } + +// refusalFinish maps a refusal that finished "stop" (or unmapped) to +// content_filter; length and tool_calls stay as the provider sent them. +func refusalFinish(finish, refusal string) string { + if refusal != "" && (finish == FinishStop || finish == "") { + return FinishContentFilter + } + return finish +} + +// streamErrorMessage interprets the "error" field of a stream chunk. Only a +// non-empty JSON object counts as a provider error (its message, or a +// generic one when it has none); null, "", false, 0 and {} are ignored. +func streamErrorMessage(raw json.RawMessage) (string, bool) { + raw = bytes.TrimSpace(raw) + if len(raw) == 0 || raw[0] != '{' { + return "", false + } + var obj map[string]json.RawMessage + if json.Unmarshal(raw, &obj) != nil || len(obj) == 0 { + return "", false + } + var msg string + if m, ok := obj["message"]; ok { + _ = json.Unmarshal(m, &msg) + } + if msg == "" { + msg = "stream error" + } + return msg, true +} + +// chatReqHeaders returns req.Headers, tolerating a nil request (rejected later). +func chatReqHeaders(req *ChatRequest) map[string]string { + if req == nil { + return nil + } + return req.Headers +} diff --git a/content.go b/content.go index 5de2173..0f0bc1c 100644 --- a/content.go +++ b/content.go @@ -3,6 +3,7 @@ package llm import ( "fmt" "mime" + "net/url" "strings" ) @@ -18,7 +19,7 @@ func validateRequestContent(messages []Message) error { return err } for _, p := range m.Parts { - total += len(p.Image) + total += len(p.Image) + len(p.Data) if total > MaxRequestImageBytes { return &ConfigError{Msg: fmt.Sprintf("message %d: request inline images exceed %d bytes", i, MaxRequestImageBytes)} } @@ -49,10 +50,16 @@ func validateMessageContent(m Message, index int) error { if p.Image != nil || p.MIMEType != "" { return &ConfigError{Msg: fmt.Sprintf("message %d part %d: text part has image fields", index, j)} } + if p.URL != "" || p.Data != nil || p.Name != "" { + return &ConfigError{Msg: fmt.Sprintf("message %d part %d: text part has non-text fields", index, j)} + } case ContentPartImage: if p.Text != "" { return &ConfigError{Msg: fmt.Sprintf("message %d part %d: image part has text fields", index, j)} } + if p.URL != "" || p.Data != nil || p.Name != "" { + return &ConfigError{Msg: fmt.Sprintf("message %d part %d: image part has non-image fields", index, j)} + } media, params, err := mime.ParseMediaType(p.MIMEType) if err != nil || len(params) != 0 || !supportedImageMIME[strings.ToLower(media)] || media != strings.ToLower(media) { return &ConfigError{Msg: fmt.Sprintf("message %d part %d: MIMEType must be an image type", index, j)} @@ -63,6 +70,14 @@ func validateMessageContent(m Message, index int) error { if len(p.Image) > MaxImageBytes { return &ConfigError{Msg: fmt.Sprintf("message %d part %d: image exceeds %d bytes", index, j, MaxImageBytes)} } + case ContentPartDocument: + if err := validateDocumentPart(p, index, j); err != nil { + return err + } + case ContentPartImageURL: + if err := validateImageURLPart(p, index, j); err != nil { + return err + } default: return &ConfigError{Msg: fmt.Sprintf("message %d part %d: unknown content part type %q", index, j, p.Type)} } @@ -70,8 +85,47 @@ func validateMessageContent(m Message, index int) error { return nil } +// maxImageURLBytes is a sanity bound on a remote image URL. +const maxImageURLBytes = 8 << 10 + +var supportedDocumentMIME = map[string]bool{"application/pdf": true, "text/plain": true} + +func validateDocumentPart(p ContentPart, index, j int) error { + media, params, err := mime.ParseMediaType(p.MIMEType) + if err != nil || len(params) != 0 || !supportedDocumentMIME[strings.ToLower(media)] { + return &ConfigError{Msg: fmt.Sprintf("message %d part %d: document MIMEType must be application/pdf or text/plain", index, j)} + } + if p.Text != "" || p.Image != nil || p.URL != "" { + return &ConfigError{Msg: fmt.Sprintf("message %d part %d: document part has non-document fields", index, j)} + } + if len(p.Data) == 0 { + return &ConfigError{Msg: fmt.Sprintf("message %d part %d: document data is empty", index, j)} + } + if len(p.Data) > MaxDocumentBytes { + return &ConfigError{Msg: fmt.Sprintf("message %d part %d: document exceeds %d bytes", index, j, MaxDocumentBytes)} + } + return nil +} + +func validateImageURLPart(p ContentPart, index, j int) error { + if p.Text != "" || p.Image != nil || p.Data != nil || p.Name != "" { + return &ConfigError{Msg: fmt.Sprintf("message %d part %d: image URL part has non-URL fields", index, j)} + } + u, err := url.Parse(p.URL) + if err != nil || len(p.URL) > maxImageURLBytes || u.Host == "" || (!strings.EqualFold(u.Scheme, "http") && !strings.EqualFold(u.Scheme, "https")) { + return &ConfigError{Msg: fmt.Sprintf("message %d part %d: image URL must be a valid http(s) URL (at most %d bytes)", index, j, maxImageURLBytes)} + } + if p.MIMEType != "" { + media, params, err := mime.ParseMediaType(p.MIMEType) + if err != nil || len(params) != 0 || !supportedImageMIME[strings.ToLower(media)] { + return &ConfigError{Msg: fmt.Sprintf("message %d part %d: MIMEType must be an image type", index, j)} + } + } + return nil +} + var supportedImageMIME = map[string]bool{ - "image/png": true, "image/jpeg": true, + "image/png": true, "image/jpeg": true, "image/jpg": true, "image/gif": true, "image/webp": true, } @@ -101,11 +155,57 @@ func partsText(m Message) string { return strings.Join(texts, "\n") } -// imageParts returns a message's inline-image parts. -func imageParts(m Message) []ContentPart { +// attachmentParts returns a message's image, image-URL and document parts. +func attachmentParts(m Message) []ContentPart { + var out []ContentPart + for _, p := range m.Parts { + if p.Type != ContentPartText { + out = append(out, p) + } + } + return out +} + +const defaultDocumentName = "document.pdf" + +// canonMIME is the canonical wire form of a (case-insensitive) MIME type. +func canonMIME(m string) string { return wireMIME(strings.ToLower(m)) } + +func documentName(p ContentPart) string { + if p.Name == "" { + return defaultDocumentName + } + return p.Name +} + +// validateOpenAIFormatParts rejects documents the OpenAI-format wires cannot +// carry: documents in tool results, and any document that is not a PDF. +// responses selects the endpoint named in the tool-result error. +func validateOpenAIFormatParts(msgs []Message, responses bool) error { + for i, m := range msgs { + for j, p := range m.Parts { + if p.Type != ContentPartDocument { + continue + } + if m.Role == RoleTool { + if responses { + return &ConfigError{Msg: "tool results cannot carry documents on the Responses API"} + } + return &ConfigError{Msg: "tool results cannot carry documents on OpenAI-format providers"} + } + if strings.ToLower(p.MIMEType) != "application/pdf" { + return &ConfigError{Msg: fmt.Sprintf("message %d part %d: OpenAI-format documents support application/pdf only", i, j)} + } + } + } + return nil +} + +// imageLikeParts returns inline-image and image-URL parts. +func imageLikeParts(m Message) []ContentPart { var out []ContentPart for _, p := range m.Parts { - if p.Type == ContentPartImage { + if p.Type == ContentPartImage || p.Type == ContentPartImageURL { out = append(out, p) } } diff --git a/controls.go b/controls.go index e2f2d1f..747d069 100644 --- a/controls.go +++ b/controls.go @@ -20,6 +20,9 @@ var schemaNameRE = regexp.MustCompile(`^[A-Za-z0-9_-]{1,64}$`) const defaultSchemaName = "response" func validateRequestControls(req *ChatRequest) error { + if req.TopK < 0 { + return &ConfigError{Msg: "TopK must not be negative"} + } if tc := req.ToolChoice; tc != nil { if len(req.Tools) == 0 { return &ConfigError{Msg: "ToolChoice requires Tools"} @@ -189,15 +192,16 @@ func anthropicJSONToolName(req *ChatRequest) string { } // anthropicToolControls appends the JSON-mode tool when requested and -// renders tool_choice. thinking reports extended thinking, which rejects -// forced tool use. -func anthropicToolControls(req *ChatRequest, out *anRequest, thinking bool) error { +// renders tool_choice. thinking reports legacy budget thinking, which rejects +// forced tool use (adaptive thinking accepts it). anyThinking (either mode) +// guards the emulated JSON tool, matching the pre-M4 behaviour. +func anthropicToolControls(req *ChatRequest, out *anRequest, thinking, anyThinking, emulateJSON bool) error { var choice *anToolChoice - if name := anthropicJSONToolName(req); name != "" { + if name := anthropicJSONToolName(req); emulateJSON && name != "" { switch { case req.ToolChoice != nil: return &ConfigError{Msg: "Anthropic JSON ResponseFormat forces its own tool; it cannot be combined with ToolChoice"} - case thinking: + case anyThinking: return &ConfigError{Msg: "Anthropic JSON ResponseFormat forces tool use, which extended thinking rejects"} case len(req.Tools) > 0: // The forced JSON tool would make every user tool uncallable. @@ -227,7 +231,7 @@ func anthropicToolControls(req *ChatRequest, out *anRequest, thinking bool) erro choice = &anToolChoice{Type: "tool", Name: tc.Name} } if thinking && (choice.Type == "any" || choice.Type == "tool") { - return &ConfigError{Msg: "Anthropic extended thinking rejects forced tool use (ToolChoice required/tool)"} + return &ConfigError{Msg: "Anthropic extended thinking with ThinkingBudget rejects forced tool use (ToolChoice required/tool)"} } } if p := req.ParallelToolCalls; p != nil && !*p && len(out.Tools) > 0 && (choice == nil || choice.Type != "none") { diff --git a/count.go b/count.go new file mode 100644 index 0000000..28383cb --- /dev/null +++ b/count.go @@ -0,0 +1,131 @@ +package llm + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net/http" + "time" +) + +// ErrUnsupported marks an operation the provider format cannot perform. +var ErrUnsupported = errors.New("llm: operation not supported") + +// UnsupportedError reports an unsupported operation; it unwraps to +// ErrUnsupported. +type UnsupportedError struct{ Op, Provider, Reason string } + +func (e *UnsupportedError) Error() string { + return fmt.Sprintf("llm: %s unsupported on %s: %s", e.Op, e.Provider, e.Reason) +} + +func (e *UnsupportedError) Unwrap() error { return ErrUnsupported } + +// CountTokens reports the input token count of req using the provider's +// counting endpoint (Anthropic /v1/messages/count_tokens, Gemini +// :countTokens). OpenAI-format providers, including the Responses API, have +// none: *UnsupportedError (errors.Is ErrUnsupported), with no network call +// and never an estimate. Request validation matches Call; Extra and +// sampling parameters are ignored. Retries, the whole-call timeout, headers +// and the attempt observer (operation "count_tokens") behave as for chat. +func (c *ChatClient) CountTokens(ctx context.Context, req *ChatRequest) (int, error) { + return c.pc.countTokens(ctx, req, c.model) +} + +func (pc *providerClient) countTokens(ctx context.Context, req *ChatRequest, model string) (int, error) { + model, err := validateChatRequest(req, model) + if err != nil { + return 0, err + } + if pc.cfg.Format != FormatAnthropic && pc.cfg.Format != FormatGemini { + return 0, &UnsupportedError{Op: "CountTokens", Provider: pc.cfg.ID, Reason: "no token counting endpoint"} + } + ctx, herr := withReqHeaders(ctx, chatReqHeaders(req)) + if herr != nil { + return 0, herr + } + body, url, err := pc.buildCountRequest(req, model) + if err != nil { + return 0, err + } + ctx = pc.withAttempts(ctx, "count_tokens", model) + ctx, cancel := context.WithTimeout(ctx, pc.requestTimeout()) + defer cancel() + var n int + err = pc.withRetry(ctx, func() (time.Duration, error) { + data, _, ra, err := pc.post(ctx, pc.buffered(), url, body) + if err != nil { + return ra, err + } + v, perr := pc.parseCountResponse(data) + if perr != nil { + return 0, terminal(&APIError{Provider: pc.cfg.ID, Status: http.StatusOK, Message: perr.Error()}) + } + n = v + return 0, nil + }, nil) + if err != nil { + return 0, err + } + return n, nil +} + +func (pc *providerClient) buildCountRequest(req *ChatRequest, model string) ([]byte, string, error) { + if pc.cfg.Format == FormatAnthropic { + full, err := buildAnthropicRequestQ(req, model, false, pc.cfg.Quirks) + if err != nil { + return nil, "", err + } + b, err := pickKeys(full, "model", "messages", "system", "tools", "tool_choice", "thinking") + return b, pc.base + "/v1/messages/count_tokens", err + } + full, err := buildGeminiRequest(req, model, false) + if err != nil { + return nil, "", err + } + inner, err := pickKeys(full, "contents", "systemInstruction", "tools") + if err != nil { + return nil, "", err + } + var m map[string]json.RawMessage + if err := json.Unmarshal(inner, &m); err != nil { + return nil, "", err + } + m["model"], _ = json.Marshal("models/" + model) + b, err := json.Marshal(map[string]any{"generateContentRequest": m}) + return b, fmt.Sprintf("%s/v1beta/models/%s:countTokens", pc.base, geminiModelPath(model)), err +} + +// pickKeys keeps only the listed top-level keys of a JSON object. +func pickKeys(body []byte, keys ...string) ([]byte, error) { + var m map[string]json.RawMessage + if err := json.Unmarshal(body, &m); err != nil { + return nil, err + } + out := make(map[string]json.RawMessage, len(keys)) + for _, k := range keys { + if v, ok := m[k]; ok { + out[k] = v + } + } + return json.Marshal(out) +} + +func (pc *providerClient) parseCountResponse(data []byte) (int, error) { + var r struct { + InputTokens *int `json:"input_tokens"` + TotalTokens *int `json:"totalTokens"` + } + if err := json.Unmarshal(data, &r); err != nil { + return 0, fmt.Errorf("llm: parse token count: %w", err) + } + p := r.InputTokens + if pc.cfg.Format == FormatGemini { + p = r.TotalTokens + } + if p == nil || *p < 0 { + return 0, errors.New("llm: token count missing or invalid in response") + } + return *p, nil +} diff --git a/dispatch_edges_test.go b/dispatch_edges_test.go index 6bed8a9..0f9fde8 100644 --- a/dispatch_edges_test.go +++ b/dispatch_edges_test.go @@ -383,7 +383,7 @@ func TestModelsGetOversizedBody(t *testing.T) { func TestBufferedCallPostBuildError(t *testing.T) { pc := newProviderClient(ProviderConfig{ID: "x", Format: FormatOpenAI, BaseURL: "http://127.0.0.1:1", APIKey: "k"}, nil, nil) // NUL is rejected by net/url at request-build time, before any dial. - _, _, err := pc.post(context.Background(), pc.buffered(), "http://127.0.0.1:1/\x00chat", nil) + _, _, _, err := pc.post(context.Background(), pc.buffered(), "http://127.0.0.1:1/\x00chat", nil) var ce *ConfigError if !errors.As(err, &ce) { t.Fatalf("err = %v (%T), want request-build ConfigError", err, err) @@ -695,7 +695,7 @@ func TestBuildAnthropicRequestArmsTail(t *testing.T) { {Role: RoleUser, Content: "q"}, {Role: RoleAssistant, ToolCalls: []ToolCall{{ID: "tu_1", Name: "f"}}}, // empty args → {} }, - Thinking: "enabled", // no explicit budget → 5000 default + Thinking: "enabled", ThinkingBudget: 5000, // explicit budget: presets are adaptive now } b, err := buildAnthropicRequest(req, "claude-x", false) if err != nil { diff --git a/embed.go b/embed.go index a78b6d8..eefc075 100644 --- a/embed.go +++ b/embed.go @@ -1,11 +1,14 @@ package llm import ( + "bytes" "context" + "encoding/base64" + "encoding/binary" "encoding/json" "fmt" + "math" "net/http" - neturl "net/url" "sort" "strings" "time" @@ -22,8 +25,10 @@ import ( // EmbedRequest describes one embeddings call. type EmbedRequest struct { - Inputs []string // required: one or more non-empty texts - Dimensions int // optional output dimensionality (0 = model default) + Headers map[string]string // per-request headers (SL03) + Inputs []string // required: one or more non-empty texts + Dimensions int // optional output dimensionality (0 = model default) + TaskType string // Gemini taskType, e.g. RETRIEVAL_QUERY (OpenAI-format ignores it) } // EmbedResult carries one vector per input, in input order. @@ -31,6 +36,7 @@ type EmbedResult struct { Embeddings [][]float64 Model string Usage Usage // PromptTokens when the provider reports it + Meta ResponseMeta } // Embed computes embeddings for req.Inputs with the named provider and @@ -69,8 +75,8 @@ type oaEmbedRequest struct { type oaEmbedResponse struct { Data []struct { - Index int `json:"index"` - Embedding []float64 `json:"embedding"` + Index int `json:"index"` + Embedding json.RawMessage `json:"embedding"` } `json:"data"` Usage *struct { PromptTokens int `json:"prompt_tokens"` @@ -81,6 +87,7 @@ type gmEmbedItem struct { Model string `json:"model"` Content gmContent `json:"content"` OutputDimensionality int `json:"outputDimensionality,omitempty"` + TaskType string `json:"taskType,omitempty"` } type gmEmbedRequest struct { @@ -102,12 +109,13 @@ func (pc *providerClient) buildEmbedRequest(model string, req EmbedRequest) ([]b Model: "models/" + model, Content: gmContent{Parts: []gmPart{{Text: in}}}, OutputDimensionality: req.Dimensions, + TaskType: strings.TrimSpace(req.TaskType), }) } b, err := json.Marshal(r) - return b, fmt.Sprintf("%s/v1beta/models/%s:batchEmbedContents", pc.base, neturl.PathEscape(model)), err + return b, fmt.Sprintf("%s/v1beta/models/%s:batchEmbedContents", pc.base, geminiModelPath(model)), err } - b, err := json.Marshal(oaEmbedRequest{Model: model, Input: req.Inputs, Dimensions: req.Dimensions, EncodingFormat: "float"}) + b, err := json.Marshal(oaEmbedRequest{Model: model, Input: req.Inputs, Dimensions: req.Dimensions, EncodingFormat: "base64"}) return b, pc.base + "/embeddings", err } @@ -153,7 +161,11 @@ func (pc *providerClient) parseEmbedResponse(data []byte, n int) ([][]float64, U if d.Index != i { return nil, Usage{}, fmt.Errorf("llm: embeddings: vector index %d out of range", d.Index) } - out[i] = d.Embedding + v, err := decodeOAEmbedding(d.Embedding) + if err != nil { + return nil, Usage{}, err + } + out[i] = v } var u Usage if r.Usage != nil { @@ -162,6 +174,36 @@ func (pc *providerClient) parseEmbedResponse(data []byte, n int) ([][]float64, U return out, u, nil } +// decodeOAEmbedding decodes one OpenAI-compatible embedding: a base64 string +// of little-endian float32 (encoding_format=base64) or a JSON float array +// (providers that ignore the parameter). +func decodeOAEmbedding(raw json.RawMessage) ([]float64, error) { + raw = bytes.TrimSpace(raw) + if len(raw) > 0 && raw[0] == '"' { + var s string + if err := json.Unmarshal(raw, &s); err != nil { + return nil, fmt.Errorf("llm: parse embeddings: %w", err) + } + b, err := base64.StdEncoding.DecodeString(s) + if err != nil { + return nil, fmt.Errorf("llm: embeddings: invalid base64 vector: %w", err) + } + if len(b)%4 != 0 { + return nil, fmt.Errorf("llm: embeddings: base64 vector is %d bytes, not a multiple of 4", len(b)) + } + v := make([]float64, len(b)/4) + for i := range v { + v[i] = float64(math.Float32frombits(binary.LittleEndian.Uint32(b[4*i:]))) + } + return v, nil + } + var v []float64 + if err := json.Unmarshal(raw, &v); err != nil { + return nil, fmt.Errorf("llm: parse embeddings: %w", err) + } + return v, nil +} + // Per-request input caps: Gemini batchEmbedContents takes at most 100 // requests, OpenAI /embeddings at most 2048 inputs. Larger calls are split // into consecutive batches and concatenated in input order. @@ -173,11 +215,16 @@ const ( // embed runs the request in provider-sized batches, each with the shared // retry ladder, all under one whole-call budget. func (pc *providerClient) embed(ctx context.Context, model string, req EmbedRequest) (*EmbedResult, error) { + ctx, herr := withReqHeaders(ctx, req.Headers) + if herr != nil { + return nil, herr + } model = strings.TrimPrefix(model, "models/") // Gemini ids as ListModels may echo them batch := openAIEmbedBatch if pc.cfg.Format == FormatGemini { batch = geminiEmbedBatch } + ctx = pc.withAttempts(ctx, "embed", model) ctx, cancel := context.WithTimeout(ctx, pc.requestTimeout()) defer cancel() res := &EmbedResult{Model: model, Embeddings: make([][]float64, 0, len(req.Inputs))} @@ -189,7 +236,7 @@ func (pc *providerClient) embed(ctx context.Context, model string, req EmbedRequ return nil, err } err = pc.withRetry(ctx, func() (time.Duration, error) { - data, ra, err := pc.post(ctx, pc.buffered(), url, body) + data, hdr, ra, err := pc.post(ctx, pc.buffered(), url, body) if err != nil { return ra, err } @@ -201,6 +248,7 @@ func (pc *providerClient) embed(ctx context.Context, model string, req EmbedRequ } res.Embeddings = append(res.Embeddings, vecs...) res.Usage.PromptTokens += usage.PromptTokens + res.Meta = parseResponseMeta(hdr) return 0, nil }, nil) if err != nil { diff --git a/followups_test.go b/followups_test.go index 41b542c..9f95ee5 100644 --- a/followups_test.go +++ b/followups_test.go @@ -232,7 +232,7 @@ func TestEmbed_OpenAI(t *testing.T) { if res.Model != "emb" || res.Usage.PromptTokens != 5 { t.Errorf("result = %+v", res) } - if got["model"] != "emb" || got["dimensions"] != float64(2) || got["encoding_format"] != "float" { + if got["model"] != "emb" || got["dimensions"] != float64(2) || got["encoding_format"] != "base64" { t.Errorf("request = %v", got) } } diff --git a/gemini.go b/gemini.go index 6adebd7..36f10af 100644 --- a/gemini.go +++ b/gemini.go @@ -30,11 +30,16 @@ type gmFnResp struct { type gmPart struct { Text string `json:"text,omitempty"` InlineData *gmBlob `json:"inlineData,omitempty"` + FileData *gmFile `json:"fileData,omitempty"` Thought bool `json:"thought,omitempty"` FunctionCall *gmFnCall `json:"functionCall,omitempty"` FunctionResponse *gmFnResp `json:"functionResponse,omitempty"` ThoughtSignature string `json:"thoughtSignature,omitempty"` } +type gmFile struct { + FileURI string `json:"fileUri"` + MIMEType string `json:"mimeType"` +} type gmBlob struct { MIMEType string `json:"mimeType"` Data string `json:"data"` @@ -48,7 +53,7 @@ type gmContent struct { type gmFnDecl struct { Name string `json:"name"` Description string `json:"description,omitempty"` - Parameters json.RawMessage `json:"parameters,omitempty"` + Parameters json.RawMessage `json:"parametersJsonSchema,omitempty"` // JSON Schema, not the restricted OpenAPI `parameters` } type gmToolGroup struct { @@ -64,6 +69,7 @@ type gmGenCfg struct { MaxOutputTokens int `json:"maxOutputTokens,omitempty"` Temperature *float64 `json:"temperature,omitempty"` TopP *float64 `json:"topP,omitempty"` + TopK *int `json:"topK,omitempty"` StopSequences []string `json:"stopSequences,omitempty"` ThinkingConfig *gmThinkCfg `json:"thinkingConfig,omitempty"` Seed *int `json:"seed,omitempty"` @@ -128,6 +134,13 @@ func mustJSONString(s string) string { return string(b) } +// geminiModelPath returns the model as a single safe URL path segment: a +// leading "models/" (as ListModels echoes ids) is stripped and the rest is +// escaped so an id can never rewrite the path or smuggle a query. +func geminiModelPath(model string) string { + return neturl.PathEscape(strings.TrimPrefix(model, "models/")) +} + // buildGeminiRequest renders the canonical request in Gemini format. func buildGeminiRequest(req *ChatRequest, model string, stream bool) ([]byte, error) { _ = model // rides the URL path @@ -138,10 +151,12 @@ func buildGeminiRequest(req *ChatRequest, model string, stream bool) ([]byte, er // System: canonical blocks + in-band system messages → systemInstruction. var sysParts []gmPart for _, b := range req.System { - sysParts = append(sysParts, gmPart{Text: b.Text}) + if b.Text != "" { // an empty part is rejected by Gemini + sysParts = append(sysParts, gmPart{Text: b.Text}) + } } for _, m := range req.Messages { - if m.Role == RoleSystem { + if m.Role == RoleSystem && m.Content != "" { sysParts = append(sysParts, gmPart{Text: m.Content}) } } @@ -158,14 +173,21 @@ func buildGeminiRequest(req *ChatRequest, model string, stream bool) ([]byte, er case RoleUser: parts := make([]gmPart, 0, maxInt(1, len(m.Parts))) if len(m.Parts) == 0 { + if m.Content == "" { + return nil, &ConfigError{Msg: fmt.Sprintf("message %d: user message has no content", i)} + } parts = append(parts, gmPart{Text: m.Content}) } else { for _, p := range m.Parts { - if p.Type == ContentPartImage { - parts = append(parts, gmPart{InlineData: &gmBlob{MIMEType: wireMIME(p.MIMEType), Data: base64.StdEncoding.EncodeToString(p.Image)}}) - } else { + if p.Type == ContentPartText { parts = append(parts, gmPart{Text: p.Text}) + continue + } + gp, err := geminiAttachment(p) + if err != nil { + return nil, err } + parts = append(parts, gp) } } out.Contents = append(out.Contents, gmContent{Role: "user", Parts: parts}) @@ -218,8 +240,12 @@ func buildGeminiRequest(req *ChatRequest, model string, stream bool) ([]byte, er FunctionResponse: &gmFnResp{Name: name, Response: resp}, }) // Images from the tool ride the same user turn as inlineData. - for _, p := range imageParts(tm) { - images = append(images, gmPart{InlineData: &gmBlob{MIMEType: wireMIME(p.MIMEType), Data: base64.StdEncoding.EncodeToString(p.Image)}}) + for _, p := range attachmentParts(tm) { + gp, err := geminiAttachment(p) + if err != nil { + return nil, err + } + images = append(images, gp) } } parts = append(parts, images...) @@ -256,6 +282,10 @@ func buildGeminiRequest(req *ChatRequest, model string, stream bool) ([]byte, er } cfg.TopP = &p } + if req.TopK > 0 { + k := req.TopK + cfg.TopK = &k + } if tc := geminiThinkingConfig(req.Thinking, req.ThinkingBudget); tc != nil { cfg.ThinkingConfig = tc } @@ -270,7 +300,7 @@ func buildGeminiRequest(req *ChatRequest, model string, stream bool) ([]byte, er } } out.ToolConfig = geminiToolConfig(req.ToolChoice) - if cfg.MaxOutputTokens != 0 || cfg.Temperature != nil || cfg.TopP != nil || + if cfg.MaxOutputTokens != 0 || cfg.Temperature != nil || cfg.TopP != nil || cfg.TopK != nil || len(cfg.StopSequences) > 0 || cfg.ThinkingConfig != nil || cfg.Seed != nil || cfg.ResponseMimeType != "" { out.GenerationConfig = &cfg @@ -313,6 +343,8 @@ type gmPromptFeedback struct { } type gmResponse struct { + ResponseID string `json:"responseId"` + ModelVersion string `json:"modelVersion"` Candidates []gmCandidate `json:"candidates"` UsageMetadata gmUsage `json:"usageMetadata"` PromptFeedback *gmPromptFeedback `json:"promptFeedback"` @@ -325,7 +357,8 @@ func mapGeminiFinishReason(s string) string { return FinishStop case "MAX_TOKENS": return FinishLength - case "SAFETY", "RECITATION", "BLOCKLIST", "PROHIBITED_CONTENT", "SPII": + case "SAFETY", "RECITATION", "BLOCKLIST", "PROHIBITED_CONTENT", "SPII", + "IMAGE_SAFETY", "IMAGE_PROHIBITED_CONTENT", "IMAGE_RECITATION": return FinishContentFilter case "": return "" @@ -419,13 +452,14 @@ func parseGeminiResponse(body []byte) (*ChatResult, error) { if len(r.Candidates) == 0 { if r.PromptFeedback != nil && r.PromptFeedback.BlockReason != "" { // A blocked prompt is a completed, filtered turn. - return &ChatResult{FinishReason: FinishContentFilter, Usage: mapGeminiUsage(r.UsageMetadata)}, nil + return &ChatResult{FinishReason: FinishContentFilter, Usage: mapGeminiUsage(r.UsageMetadata), Meta: ResponseMeta{ID: r.ResponseID, Model: r.ModelVersion}}, nil } return nil, fmt.Errorf("llm: response has no candidates") } acc := newStreamAccum() c := r.Candidates[0] foldGeminiParts(c.Content.Parts, acc) + acc.noteMeta(r.ResponseID, r.ModelVersion) res := acc.result() res.FinishReason = geminiFinish(c.FinishReason, len(res.ToolCalls) > 0) res.Usage = mapGeminiUsage(r.UsageMetadata) @@ -435,9 +469,12 @@ func parseGeminiResponse(body []byte) (*ChatResult, error) { // ── streaming ──────────────────────────────────────────────────────────── type gmStreamChunk struct { + ResponseID string `json:"responseId"` + ModelVersion string `json:"modelVersion"` Candidates []gmCandidate `json:"candidates"` PromptFeedback *gmPromptFeedback `json:"promptFeedback"` UsageMetadata *gmUsage `json:"usageMetadata"` // nil = chunk carries no usage update + Error json.RawMessage `json:"error"` // see streamErrorMessage } // mapGeminiStreamEvent folds one Gemini SSE chunk into acc. Chunks carry @@ -448,6 +485,10 @@ func mapGeminiStreamEvent(data []byte, acc *streamAccum) ([]Delta, bool, error) if err := json.Unmarshal(data, &c); err != nil { return nil, false, fmt.Errorf("llm: parse stream chunk: %w", err) } + if msg, ok := streamErrorMessage(c.Error); ok { + return nil, false, fmt.Errorf("llm: provider error: %s", msg) + } + acc.noteMeta(c.ResponseID, c.ModelVersion) if c.UsageMetadata != nil { acc.usage = mapGeminiUsage(*c.UsageMetadata) } @@ -518,3 +559,17 @@ func listModelsGemini(ctx context.Context, pc *providerClient) ([]Model, error) } return nil, fmt.Errorf("%w (%d pages)", ErrModelListTruncated, maxModelPages) } + +// geminiAttachment renders an image, image-URL or document part. +func geminiAttachment(p ContentPart) (gmPart, error) { + switch p.Type { + case ContentPartImageURL: + if p.MIMEType == "" { + return gmPart{}, &ConfigError{Msg: "Gemini image URL parts require a MIME type (use ImageURLPartMIME or WithMIME)"} + } + return gmPart{FileData: &gmFile{FileURI: p.URL, MIMEType: canonMIME(p.MIMEType)}}, nil + case ContentPartDocument: + return gmPart{InlineData: &gmBlob{MIMEType: canonMIME(p.MIMEType), Data: base64.StdEncoding.EncodeToString(p.Data)}}, nil + } + return gmPart{InlineData: &gmBlob{MIMEType: wireMIME(p.MIMEType), Data: base64.StdEncoding.EncodeToString(p.Image)}}, nil +} diff --git a/golden_wire_test.go b/golden_wire_test.go new file mode 100644 index 0000000..8021977 --- /dev/null +++ b/golden_wire_test.go @@ -0,0 +1,87 @@ +package llm + +// Golden wire bodies recorded from the pre-change code (feature m6b, AC-7): +// a request that uses none of the new fields (DocumentPart, ImageURLPart, +// TopK, Gemini generationConfig in Extra, EmbedRequest.TaskType) must keep +// marshalling to these exact bytes on every format. These tests PASS before +// any production change and must keep passing. + +import ( + "encoding/json" + "testing" +) + +func goldenReq() *ChatRequest { + return &ChatRequest{ + System: []SystemBlock{{Text: "be brief"}}, + Messages: []Message{ + {Role: RoleUser, Content: "weather in Paris?"}, + {Role: RoleAssistant, Content: "checking", ToolCalls: []ToolCall{{ID: "c1", Name: "wx", Arguments: `{"city":"Paris"}`}}}, + {Role: RoleTool, ToolCallID: "c1", ToolName: "wx", Content: "sunny"}, + {Role: RoleUser, Parts: []ContentPart{TextPart("and"), ImagePart("image/png", []byte{1, 2, 3}), TextPart("London?")}}, + }, + Tools: []ToolDef{{Name: "wx", Description: "weather", Parameters: json.RawMessage(`{"type":"object","properties":{"city":{"type":"string"}}}`)}}, + MaxTokens: 256, + Temperature: 0.5, + TopP: 0.9, + Stop: []string{"END"}, + } +} + +func goldenToolImgReq() *ChatRequest { + r := goldenReq() + r.Messages = []Message{ + {Role: RoleUser, Content: "look"}, + {Role: RoleAssistant, ToolCalls: []ToolCall{{ID: "c2", Name: "wx", Arguments: `{}`}}}, + {Role: RoleTool, ToolCallID: "c2", ToolName: "wx", Parts: []ContentPart{TextPart("chart"), ImagePart("image/png", []byte{9})}}, + } + return r +} + +var goldenWires = []struct { + name string + format Format + model string + req func() *ChatRequest + url string + body string +}{ + {"openai", FormatOpenAI, "gpt-4o", goldenReq, "https://x/chat/completions", `{"model":"gpt-4o","messages":[{"role":"system","content":"be brief"},{"role":"user","content":"weather in Paris?"},{"role":"assistant","content":"checking","tool_calls":[{"id":"c1","type":"function","function":{"name":"wx","arguments":"{\"city\":\"Paris\"}"}}]},{"role":"tool","content":"sunny","tool_call_id":"c1"},{"role":"user","content":[{"type":"text","text":"and"},{"type":"image_url","image_url":{"url":"data:image/png;base64,AQID"}},{"type":"text","text":"London?"}]}],"tools":[{"type":"function","function":{"name":"wx","description":"weather","parameters":{"type":"object","properties":{"city":{"type":"string"}}}}}],"max_tokens":256,"temperature":0.5,"top_p":0.9,"stop":["END"]}`}, + {"toolimg-openai", FormatOpenAI, "gpt-4o", goldenToolImgReq, "https://x/chat/completions", `{"model":"gpt-4o","messages":[{"role":"system","content":"be brief"},{"role":"user","content":"look"},{"role":"assistant","content":"","tool_calls":[{"id":"c2","type":"function","function":{"name":"wx","arguments":"{}"}}]},{"role":"tool","content":"chart","tool_call_id":"c2"},{"role":"user","content":[{"type":"text","text":"Images returned by tool call c2:"},{"type":"image_url","image_url":{"url":"data:image/png;base64,CQ=="}}]}],"tools":[{"type":"function","function":{"name":"wx","description":"weather","parameters":{"type":"object","properties":{"city":{"type":"string"}}}}}],"max_tokens":256,"temperature":0.5,"top_p":0.9,"stop":["END"]}`}, + {"responses", FormatOpenAI, "gpt-5.6", goldenReq, "https://x/responses", `{"model":"gpt-5.6","instructions":"be brief","input":[{"role":"user","content":"weather in Paris?"},{"type":"function_call","call_id":"c1","name":"wx","arguments":"{\"city\":\"Paris\"}"},{"role":"assistant","content":"checking"},{"type":"function_call_output","call_id":"c1","output":"sunny"},{"role":"user","content":[{"type":"input_text","text":"and"},{"type":"input_image","image_url":"data:image/png;base64,AQID"},{"type":"input_text","text":"London?"}]}],"tools":[{"type":"function","name":"wx","description":"weather","parameters":{"type":"object","properties":{"city":{"type":"string"}}}}],"max_output_tokens":256,"store":false,"include":["reasoning.encrypted_content"],"reasoning":{"effort":"medium","summary":"auto"}}`}, + {"toolimg-responses", FormatOpenAI, "gpt-5.6", goldenToolImgReq, "https://x/responses", `{"model":"gpt-5.6","instructions":"be brief","input":[{"role":"user","content":"look"},{"type":"function_call","call_id":"c2","name":"wx","arguments":"{}"},{"type":"function_call_output","call_id":"c2","output":[{"type":"input_text","text":"chart"},{"type":"input_image","image_url":"data:image/png;base64,CQ=="}]}],"tools":[{"type":"function","name":"wx","description":"weather","parameters":{"type":"object","properties":{"city":{"type":"string"}}}}],"max_output_tokens":256,"store":false,"include":["reasoning.encrypted_content"],"reasoning":{"effort":"medium","summary":"auto"}}`}, + {"anthropic", FormatAnthropic, "claude", goldenReq, "https://x/v1/messages", `{"model":"claude","max_tokens":256,"messages":[{"role":"user","content":[{"type":"text","text":"weather in Paris?"}]},{"role":"assistant","content":[{"type":"text","text":"checking"},{"type":"tool_use","id":"c1","name":"wx","input":{"city":"Paris"}}]},{"role":"user","content":[{"type":"tool_result","tool_use_id":"c1","content":"sunny"}]},{"role":"user","content":[{"type":"text","text":"and"},{"type":"image","source":{"type":"base64","media_type":"image/png","data":"AQID"}},{"type":"text","text":"London?"}]}],"system":[{"type":"text","text":"be brief"}],"tools":[{"name":"wx","description":"weather","input_schema":{"type":"object","properties":{"city":{"type":"string"}}}}],"temperature":0.5,"top_p":0.9,"stop_sequences":["END"]}`}, + {"toolimg-anthropic", FormatAnthropic, "claude", goldenToolImgReq, "https://x/v1/messages", `{"model":"claude","max_tokens":256,"messages":[{"role":"user","content":[{"type":"text","text":"look"}]},{"role":"assistant","content":[{"type":"tool_use","id":"c2","name":"wx","input":{}}]},{"role":"user","content":[{"type":"tool_result","tool_use_id":"c2","content":[{"type":"text","text":"chart"},{"type":"image","source":{"type":"base64","media_type":"image/png","data":"CQ=="}}]}]}],"system":[{"type":"text","text":"be brief"}],"tools":[{"name":"wx","description":"weather","input_schema":{"type":"object","properties":{"city":{"type":"string"}}}}],"temperature":0.5,"top_p":0.9,"stop_sequences":["END"]}`}, + {"gemini", FormatGemini, "gemini", goldenReq, "https://x/v1beta/models/gemini:generateContent", `{"systemInstruction":{"parts":[{"text":"be brief"}]},"contents":[{"role":"user","parts":[{"text":"weather in Paris?"}]},{"role":"model","parts":[{"text":"checking"},{"functionCall":{"name":"wx","args":{"city":"Paris"}}}]},{"role":"user","parts":[{"functionResponse":{"name":"wx","response":{"result":"sunny"}}}]},{"role":"user","parts":[{"text":"and"},{"inlineData":{"mimeType":"image/png","data":"AQID"}},{"text":"London?"}]}],"tools":[{"functionDeclarations":[{"name":"wx","description":"weather","parametersJsonSchema":{"type":"object","properties":{"city":{"type":"string"}}}}]}],"generationConfig":{"maxOutputTokens":256,"temperature":0.5,"topP":0.9,"stopSequences":["END"]}}`}, + {"toolimg-gemini", FormatGemini, "gemini", goldenToolImgReq, "https://x/v1beta/models/gemini:generateContent", `{"systemInstruction":{"parts":[{"text":"be brief"}]},"contents":[{"role":"user","parts":[{"text":"look"}]},{"role":"model","parts":[{"functionCall":{"name":"wx","args":{}}}]},{"role":"user","parts":[{"functionResponse":{"name":"wx","response":{"result":"chart"}}},{"inlineData":{"mimeType":"image/png","data":"CQ=="}}]}],"tools":[{"functionDeclarations":[{"name":"wx","description":"weather","parametersJsonSchema":{"type":"object","properties":{"city":{"type":"string"}}}}]}],"generationConfig":{"maxOutputTokens":256,"temperature":0.5,"topP":0.9,"stopSequences":["END"]}}`}, +} + +func TestGoldenWireBodiesUnchanged(t *testing.T) { + for _, c := range goldenWires { + t.Run(c.name, func(t *testing.T) { + pc := newProviderClient(ProviderConfig{ID: string(c.format), Format: c.format, BaseURL: "https://x"}, nil, nil) + body, url, err := pc.buildChatRequest(c.req(), c.model, false) + if err != nil { + t.Fatal(err) + } + if url != c.url { + t.Errorf("url = %s, want %s", url, c.url) + } + if string(body) != c.body { + t.Errorf("wire drifted\n got: %s\nwant: %s", body, c.body) + } + }) + } +} + +func TestGoldenGeminiEmbedBodyUnchanged(t *testing.T) { + pc := newProviderClient(ProviderConfig{ID: "gemini", Format: FormatGemini, BaseURL: "https://x"}, nil, nil) + body, _, err := pc.buildEmbedRequest("text-embedding-004", EmbedRequest{Inputs: []string{"a", "b"}, Dimensions: 8}) + if err != nil { + t.Fatal(err) + } + want := `{"requests":[{"model":"models/text-embedding-004","content":{"parts":[{"text":"a"}]},"outputDimensionality":8},{"model":"models/text-embedding-004","content":{"parts":[{"text":"b"}]},"outputDimensionality":8}]}` + if string(body) != want { + t.Errorf("embed wire drifted\n got: %s\nwant: %s", body, want) + } +} diff --git a/llm.go b/llm.go index 2ed1a02..ec501c6 100644 --- a/llm.go +++ b/llm.go @@ -17,20 +17,31 @@ import ( // environment or is set explicitly; every authenticated provider is usable // concurrently through the SDK's shared connection pool. type SDK struct { - mu sync.RWMutex - providers map[string]*Provider - order []string - timeout time.Duration - cacheTTL time.Duration - rt http.RoundTripper - retry RetryPolicy - idle time.Duration - observer func(LearnEvent) + mu sync.RWMutex + providers map[string]*Provider + order []string + timeout time.Duration + cacheTTL time.Duration + rt http.RoundTripper + retry RetryPolicy + idle time.Duration + observer func(LearnEvent) + attemptObs func(AttemptEvent) } // clientOpts snapshots the SDK-level knobs every provider client honors. func (s *SDK) clientOpts() clientOpts { - return clientOpts{retry: s.retry, idle: s.idle, observer: s.observer} + return clientOpts{retry: s.retry, idle: s.idle, observer: s.observer, attempt: s.attemptObs} +} + +// listTimeout is the whole-call budget for model discovery: the configured +// request timeout, or 30s (the long-standing listing default) when the +// request timeout is left at its chat default. +func (s *SDK) listTimeout() time.Duration { + if s.timeout == DefaultTimeout { + return 30 * time.Second + } + return s.timeout } // newClient builds a provider client for p with the SDK's settings. @@ -277,6 +288,7 @@ type Provider struct { mu sync.Mutex cached []Model + cachedOK bool // cached holds a successful listing (possibly empty) cachedAt time.Time flight *modelFlight // in-progress listing shared by concurrent callers } @@ -320,9 +332,8 @@ func (p *Provider) ListModels(ctx context.Context, opts ...ListOption) ([]Model, ttl := p.sdk.cacheTTL if !lo.force && ttl > 0 { p.mu.Lock() - if p.cached != nil && time.Since(p.cachedAt) < ttl { - out := make([]Model, len(p.cached)) - copy(out, p.cached) + if p.cachedOK && time.Since(p.cachedAt) < ttl { + out := cloneModels(p.cached) p.mu.Unlock() return out, nil } @@ -332,15 +343,13 @@ func (p *Provider) ListModels(ctx context.Context, opts ...ListOption) ([]Model, return nil, &ConfigError{Msg: p.cfg.ID + " has no API key"} } p.clientOnce.Do(func() { - p.listClient = p.sdk.newClient(p, 30*time.Second, nil) + p.listClient = p.sdk.newClient(p, p.sdk.listTimeout(), nil) }) models, err := p.fetchModels(ctx) if err != nil { return nil, err } - out := make([]Model, len(models)) - copy(out, models) - return out, nil + return cloneModels(models), nil } // modelFlight is one in-progress upstream model listing that concurrent @@ -392,7 +401,7 @@ func (p *Provider) lead(ctx context.Context, f *modelFlight) { p.mu.Lock() p.flight = nil if f.err == nil { - p.cached, p.cachedAt = f.models, time.Now() + p.cached, p.cachedOK, p.cachedAt = f.models, true, time.Now() } p.mu.Unlock() close(f.done) diff --git a/m1_errorfield_test.go b/m1_errorfield_test.go new file mode 100644 index 0000000..9241ff7 --- /dev/null +++ b/m1_errorfield_test.go @@ -0,0 +1,67 @@ +package llm + +import ( + "context" + "strings" + "testing" +) + +// A non-object "error" value on a stream chunk is not a provider error and +// must not fail an otherwise-successful stream; an error object with a +// message still does. +func TestM1_StreamErrorFieldNonObjectIgnored(t *testing.T) { + formats := []struct { + name string + format Format + chunk func(errField string) string // one content chunk carrying the error field + finish string + want string + }{ + { + "openai", FormatOpenAI, + func(e string) string { + return "data: {\"error\":" + e + ",\"choices\":[{\"delta\":{\"content\":\"hello\"}}]}\n\n" + }, + m1Finish + "data: [DONE]\n\n", "hello", + }, + { + "gemini", FormatGemini, + func(e string) string { + return "data: {\"error\":" + e + ",\"candidates\":[{\"content\":{\"parts\":[{\"text\":\"hello\"}]}}]}\n\n" + }, + "data: {\"candidates\":[{\"content\":{\"parts\":[]},\"finishReason\":\"STOP\"}]}\n\n", "hello", + }, + } + values := []struct { + errField string + wantErr bool + }{ + {`""`, false}, + {`false`, false}, + {`null`, false}, + {`0`, false}, + {`{"message":"upstream overloaded"}`, true}, + } + for _, f := range formats { + for _, v := range values { + t.Run(f.name+"/"+v.errField, func(t *testing.T) { + fastBackoff(t) + srv := m1StreamServer(t, f.chunk(v.errField)+f.finish) + cc := newTestClient(t, ProviderConfig{ID: f.name, Format: f.format, BaseURL: srv.URL, APIKey: "k"}, srv) + res, err := cc.CallStream(context.Background(), m1Req(), func(Delta) error { return nil }) + if v.wantErr { + if err == nil || !strings.Contains(err.Error(), "upstream overloaded") { + t.Fatalf("error object must fail the stream with its message, got res=%+v err=%v", res, err) + } + return + } + if err != nil { + t.Fatalf("non-object error value %s failed the stream: %v", v.errField, err) + } + if res.Content != f.want { + t.Fatalf("content = %q, want %q", res.Content, f.want) + } + }) + } + } +} diff --git a/m1_retry_test.go b/m1_retry_test.go new file mode 100644 index 0000000..0f7341c --- /dev/null +++ b/m1_retry_test.go @@ -0,0 +1,147 @@ +package llm + +import ( + "context" + "errors" + "net" + "net/http" + "net/http/httptest" + "net/url" + "sync/atomic" + "testing" + "time" +) + +// Bug 4: a huge Retry-After must clamp, never overflow negative. +func TestM1_ParseRetryAfterOverflowClamps(t *testing.T) { + for _, v := range []string{"9223372037", "99999999999", "9223372036854775807"} { + got := parseRetryAfter(v, time.Now()) + if got != maxRetryAfter { + t.Errorf("parseRetryAfter(%q) = %v, want %v", v, got, maxRetryAfter) + } + } +} + +// Bug 5: oversized responses are terminal; they must not be retried (each +// retry is a new billed generation). +func TestM1_OversizedResponseNotRetried(t *testing.T) { + cases := []struct { + name string + size int + call func(s *SDK) error + }{ + {"chat", maxResponseSize + 1, func(s *SDK) error { + cc, err := s.Chat("openai", "m") + if err != nil { + return err + } + _, err = cc.Call(context.Background(), m1Req()) + return err + }}, + {"speak", maxResponseSize + 1, func(s *SDK) error { + _, err := s.Speak(context.Background(), "openai", "tts-1", SpeakRequest{Text: "hi", Voice: "alloy"}) + return err + }}, + {"transcribe", maxResponseSize + 1, func(s *SDK) error { + _, err := s.Transcribe(context.Background(), "openai", "whisper-1", TranscribeRequest{Audio: []byte("abc")}) + return err + }}, + {"models", maxModelsResponseSize + 1, func(s *SDK) error { + p, err := s.Provider("openai") + if err != nil { + return err + } + _, err = p.ListModels(context.Background()) + return err + }}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + fastBackoff(t) + var requests atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + requests.Add(1) + buf := make([]byte, 1<<20) + for n := 0; n < tc.size; n += len(buf) { + if _, err := w.Write(buf); err != nil { + return + } + } + })) + defer srv.Close() + s := New(WithRetryPolicy(RetryPolicy{MaxAttempts: 3})) + s.put(ProviderConfig{ID: "openai", Format: FormatOpenAI, BaseURL: srv.URL, APIKey: "k"}) + if err := tc.call(s); err == nil { + t.Fatal("expected an oversize error") + } + if n := requests.Load(); n != 1 { + t.Fatalf("oversized response retried: %d requests, want 1", n) + } + }) + } +} + +// Bug 6: when the wait would outlive the context deadline, return the last +// provider/transport error promptly instead of sleeping to the deadline and +// returning a bare context error. +func TestM1_BackoffBeyondDeadlineKeepsLastError(t *testing.T) { + cases := []struct { + name string + handler http.HandlerFunc + check func(t *testing.T, err error) + }{ + {"503 with Retry-After", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Retry-After", "60") + w.WriteHeader(http.StatusServiceUnavailable) + }, func(t *testing.T, err error) { + var ae *APIError + if !errors.As(err, &ae) || ae.Status != http.StatusServiceUnavailable { + t.Fatalf("last APIError lost: %v", err) + } + }}, + {"500 with long backoff", func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusInternalServerError) + }, func(t *testing.T, err error) { + var ae *APIError + if !errors.As(err, &ae) || ae.Status != http.StatusInternalServerError { + t.Fatalf("last APIError lost: %v", err) + } + }}, + {"transport error", func(w http.ResponseWriter, r *http.Request) { + c, _, err := w.(http.Hijacker).Hijack() + if err == nil { + _ = c.Close() + } + }, func(t *testing.T, err error) { + var ue *url.Error + var ne net.Error + if !errors.As(err, &ue) && !errors.As(err, &ne) { + t.Fatalf("last transport error lost: %v", err) + } + }}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + old := backoffUnit + backoffUnit = time.Minute + t.Cleanup(func() { backoffUnit = old }) + srv := httptest.NewServer(tc.handler) + defer srv.Close() + cc := newTestClient(t, ProviderConfig{ID: "openai", Format: FormatOpenAI, BaseURL: srv.URL, APIKey: "k"}, srv) + backoffUnit = time.Minute // newTestClient shrinks it; restore the long unit + + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) + defer cancel() + start := time.Now() + _, err := cc.Call(ctx, m1Req()) + elapsed := time.Since(start) + if err == nil { + t.Fatal("expected an error") + } + if elapsed > time.Second { + t.Errorf("slept past the budget: returned after %v (deadline 2s)", elapsed) + } + tc.check(t, err) + }) + } +} diff --git a/m1_stream_test.go b/m1_stream_test.go new file mode 100644 index 0000000..e0e5694 --- /dev/null +++ b/m1_stream_test.go @@ -0,0 +1,125 @@ +package llm + +import ( + "context" + "fmt" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" +) + +func m1StreamServer(t *testing.T, body string) *httptest.Server { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/event-stream") + _, _ = fmt.Fprint(w, body) + })) + t.Cleanup(srv.Close) + return srv +} + +func m1OpenAIClient(t *testing.T, srv *httptest.Server) *ChatClient { + t.Helper() + return newTestClient(t, ProviderConfig{ID: "openai", Format: FormatOpenAI, BaseURL: srv.URL, APIKey: "k"}, srv) +} + +func m1Req() *ChatRequest { + return &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "hi"}}} +} + +func m1Chunk(content string) string { + return fmt.Sprintf("data: {\"choices\":[{\"delta\":{\"content\":%q}}]}\n\n", content) +} + +const m1Finish = "data: {\"choices\":[{\"delta\":{},\"finish_reason\":\"stop\"}]}\n\n" + +// Bug 1: a slow consumer callback must not trip the idle watchdog; only +// provider silence counts. +func TestM1_SlowOnDeltaDoesNotTripIdleWatchdog(t *testing.T) { + setIdleForTest(t, 100*time.Millisecond) + var body strings.Builder + for i := 0; i < 8; i++ { + body.WriteString(m1Chunk("x")) + } + body.WriteString(m1Finish + "data: [DONE]\n\n") + srv := m1StreamServer(t, body.String()) + cc := m1OpenAIClient(t, srv) + + res, err := cc.CallStream(context.Background(), m1Req(), func(Delta) error { + time.Sleep(150 * time.Millisecond) + return nil + }) + if err != nil { + t.Fatalf("slow consumer must not cause an error, got: %v", err) + } + if res == nil || res.Content != "xxxxxxxx" || res.FinishReason != FinishStop { + t.Fatalf("result = %+v", res) + } +} + +// Bug 2: provider error events and a bare [DONE] must not be silent success. +func TestM1_OpenAIStreamErrorAndBareDone(t *testing.T) { + cases := []struct { + name string + body string + wantMsg string + }{ + {"error event then DONE", "data: {\"error\":{\"message\":\"upstream overloaded\"}}\n\ndata: [DONE]\n\n", "upstream overloaded"}, + {"bare DONE", "data: [DONE]\n\n", ""}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + fastBackoff(t) + srv := m1StreamServer(t, tc.body) + cc := m1OpenAIClient(t, srv) + res, err := cc.CallStream(context.Background(), m1Req(), func(Delta) error { return nil }) + if err == nil { + t.Fatalf("expected an error, got success: %+v", res) + } + if tc.wantMsg != "" && !strings.Contains(err.Error(), tc.wantMsg) { + t.Fatalf("provider message lost: %v", err) + } + }) + } +} + +// Bug 2 sibling: Gemini stream error chunk must preserve the provider message. +func TestM1_GeminiStreamErrorChunkPreservesMessage(t *testing.T) { + fastBackoff(t) + srv := m1StreamServer(t, "data: {\"error\":{\"code\":503,\"message\":\"upstream overloaded\",\"status\":\"UNAVAILABLE\"}}\n\n") + cc := newTestClient(t, ProviderConfig{ID: "gemini", Format: FormatGemini, BaseURL: srv.URL, APIKey: "k"}, srv) + res, err := cc.CallStream(context.Background(), m1Req(), func(Delta) error { return nil }) + if err == nil { + t.Fatalf("expected an error, got success: %+v", res) + } + if !strings.Contains(err.Error(), "upstream overloaded") { + t.Fatalf("provider message lost: %v", err) + } +} + +// Bug 3: a leading UTF-8 BOM must be stripped and bare-CR line endings +// accepted, per the SSE spec. +func TestM1_SSEBOMAndBareCR(t *testing.T) { + cases := []struct { + name string + body string + }{ + {"BOM", "\xef\xbb\xbf" + m1Chunk("hello") + m1Finish + "data: [DONE]\n\n"}, + {"bare CR", strings.NewReplacer("\n", "\r").Replace(m1Chunk("hello") + m1Finish + "data: [DONE]\n\n")}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + srv := m1StreamServer(t, tc.body) + cc := m1OpenAIClient(t, srv) + res, err := cc.CallStream(context.Background(), m1Req(), func(Delta) error { return nil }) + if err != nil { + t.Fatalf("err: %v", err) + } + if res.Content != "hello" || res.FinishReason != FinishStop { + t.Fatalf("first event dropped or stream mis-parsed: %+v", res) + } + }) + } +} diff --git a/m1_transport_test.go b/m1_transport_test.go new file mode 100644 index 0000000..7e6a5c6 --- /dev/null +++ b/m1_transport_test.go @@ -0,0 +1,41 @@ +package llm + +import ( + "net/http" + "net/url" + "testing" +) + +// Bug 7: a scheme downgrade on the same host must not be followed (it would +// resend API-key headers in plaintext). +func TestM1_SameHostRedirectRefusesSchemeDowngrade(t *testing.T) { + mk := func(u string) *http.Request { + pu, err := url.Parse(u) + if err != nil { + t.Fatal(err) + } + return &http.Request{URL: pu} + } + cases := []struct { + name string + from, to string + follow bool + }{ + {"https to http same host", "https://api.example.com/v1/chat", "http://api.example.com/v1/chat", false}, + {"https to http explicit port", "https://api.example.com/v1/chat", "http://api.example.com:443/v1/chat", false}, + {"https to https same host", "https://api.example.com/a", "https://api.example.com/b", true}, + {"http to https same host", "http://api.example.com/a", "https://api.example.com/b", true}, + {"other host", "https://api.example.com/a", "https://evil.example.net/a", false}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + err := sameHostRedirect(mk(tc.to), []*http.Request{mk(tc.from)}) + if tc.follow && err != nil { + t.Fatalf("redirect should be followed, got %v", err) + } + if !tc.follow && err != http.ErrUseLastResponse { + t.Fatalf("redirect should be refused, got %v", err) + } + }) + } +} diff --git a/m2_anthropic_gemini_test.go b/m2_anthropic_gemini_test.go new file mode 100644 index 0000000..30ce292 --- /dev/null +++ b/m2_anthropic_gemini_test.go @@ -0,0 +1,111 @@ +package llm + +import ( + "encoding/json" + "strings" + "testing" +) + +const m2AnthropicNoArgStream = `event: message_start +data: {"type":"message_start","message":{"usage":{"input_tokens":3}}} + +event: content_block_start +data: {"type":"content_block_start","index":0,"content_block":{"type":"tool_use","id":"tu1","name":"%s","input":{}}} + +event: content_block_delta +data: {"type":"content_block_delta","index":0,"delta":{"type":"input_json_delta","partial_json":""}} + +event: content_block_stop +data: {"type":"content_block_stop","index":0} + +event: message_delta +data: {"type":"message_delta","delta":{"stop_reason":"tool_use"},"usage":{"output_tokens":2}} + +event: message_stop +data: {"type":"message_stop"} + +` + +func m2AnthropicStreamBody(name string) string { + return strings.Replace(m2AnthropicNoArgStream, "%s", name, 1) +} + +// Bug 5: a no-argument Anthropic tool streams input:{} then an empty +// partial_json; the result must carry "{}" (as buffered does), valid JSON. +func TestM2_AnthropicNoArgToolArgumentsNormalized(t *testing.T) { + buffered := `{"content":[{"type":"tool_use","id":"tu1","name":"f","input":{}}],"stop_reason":"tool_use","usage":{"input_tokens":3,"output_tokens":2}}` + for _, stream := range []bool{false, true} { + body := buffered + if stream { + body = m2AnthropicStreamBody("f") + } + c := m2Client(t, FormatAnthropic, "anthropic", body, stream) + res, ds := m2Run(t, c, m2Req(), stream) + if len(res.ToolCalls) != 1 { + t.Fatalf("stream=%v: calls = %+v", stream, res.ToolCalls) + } + args := res.ToolCalls[0].Arguments + if args != "{}" || !json.Valid([]byte(args)) { + t.Errorf("stream=%v: Arguments = %q, want {}", stream, args) + } + _ = ds + } +} + +// Bug 5 (JSON mode, native since M4 SL02): the reply text is returned +// unchanged as Content, buffered and streaming alike. (The empty-input +// tool folding to "{}" returns in SL03 behind the AnthropicToolJSON quirk.) +func TestM2_AnthropicJSONModeEmptyInputContent(t *testing.T) { + req := &ChatRequest{ + Messages: []Message{{Role: RoleUser, Content: "hi"}}, + ResponseFormat: &ResponseFormat{Type: ResponseJSONObject}, + } + for _, stream := range []bool{false, true} { + body := m4TextReply(`{}`, stream) + c := m2Client(t, FormatAnthropic, "anthropic", body, stream) + res, _ := m2Run(t, c, req, stream) + if res.Content != "{}" { + t.Errorf("stream=%v: Content = %q, want {}", stream, res.Content) + } + if res.FinishReason != FinishStop { + t.Errorf("stream=%v: finish = %q, want stop", stream, res.FinishReason) + } + } +} + +// Bug 6: Gemini image-safety finish reasons are content filters, like +// SAFETY; IMAGE_OTHER and unknown reasons stay unmapped. +func TestM2_GeminiImageSafetyFinishReasons(t *testing.T) { + cases := map[string]string{ + "IMAGE_SAFETY": FinishContentFilter, + "IMAGE_PROHIBITED_CONTENT": FinishContentFilter, + "IMAGE_RECITATION": FinishContentFilter, + "SAFETY": FinishContentFilter, + "PROHIBITED_CONTENT": FinishContentFilter, + "SPII": FinishContentFilter, + "BLOCKLIST": FinishContentFilter, + "RECITATION": FinishContentFilter, + "IMAGE_OTHER": "", + "OTHER": "", + } + for in, want := range cases { + if got := mapGeminiFinishReason(in); got != want { + t.Errorf("mapGeminiFinishReason(%q) = %q, want %q", in, got, want) + } + } +} + +func TestM2_GeminiImageSafetyEndToEnd(t *testing.T) { + body := `{"candidates":[{"content":{"parts":[{"text":"x"}]},"finishReason":"IMAGE_SAFETY"}]}` + for _, stream := range []bool{false, true} { + b := body + if stream { + b = "data: " + body + "\n\n" + } + c := m2Client(t, FormatGemini, "gemini", b, stream) + res, _ := m2Run(t, c, &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "hi"}}}, stream) + if res.FinishReason != FinishContentFilter { + t.Errorf("stream=%v: finish = %q, want content_filter", stream, res.FinishReason) + } + } +} diff --git a/m2_helpers_test.go b/m2_helpers_test.go new file mode 100644 index 0000000..94f1f33 --- /dev/null +++ b/m2_helpers_test.go @@ -0,0 +1,50 @@ +package llm + +import ( + "context" + "fmt" + "net/http" + "net/http/httptest" + "testing" +) + +// m2Client builds a ChatClient of the given format against a server that +// answers every request with body (SSE when stream is true). +func m2Client(t *testing.T, format Format, id, body string, stream bool) *ChatClient { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if stream { + w.Header().Set("Content-Type", "text/event-stream") + } else { + w.Header().Set("Content-Type", "application/json") + } + _, _ = fmt.Fprint(w, body) + })) + t.Cleanup(srv.Close) + return newTestClient(t, ProviderConfig{ID: id, Format: format, BaseURL: srv.URL, APIKey: "k"}, srv) +} + +// m2Run executes the request buffered or streaming and returns the result +// plus every delta (streaming only). +func m2Run(t *testing.T, c *ChatClient, req *ChatRequest, stream bool) (*ChatResult, []Delta) { + t.Helper() + var ds []Delta + var res *ChatResult + var err error + if stream { + res, err = c.CallStream(context.Background(), req, func(d Delta) error { ds = append(ds, d); return nil }) + } else { + res, err = c.Call(context.Background(), req) + } + if err != nil { + t.Fatalf("call (stream=%v): %v", stream, err) + } + return res, ds +} + +func m2Req() *ChatRequest { + return &ChatRequest{ + Messages: []Message{{Role: RoleUser, Content: "hi"}}, + Tools: []ToolDef{{Name: "f", Description: "d", Parameters: []byte(`{"type":"object"}`)}}, + } +} diff --git a/m2_openai_test.go b/m2_openai_test.go new file mode 100644 index 0000000..d3bb667 --- /dev/null +++ b/m2_openai_test.go @@ -0,0 +1,75 @@ +package llm + +import "testing" + +// Bug 1: a tool-call turn must finish tool_calls even when an OpenAI-format +// provider reports finish_reason "stop" (named tool_choice, gateways). +func TestM2_OpenAIToolTurnWithStopFinishIsToolCalls(t *testing.T) { + buffered := `{"choices":[{"message":{"role":"assistant","content":"","tool_calls":[{"id":"c1","type":"function","function":{"name":"f","arguments":"{}"}}]},"finish_reason":"stop"}]}` + streamed := `data: {"choices":[{"delta":{"tool_calls":[{"index":0,"id":"c1","function":{"name":"f","arguments":"{}"}}]}}]}` + "\n\n" + + `data: {"choices":[{"delta":{},"finish_reason":"stop"}]}` + "\n\n" + "data: [DONE]\n\n" + for _, stream := range []bool{false, true} { + body := buffered + if stream { + body = streamed + } + c := m2Client(t, FormatOpenAI, "openai", body, stream) + res, _ := m2Run(t, c, m2Req(), stream) + if len(res.ToolCalls) != 1 { + t.Fatalf("stream=%v: tool calls = %+v", stream, res.ToolCalls) + } + if res.FinishReason != FinishToolCalls { + t.Errorf("stream=%v: finish = %q, want %q", stream, res.FinishReason, FinishToolCalls) + } + } +} + +// Bug 1 control: a plain stop with no tool calls stays stop. +func TestM2_OpenAIPlainStopStaysStop(t *testing.T) { + c := m2Client(t, FormatOpenAI, "openai", `{"choices":[{"message":{"role":"assistant","content":"x"},"finish_reason":"stop"}]}`, false) + res, _ := m2Run(t, c, m2Req(), false) + if res.FinishReason != FinishStop { + t.Errorf("finish = %q, want stop", res.FinishReason) + } +} + +// Bug 2: streamed tool-call deltas without an index but with distinct ids +// are separate calls; ToolIndex is the position in ChatResult.ToolCalls. +func TestM2_OpenAIStreamToolCallsWithoutIndex(t *testing.T) { + body := `data: {"choices":[{"delta":{"tool_calls":[{"id":"a","function":{"name":"f","arguments":"{\"x\":1}"}}]}}]}` + "\n\n" + + `data: {"choices":[{"delta":{"tool_calls":[{"id":"b","function":{"name":"g","arguments":"{\"y\":2}"}}]}}]}` + "\n\n" + + `data: {"choices":[{"delta":{},"finish_reason":"tool_calls"}]}` + "\n\n" + "data: [DONE]\n\n" + c := m2Client(t, FormatOpenAI, "openai", body, true) + res, ds := m2Run(t, c, m2Req(), true) + want := []ToolCall{{ID: "a", Name: "f", Arguments: `{"x":1}`}, {ID: "b", Name: "g", Arguments: `{"y":2}`}} + if len(res.ToolCalls) != len(want) { + t.Fatalf("tool calls = %+v, want %+v", res.ToolCalls, want) + } + for i, w := range want { + if res.ToolCalls[i] != w { + t.Errorf("call %d = %+v, want %+v", i, res.ToolCalls[i], w) + } + } + var idx []int + for _, d := range ds { + if d.Kind == DeltaToolArgs { + idx = append(idx, d.ToolIndex) + } + } + if len(idx) != 2 || idx[0] != 0 || idx[1] != 1 { + t.Errorf("ToolIndex sequence = %v, want [0 1]", idx) + } +} + +// Bug 2 control: continuation chunks that omit both index and id (or repeat +// the same id) still extend the current call. +func TestM2_OpenAIStreamNoIndexContinuationStaysOneCall(t *testing.T) { + body := `data: {"choices":[{"delta":{"tool_calls":[{"id":"a","function":{"name":"f","arguments":"{\"x\""}}]}}]}` + "\n\n" + + `data: {"choices":[{"delta":{"tool_calls":[{"function":{"arguments":":1}"}}]}}]}` + "\n\n" + + `data: {"choices":[{"delta":{},"finish_reason":"tool_calls"}]}` + "\n\n" + "data: [DONE]\n\n" + c := m2Client(t, FormatOpenAI, "openai", body, true) + res, _ := m2Run(t, c, m2Req(), true) + if len(res.ToolCalls) != 1 || res.ToolCalls[0].Arguments != `{"x":1}` { + t.Errorf("tool calls = %+v", res.ToolCalls) + } +} diff --git a/m2_responses_test.go b/m2_responses_test.go new file mode 100644 index 0000000..5fdb41b --- /dev/null +++ b/m2_responses_test.go @@ -0,0 +1,58 @@ +package llm + +import "testing" + +// Bug 3: an incomplete response whose reason is not max_output_tokens or +// content_filter is an unmapped stop reason: "" (invariant 1), never +// tool_calls or length. +func TestM2_ResponsesIncompleteUnknownReasonIsEmpty(t *testing.T) { + inc := func(reason string) *rsResponse { + r := &rsResponse{Status: "incomplete"} + if reason != "" { + r.IncompleteDetails = &struct { + Reason string `json:"reason"` + }{Reason: reason} + } + return r + } + cases := []struct { + name string + r *rsResponse + hasTools bool + want string + }{ + {"unknown reason, tools", inc("something_new"), true, ""}, + {"unknown reason, no tools", inc("something_new"), false, ""}, + {"max output stays length", inc("max_output_tokens"), true, FinishLength}, + {"content filter stays", inc("content_filter"), false, FinishContentFilter}, + } + for _, c := range cases { + if got := responsesFinishReason(c.r, c.hasTools); got != c.want { + t.Errorf("%s: %q, want %q", c.name, got, c.want) + } + } +} + +// Bug 3 through the buffered parser and the stream mapper. +func TestM2_ResponsesIncompleteUnknownReasonParsers(t *testing.T) { + res, err := parseResponsesAPI([]byte(`{"status":"incomplete","incomplete_details":{"reason":"something_new"},"output":[{"type":"function_call","call_id":"c1","name":"f","arguments":"{}"}]}`)) + if err != nil { + t.Fatal(err) + } + if res.FinishReason != "" { + t.Errorf("buffered finish = %q, want \"\"", res.FinishReason) + } + acc := newStreamAccum() + for _, e := range []string{ + `{"type":"response.output_item.added","output_index":0,"item":{"type":"function_call","call_id":"c1","name":"f"}}`, + `{"type":"response.function_call_arguments.delta","output_index":0,"delta":"{}"}`, + `{"type":"response.incomplete","response":{"status":"incomplete","incomplete_details":{"reason":"something_new"}}}`, + } { + if _, _, err := mapResponsesStreamEvent([]byte(e), acc); err != nil { + t.Fatal(err) + } + } + if got := acc.result().FinishReason; got != "" { + t.Errorf("streaming finish = %q, want \"\"", got) + } +} diff --git a/m3_helpers_test.go b/m3_helpers_test.go new file mode 100644 index 0000000..ae5d35e --- /dev/null +++ b/m3_helpers_test.go @@ -0,0 +1,79 @@ +package llm + +import ( + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "net/http/httptest" + "sync" + "testing" +) + +// m3Wire is what a test server captured from one request. +type m3Wire struct { + mu sync.Mutex + hits int + escPath string + rawQ string + body []byte +} + +func (w *m3Wire) snapshot() (int, string, string, []byte) { + w.mu.Lock() + defer w.mu.Unlock() + return w.hits, w.escPath, w.rawQ, w.body +} + +// m3Reply canned success bodies per wire format (buffered). +func m3Reply(format Format) string { + switch format { + case FormatAnthropic: + return `{"id":"m","type":"message","role":"assistant","content":[{"type":"text","text":"ok"}],"stop_reason":"end_turn","usage":{"input_tokens":1,"output_tokens":1}}` + case FormatGemini: + return `{"candidates":[{"content":{"role":"model","parts":[{"text":"ok"}]},"finishReason":"STOP"}]}` + default: + return `{"id":"c","choices":[{"index":0,"message":{"role":"assistant","content":"ok"},"finish_reason":"stop"}],"usage":{"prompt_tokens":1,"completion_tokens":1}}` + } +} + +// m3Server records every request and answers 200 with body (SSE when sse). +func m3Server(t *testing.T, body string, sse bool) (*httptest.Server, *m3Wire) { + t.Helper() + rec := &m3Wire{} + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + b, _ := io.ReadAll(r.Body) + rec.mu.Lock() + rec.hits++ + rec.escPath, rec.rawQ, rec.body = r.URL.EscapedPath(), r.URL.RawQuery, b + rec.mu.Unlock() + if sse { + w.Header().Set("Content-Type", "text/event-stream") + } else { + w.Header().Set("Content-Type", "application/json") + } + _, _ = fmt.Fprint(w, body) + })) + t.Cleanup(srv.Close) + return srv, rec +} + +// m3Chat sends req through a ChatClient of the given format and returns the +// call error plus the captured wire request. +func m3Chat(t *testing.T, format Format, req *ChatRequest) (*m3Wire, error) { + t.Helper() + srv, rec := m3Server(t, m3Reply(format), false) + c := newTestClient(t, ProviderConfig{ID: string(format) + "-m3", Format: format, BaseURL: srv.URL, APIKey: "k"}, srv) + _, err := c.Call(context.Background(), req) + return rec, err +} + +func m3JSON(t *testing.T, b []byte) map[string]any { + t.Helper() + var m map[string]any + if err := json.Unmarshal(b, &m); err != nil { + t.Fatalf("captured body is not JSON: %v\n%s", err, b) + } + return m +} diff --git a/m3_request_validity_test.go b/m3_request_validity_test.go new file mode 100644 index 0000000..5ec496e --- /dev/null +++ b/m3_request_validity_test.go @@ -0,0 +1,359 @@ +package llm + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net/http" + "net/url" + "strings" + "testing" +) + +// ── Bug 1: image/jpg alias accepted at the boundary ───────────────────── + +func TestM3JPGAliasAcceptedEndToEnd(t *testing.T) { + png := []byte{1, 2, 3} + for _, format := range []Format{FormatOpenAI, FormatAnthropic, FormatGemini} { + t.Run(string(format)+"/user", func(t *testing.T) { + req := &ChatRequest{Messages: []Message{{Role: RoleUser, Parts: []ContentPart{ImagePart("image/jpg", png)}}}} + rec, err := m3Chat(t, format, req) + if err != nil { + t.Fatalf("image/jpg must be accepted (README): %v", err) + } + _, _, _, body := rec.snapshot() + if strings.Contains(string(body), "image/jpg") || !strings.Contains(string(body), "image/jpeg") { + t.Fatalf("wire must carry image/jpeg only: %s", body) + } + }) + } + // Tool-result images ride the same validation + normalization. + for _, format := range []Format{FormatAnthropic, FormatGemini} { + t.Run(string(format)+"/tool", func(t *testing.T) { + req := &ChatRequest{Messages: []Message{ + {Role: RoleUser, Content: "look"}, + {Role: RoleAssistant, ToolCalls: []ToolCall{{ID: "c1", Name: "shot", Arguments: "{}"}}}, + {Role: RoleTool, ToolCallID: "c1", ToolName: "shot", Parts: []ContentPart{ImagePart("image/jpg", png)}}, + }} + rec, err := m3Chat(t, format, req) + if err != nil { + t.Fatalf("image/jpg in a tool result must be accepted: %v", err) + } + _, _, _, body := rec.snapshot() + if strings.Contains(string(body), "image/jpg") || !strings.Contains(string(body), "image/jpeg") { + t.Fatalf("wire must carry image/jpeg only: %s", body) + } + }) + } +} + +// ── Bug 2: empty text never yields an empty/invalid block ─────────────── +// Policy: an empty text block is skipped when the message/system still has +// other content; a user message left with no content at all is a ConfigError +// before any network I/O. Empty system text simply contributes nothing. + +// m3Blocks returns the content blocks (anthropic) or parts (gemini) of every +// message plus the system blocks/parts. +func m3AllBlocks(t *testing.T, format Format, body []byte) [][]any { + t.Helper() + m := m3JSON(t, body) + var groups [][]any + if format == FormatAnthropic { + if s, ok := m["system"].([]any); ok { + groups = append(groups, s) + } + for _, msg := range m["messages"].([]any) { + groups = append(groups, msg.(map[string]any)["content"].([]any)) + } + return groups + } + if si, ok := m["systemInstruction"].(map[string]any); ok { + groups = append(groups, si["parts"].([]any)) + } + for _, c := range m["contents"].([]any) { + groups = append(groups, c.(map[string]any)["parts"].([]any)) + } + return groups +} + +func m3AssertValidBlocks(t *testing.T, format Format, body []byte) { + t.Helper() + for _, g := range m3AllBlocks(t, format, body) { + if len(g) == 0 { + t.Errorf("empty content array on wire: %s", body) + } + for _, b := range g { + blk := b.(map[string]any) + if format == FormatAnthropic { + if blk["type"] == "text" { + if s, ok := blk["text"].(string); !ok || s == "" { + t.Errorf("text block without non-empty text: %v in %s", blk, body) + } + } + } else if len(blk) == 0 { + t.Errorf("empty Gemini part {} on wire: %s", body) + } + } + } +} + +func TestM3EmptyTextSkippedWhenOtherContent(t *testing.T) { + cases := []struct { + name string + req *ChatRequest + }{ + {"in-band empty system message", &ChatRequest{Messages: []Message{ + {Role: RoleSystem, Content: ""}, {Role: RoleUser, Content: "hi"}}}}, + {"empty SystemBlock", &ChatRequest{System: []SystemBlock{{Text: ""}}, + Messages: []Message{{Role: RoleUser, Content: "hi"}}}}, + {"empty SystemBlock beside real one", &ChatRequest{System: []SystemBlock{{Text: ""}, {Text: "be terse"}}, + Messages: []Message{{Role: RoleUser, Content: "hi"}}}}, + } + for _, format := range []Format{FormatAnthropic, FormatGemini} { + for _, c := range cases { + t.Run(string(format)+"/"+c.name, func(t *testing.T) { + rec, err := m3Chat(t, format, c.req) + if err != nil { + t.Fatalf("empty system text must be skipped, not fail: %v", err) + } + _, _, _, body := rec.snapshot() + m3AssertValidBlocks(t, format, body) + if !strings.Contains(string(body), "hi") { + t.Fatalf("user content lost: %s", body) + } + if strings.Contains(c.name, "beside") && !strings.Contains(string(body), "be terse") { + t.Fatalf("real system block lost: %s", body) + } + }) + } + } +} + +func TestM3MessageWithNoContentRejected(t *testing.T) { + cases := []struct { + name string + msgs []Message + }{ + {"user empty content", []Message{{Role: RoleUser, Content: ""}}}, + {"user empty after history", []Message{{Role: RoleUser, Content: "a"}, {Role: RoleAssistant, Content: "b"}, {Role: RoleUser, Content: ""}}}, + } + for _, format := range []Format{FormatAnthropic, FormatGemini} { + for _, c := range cases { + t.Run(string(format)+"/"+c.name, func(t *testing.T) { + rec, err := m3Chat(t, format, &ChatRequest{Messages: c.msgs}) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Fatalf("err = %T (%v), want *ConfigError", err, err) + } + if hits, _, _, _ := rec.snapshot(); hits != 0 { + t.Fatalf("invalid request reached the network (%d hits)", hits) + } + }) + } + } +} + +// ── Bug 3: empty assistant turn leaves no whitespace-only block ───────── + +func TestM3EmptyAssistantTurnDropped(t *testing.T) { + cases := []struct { + name string + msgs []Message + wantRoles []string + }{ + {"between users merges them", []Message{ + {Role: RoleUser, Content: "first"}, {Role: RoleAssistant, Content: ""}, {Role: RoleUser, Content: "second"}}, + []string{"user"}}, + {"trailing empty assistant dropped", []Message{ + {Role: RoleUser, Content: "first"}, {Role: RoleAssistant, Content: ""}}, + []string{"user"}}, + {"mid-history keeps alternation", []Message{ + {Role: RoleUser, Content: "a"}, {Role: RoleAssistant, Content: "b"}, + {Role: RoleUser, Content: "c"}, {Role: RoleAssistant, Content: ""}, {Role: RoleUser, Content: "d"}}, + []string{"user", "assistant", "user"}}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + rec, err := m3Chat(t, FormatAnthropic, &ChatRequest{Messages: c.msgs}) + if err != nil { + t.Fatalf("call: %v", err) + } + _, _, _, body := rec.snapshot() + m3AssertValidBlocks(t, FormatAnthropic, body) + for _, g := range m3AllBlocks(t, FormatAnthropic, body) { + for _, b := range g { + if s, _ := b.(map[string]any)["text"].(string); s != "" && strings.TrimSpace(s) == "" { + t.Errorf("whitespace-only text block on wire: %s", body) + } + } + } + var roles []string + for _, msg := range m3JSON(t, body)["messages"].([]any) { + roles = append(roles, msg.(map[string]any)["role"].(string)) + } + if fmt.Sprint(roles) != fmt.Sprint(c.wantRoles) { + t.Fatalf("roles = %v, want %v (alternation)\n%s", roles, c.wantRoles, body) + } + // merged user turn keeps every original text + for _, m := range c.msgs { + if m.Role == RoleUser && !strings.Contains(string(body), `"`+m.Content+`"`) { + t.Errorf("user text %q lost: %s", m.Content, body) + } + } + }) + } +} + +// ── Bug 4: Gemini tool schemas travel as parametersJsonSchema ─────────── + +func TestM3GeminiToolSchemaIsJSONSchema(t *testing.T) { + schemas := map[string]string{ + "nullable type array": `{"type":"object","properties":{"a":{"type":["string","null"]}}}`, + "ref and defs": `{"type":"object","$defs":{"x":{"type":"string"}},"properties":{"a":{"$ref":"#/$defs/x"}}}`, + "const and additional": `{"type":"object","additionalProperties":false,"properties":{"k":{"const":"v"}}}`, + } + for name, schema := range schemas { + t.Run(name, func(t *testing.T) { + req := &ChatRequest{ + Messages: []Message{{Role: RoleUser, Content: "hi"}}, + Tools: []ToolDef{{Name: "f", Description: "d", Parameters: json.RawMessage(schema)}}, + } + rec, err := m3Chat(t, FormatGemini, req) + if err != nil { + t.Fatal(err) + } + _, _, _, body := rec.snapshot() + tools := m3JSON(t, body)["tools"].([]any) + decl := tools[0].(map[string]any)["functionDeclarations"].([]any)[0].(map[string]any) + if _, bad := decl["parameters"]; bad { + t.Errorf("JSON Schema must not ride the restricted OpenAPI `parameters` field: %v", decl) + } + got, ok := decl["parametersJsonSchema"] + if !ok { + t.Fatalf("parametersJsonSchema missing: %v", decl) + } + gb, _ := json.Marshal(got) + var want, have any + _ = json.Unmarshal([]byte(schema), &want) + _ = json.Unmarshal(gb, &have) + if fmt.Sprint(want) != fmt.Sprint(have) { + t.Errorf("schema altered: got %s want %s", gb, schema) + } + }) + } + t.Run("no schema sends neither field", func(t *testing.T) { + req := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "hi"}}, Tools: []ToolDef{{Name: "f"}}} + rec, err := m3Chat(t, FormatGemini, req) + if err != nil { + t.Fatal(err) + } + _, _, _, body := rec.snapshot() + if strings.Contains(string(body), `"parameters"`) { + t.Errorf("unexpected parameters key: %s", body) + } + }) +} + +// ── Bug 5: every Gemini URL builder normalizes the model id ───────────── + +func TestM3GeminiModelURLNormalized(t *testing.T) { + models := []struct{ in, wantSeg string }{ + {"models/gemini-x", "gemini-x"}, + {"gemini-x", "gemini-x"}, + {"gem/ini?x=1", url.PathEscape("gem/ini?x=1")}, + } + ttsBody := `{"candidates":[{"content":{"parts":[{"inlineData":{"mimeType":"audio/L16;rate=24000","data":"AQID"}}]}}]}` + sseBody := "data: " + m3Reply(FormatGemini) + "\n\n" + for _, mm := range models { + check := func(t *testing.T, rec *m3Wire, suffix string) { + t.Helper() + _, esc, rawQ, _ := rec.snapshot() + if want := "/v1beta/models/" + mm.wantSeg + suffix; esc != want { + t.Errorf("path = %q, want %q", esc, want) + } + if rawQ != "" && !strings.HasPrefix(suffix, ":streamGenerate") { + t.Errorf("model smuggled a query: %q", rawQ) + } + } + t.Run("chat/"+mm.in, func(t *testing.T) { + srv, rec := m3Server(t, m3Reply(FormatGemini), false) + c := newTestClient(t, ProviderConfig{ID: "g", Format: FormatGemini, BaseURL: srv.URL, APIKey: "k"}, srv) + c.model = mm.in + if _, err := c.Call(context.Background(), &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "hi"}}}); err != nil { + t.Fatal(err) + } + check(t, rec, ":generateContent") + }) + t.Run("stream/"+mm.in, func(t *testing.T) { + srv, rec := m3Server(t, sseBody, true) + c := newTestClient(t, ProviderConfig{ID: "g", Format: FormatGemini, BaseURL: srv.URL, APIKey: "k"}, srv) + c.model = mm.in + if _, err := c.CallStream(context.Background(), &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "hi"}}}, func(Delta) error { return nil }); err != nil { + t.Fatal(err) + } + check(t, rec, ":streamGenerateContent") + }) + t.Run("embed/"+mm.in, func(t *testing.T) { + srv, rec := m3Server(t, `{"embeddings":[{"values":[0.1]}]}`, false) + s := newTestSDK(t, ProviderConfig{ID: "g", Format: FormatGemini, APIKey: "k"}, srv) + if _, err := s.Embed(context.Background(), "g", mm.in, EmbedRequest{Inputs: []string{"a"}}); err != nil { + t.Fatal(err) + } + check(t, rec, ":batchEmbedContents") + }) + t.Run("tts/"+mm.in, func(t *testing.T) { + srv, rec := m3Server(t, ttsBody, false) + s := newTestSDK(t, ProviderConfig{ID: "g", Format: FormatGemini, APIKey: "k"}, srv) + if _, err := s.Speak(context.Background(), "g", mm.in, SpeakRequest{Text: "hi", Voice: "Kore"}); err != nil { + t.Fatal(err) + } + check(t, rec, ":generateContent") + }) + } +} + +// ── Bug 6: non-JSON transcription formats return the raw body as Text ─── + +func TestM3TranscribeNonJSONFormatsReturnRawBody(t *testing.T) { + cases := []struct { + format, ctype, body string + }{ + {"text", "text/plain; charset=utf-8", "hello world\n"}, + {"srt", "application/x-subrip", "1\n00:00:00,000 --> 00:00:01,000\nhello\n\n"}, + {"vtt", "text/vtt", "WEBVTT\n\n00:00.000 --> 00:01.000\nhello\n"}, + } + for _, c := range cases { + t.Run(c.format, func(t *testing.T) { + srv, rec := newSTTServer(http.StatusOK, c.ctype, c.body) + defer srv.Close() + s := newTestSDK(t, ProviderConfig{ID: "openai", Format: FormatOpenAI, APIKey: "k"}, srv) + res, err := s.Transcribe(t.Context(), "openai", "whisper-1", TranscribeRequest{Audio: []byte{1}, Format: c.format}) + if err != nil { + t.Fatalf("billed transcription must not fail on a %s body: %v", c.format, err) + } + if res.Text != c.body { + t.Errorf("Text = %q, want raw body %q", res.Text, c.body) + } + if res.Model != "whisper-1" { + t.Errorf("Model = %q", res.Model) + } + vals, _ := sttParse(t, *rec) + if vals["response_format"] != c.format { + t.Errorf("response_format = %q, want %q", vals["response_format"], c.format) + } + }) + } + // JSON formats keep decoding (guard against over-broad raw handling). + for _, f := range []string{"", "json", "verbose_json"} { + t.Run("json/"+f, func(t *testing.T) { + srv, _ := newSTTServer(http.StatusOK, "application/json", `{"text":"hi","language":"en","duration":1.5}`) + defer srv.Close() + s := newTestSDK(t, ProviderConfig{ID: "openai", Format: FormatOpenAI, APIKey: "k"}, srv) + res, err := s.Transcribe(t.Context(), "openai", "whisper-1", TranscribeRequest{Audio: []byte{1}, Format: f}) + if err != nil || res.Text != "hi" { + t.Fatalf("res=%+v err=%v", res, err) + } + }) + } +} diff --git a/m4_json_test.go b/m4_json_test.go new file mode 100644 index 0000000..85cb684 --- /dev/null +++ b/m4_json_test.go @@ -0,0 +1,211 @@ +package llm + +import ( + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "net/http/httptest" + "strconv" + "testing" +) + +// m4TextReply renders an Anthropic text reply (buffered JSON or SSE) whose +// text block is text. +func m4TextReply(text string, stream bool) string { + q := strconv.Quote(text) + if !stream { + return `{"content":[{"type":"text","text":` + q + `}],"stop_reason":"end_turn","usage":{"input_tokens":3,"output_tokens":2}}` + } + // Split the text across two deltas to exercise accumulation. + h := len(text) / 2 + a, b := strconv.Quote(text[:h]), strconv.Quote(text[h:]) + return `event: message_start +data: {"type":"message_start","message":{"usage":{"input_tokens":3}}} + +event: content_block_start +data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}} + +event: content_block_delta +data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":` + a + `}} + +event: content_block_delta +data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":` + b + `}} + +event: content_block_stop +data: {"type":"content_block_stop","index":0} + +event: message_delta +data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":2}} + +event: message_stop +data: {"type":"message_stop"} + +` +} + +// m4Wire sends req through a real client against an httptest server and +// returns the captured request body plus the result. +func m4Wire(t *testing.T, req *ChatRequest, reply string, stream bool) (map[string]any, *ChatResult) { + t.Helper() + var body map[string]any + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + b, _ := io.ReadAll(r.Body) + _ = json.Unmarshal(b, &body) + if stream { + w.Header().Set("Content-Type", "text/event-stream") + } else { + w.Header().Set("Content-Type", "application/json") + } + _, _ = fmt.Fprint(w, reply) + })) + t.Cleanup(srv.Close) + c := newTestClient(t, ProviderConfig{ID: "anthropic", Format: FormatAnthropic, BaseURL: srv.URL, APIKey: "k"}, srv) + res, _ := m2Run(t, c, req, stream) + return body, res +} + +func TestM4_NativeJSONWireShape(t *testing.T) { + msgs := []Message{{Role: RoleUser, Content: "x"}} + schemaRF := &ResponseFormat{Type: ResponseJSONSchema, Name: "answer", Schema: testSchema} + cases := []struct { + name string + req *ChatRequest + wantOC string + wantTools int + wantChoice string // "" = absent + wantThink string // "" = absent + }{ + {"schema", &ChatRequest{Messages: msgs, ResponseFormat: schemaRF}, + `{"format":{"type":"json_schema","schema":` + string(testSchema) + `}}`, 0, "", ""}, + {"json_object", &ChatRequest{Messages: msgs, ResponseFormat: &ResponseFormat{Type: ResponseJSONObject}}, + `{"format":{"type":"json_schema","schema":{"type":"object"}}}`, 0, "", ""}, + {"schema without name", &ChatRequest{Messages: msgs, ResponseFormat: &ResponseFormat{Type: ResponseJSONSchema, Schema: testSchema}}, + `{"format":{"type":"json_schema","schema":` + string(testSchema) + `}}`, 0, "", ""}, + {"merged with effort", &ChatRequest{Messages: msgs, Thinking: "high", ResponseFormat: schemaRF}, + `{"effort":"high","format":{"type":"json_schema","schema":` + string(testSchema) + `}}`, 0, "", `{"type":"adaptive"}`}, + {"thinking enabled (no effort)", &ChatRequest{Messages: msgs, Thinking: "enabled", ResponseFormat: schemaRF}, + `{"format":{"type":"json_schema","schema":` + string(testSchema) + `}}`, 0, "", `{"type":"adaptive"}`}, + {"with tools", &ChatRequest{Messages: msgs, Tools: []ToolDef{{Name: "f", Parameters: json.RawMessage(`{"type":"object"}`)}}, ResponseFormat: schemaRF}, + `{"format":{"type":"json_schema","schema":` + string(testSchema) + `}}`, 1, "", ""}, + {"with auto ToolChoice", &ChatRequest{Messages: msgs, Tools: []ToolDef{{Name: "f"}}, ToolChoice: &ToolChoice{Mode: ToolChoiceAuto}, ResponseFormat: schemaRF}, + `{"format":{"type":"json_schema","schema":` + string(testSchema) + `}}`, 1, `{"type":"auto"}`, ""}, + {"with forced ToolChoice and adaptive thinking", &ChatRequest{Messages: msgs, Tools: []ToolDef{{Name: "f"}}, ToolChoice: &ToolChoice{Mode: ToolChoiceRequired}, Thinking: "low", ResponseFormat: schemaRF}, + `{"effort":"low","format":{"type":"json_schema","schema":` + string(testSchema) + `}}`, 1, `{"type":"any"}`, `{"type":"adaptive"}`}, + {"non-object schema passes through", &ChatRequest{Messages: msgs, ResponseFormat: &ResponseFormat{Type: ResponseJSONSchema, Schema: json.RawMessage(`{"type":"array"}`)}}, + `{"format":{"type":"json_schema","schema":{"type":"array"}}}`, 0, "", ""}, + {"legacy budget keeps JSON, no effort", &ChatRequest{Messages: msgs, Thinking: "high", ThinkingBudget: 2000, ResponseFormat: schemaRF}, + `{"format":{"type":"json_schema","schema":` + string(testSchema) + `}}`, 0, "", `{"type":"enabled","budget_tokens":2000}`}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + m, _ := buildBody(t, FormatAnthropic, "claude", tc.req) + jsonEq(t, "output_config", m["output_config"], tc.wantOC) + tools, _ := m["tools"].([]any) + if len(tools) != tc.wantTools { + t.Errorf("tools = %v, want %d (no synthetic JSON tool)", tools, tc.wantTools) + } + for _, x := range tools { + if n := x.(map[string]any)["name"]; n == "answer" || n == anthropicJSONToolDefault { + t.Errorf("synthetic JSON tool leaked: %v", n) + } + } + if tc.wantChoice == "" { + if m["tool_choice"] != nil { + t.Errorf("tool_choice = %v, want absent", m["tool_choice"]) + } + } else { + jsonEq(t, "tool_choice", m["tool_choice"], tc.wantChoice) + } + if tc.wantThink == "" { + if m["thinking"] != nil { + t.Errorf("thinking = %v, want absent", m["thinking"]) + } + } else { + jsonEq(t, "thinking", m["thinking"], tc.wantThink) + } + }) + } +} + +// Over a real HTTP exchange, native JSON mode leaves the reply text +// untouched (no folding) with the normal finish mapping, buffered and +// streaming, and sends output_config.format on the wire. +func TestM4_NativeJSONReplyUnchanged(t *testing.T) { + req := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, + Thinking: "medium", + ResponseFormat: &ResponseFormat{Type: ResponseJSONSchema, Name: "answer", Schema: testSchema}} + for _, stream := range []bool{false, true} { + body, res := m4Wire(t, req, m4TextReply(`{"a":1}`, stream), stream) + jsonEq(t, fmt.Sprintf("stream=%v output_config", stream), body["output_config"], + `{"effort":"medium","format":{"type":"json_schema","schema":`+string(testSchema)+`}}`) + if body["tools"] != nil || body["tool_choice"] != nil { + t.Errorf("stream=%v: wire tools/tool_choice = %v/%v", stream, body["tools"], body["tool_choice"]) + } + if res.Content != `{"a":1}` || len(res.ToolCalls) != 0 || res.FinishReason != FinishStop { + t.Errorf("stream=%v: result = %+v", stream, res) + } + } +} + +// A genuine tool call alongside native JSON mode stays a tool call, even if +// its name equals the schema name (no folding without the quirk). +func TestM4_NativeJSONNoFolding(t *testing.T) { + reply := `{"content":[{"type":"tool_use","id":"t","name":"answer","input":{"a":1}}],"stop_reason":"tool_use","usage":{"input_tokens":1,"output_tokens":2}}` + req := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, + Tools: []ToolDef{{Name: "answer", Parameters: json.RawMessage(`{"type":"object"}`)}}, + ResponseFormat: &ResponseFormat{Type: ResponseJSONSchema, Name: "answer", Schema: testSchema}} + _, res := m4Wire(t, req, reply, false) + if res.Content != "" || len(res.ToolCalls) != 1 || res.ToolCalls[0].Name != "answer" || res.FinishReason != FinishToolCalls { + t.Errorf("result = %+v, want untouched tool call", res) + } +} + +// Extra["output_config"] replaces the SDK-built object (shallow top-level +// merge); other Extra keys still merge. +func TestM4_ExtraOverridesOutputConfig(t *testing.T) { + req := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, + Thinking: "high", + ResponseFormat: &ResponseFormat{Type: ResponseJSONSchema, Schema: testSchema}, + Extra: map[string]any{"output_config": map[string]any{"effort": "xhigh"}, "service_tier": "auto"}} + for _, stream := range []bool{false, true} { + body, _ := m4Wire(t, req, m4TextReply(`{"a":1}`, stream), stream) + jsonEq(t, fmt.Sprintf("stream=%v output_config", stream), body["output_config"], `{"effort":"xhigh"}`) + if body["service_tier"] != "auto" { + t.Errorf("stream=%v: other Extra keys must merge: %v", stream, body) + } + jsonEq(t, "thinking untouched", body["thinking"], `{"type":"adaptive"}`) + } + // Extra can also supply output_config when the SDK builds none. + plain := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, + Extra: map[string]any{"output_config": map[string]any{"effort": "xhigh"}}} + m, _ := buildBody(t, FormatAnthropic, "claude", plain) + _ = m + body, _ := m4Wire(t, plain, m4TextReply("ok", false), false) + jsonEq(t, "extra only", body["output_config"], `{"effort":"xhigh"}`) +} + +// The builtin anthropic provider resolves to the native path through the +// public entry point (no quirk consulted from the client for JSON mode). +func TestM4_NativeJSONViaPublicChat(t *testing.T) { + var body map[string]any + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + b, _ := io.ReadAll(r.Body) + _ = json.Unmarshal(b, &body) + w.Header().Set("Content-Type", "application/json") + _, _ = fmt.Fprint(w, m4TextReply(`{"a":1}`, false)) + })) + t.Cleanup(srv.Close) + cfg := ProviderConfig{ID: "anthropic", Format: FormatAnthropic, BaseURL: srv.URL, APIKey: "k"} + c := newTestClient(t, cfg, srv) + res, err := c.Call(context.Background(), &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, + ResponseFormat: &ResponseFormat{Type: ResponseJSONObject}}) + if err != nil { + t.Fatal(err) + } + jsonEq(t, "output_config", body["output_config"], `{"format":{"type":"json_schema","schema":{"type":"object"}}}`) + if res.Content != `{"a":1}` { + t.Errorf("Content = %q", res.Content) + } +} diff --git a/m4_quirk_test.go b/m4_quirk_test.go new file mode 100644 index 0000000..787f5fa --- /dev/null +++ b/m4_quirk_test.go @@ -0,0 +1,194 @@ +package llm + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +// m4QuirkGateway registers a custom Anthropic-format provider through the +// public options, with the AnthropicToolJSON quirk set as given, and returns +// a chat client plus a pointer to the last captured request body. +func m4QuirkGateway(t *testing.T, quirkOn bool, reply func(stream bool) string) (*ChatClient, *map[string]any) { + t.Helper() + body := map[string]any{} + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + b, _ := io.ReadAll(r.Body) + body = map[string]any{} + _ = json.Unmarshal(b, &body) + stream, _ := body["stream"].(bool) + if stream { + w.Header().Set("Content-Type", "text/event-stream") + } else { + w.Header().Set("Content-Type", "application/json") + } + _, _ = fmt.Fprint(w, reply(stream)) + })) + t.Cleanup(srv.Close) + sdk := New(WithProvider("gw", + WithFormat(FormatAnthropic), WithBaseURL(srv.URL), WithAPIKey("k"), + WithQuirks(Quirks{AnthropicToolJSON: quirkOn}))) + c, err := sdk.Chat("gw", "claude-x") + if err != nil { + t.Fatal(err) + } + return c, &body +} + +func m4ToolReply(stream bool) string { + if !stream { + return `{"content":[{"type":"tool_use","id":"t","name":"answer","input":{"a":1}}],"stop_reason":"tool_use","usage":{"input_tokens":1,"output_tokens":2}}` + } + return strings.Join([]string{ + `event: message_start` + "\n" + `data: {"type":"message_start","message":{"usage":{"input_tokens":1}}}`, + `data: {"type":"content_block_start","index":0,"content_block":{"type":"tool_use","id":"t","name":"answer"}}`, + `data: {"type":"content_block_delta","index":0,"delta":{"type":"input_json_delta","partial_json":"{\"a\":"}}`, + `data: {"type":"content_block_delta","index":0,"delta":{"type":"input_json_delta","partial_json":"1}"}}`, + `data: {"type":"message_delta","delta":{"stop_reason":"tool_use"},"usage":{"output_tokens":2}}`, + `data: {"type":"message_stop"}`, + }, "\n\n") + "\n\n" +} + +func m4JSONReq() *ChatRequest { + return &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, + ResponseFormat: &ResponseFormat{Type: ResponseJSONSchema, Name: "answer", Schema: testSchema}} +} + +func TestM4Quirk_BuiltinAnthropicOff(t *testing.T) { + p, err := New().Provider("anthropic") + if err != nil { + t.Fatal(err) + } + if p.Config().Quirks.AnthropicToolJSON { + t.Error("built-in anthropic must have AnthropicToolJSON off") + } +} + +func TestM4Quirk_CustomProviderSetsFlag(t *testing.T) { + sdk := New(WithProvider("gw", WithFormat(FormatAnthropic), WithBaseURL("http://x"), WithAPIKey("k"), + WithQuirks(Quirks{AnthropicToolJSON: true}))) + p, err := sdk.Provider("gw") + if err != nil { + t.Fatal(err) + } + if !p.Config().Quirks.AnthropicToolJSON { + t.Error("custom provider must carry AnthropicToolJSON through WithQuirks") + } +} + +// Quirk ON: forced synthetic tool, tool_choice, no output_config.format. +func TestM4Quirk_EmulationRequestBody(t *testing.T) { + c, body := m4QuirkGateway(t, true, m4ToolReply) + if _, err := c.Call(context.Background(), m4JSONReq()); err != nil { + t.Fatal(err) + } + jsonEq(t, "tool_choice", (*body)["tool_choice"], `{"type":"tool","name":"answer"}`) + tools, _ := (*body)["tools"].([]any) + if len(tools) != 1 { + t.Fatalf("tools = %v", (*body)["tools"]) + } + last := tools[len(tools)-1].(map[string]any) + if last["name"] != "answer" { + t.Fatalf("json tool = %v", last) + } + jsonEq(t, "json tool schema", last["input_schema"], string(testSchema)) + if oc, ok := (*body)["output_config"].(map[string]any); ok && oc["format"] != nil { + t.Errorf("emulation must not send output_config.format: %v", oc) + } + + // json_object: default tool name and {"type":"object"} schema. + obj := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, + ResponseFormat: &ResponseFormat{Type: ResponseJSONObject}} + if _, err := c.Call(context.Background(), obj); err != nil { + t.Fatal(err) + } + tools, _ = (*body)["tools"].([]any) + if len(tools) != 1 || tools[0].(map[string]any)["name"] != anthropicJSONToolDefault { + t.Fatalf("default json tool = %v", (*body)["tools"]) + } + jsonEq(t, "default schema", tools[0].(map[string]any)["input_schema"], `{"type":"object"}`) +} + +// Quirk ON: the tool input is folded back into Content (buffered + stream). +func TestM4Quirk_EmulationResultFolding(t *testing.T) { + c, _ := m4QuirkGateway(t, true, m4ToolReply) + res, err := c.Call(context.Background(), m4JSONReq()) + if err != nil { + t.Fatal(err) + } + if res.Content != `{"a":1}` || len(res.ToolCalls) != 0 || res.FinishReason != FinishStop { + t.Errorf("buffered = %+v", res) + } + var deltas []Delta + res, err = c.CallStream(context.Background(), m4JSONReq(), func(d Delta) error { deltas = append(deltas, d); return nil }) + if err != nil { + t.Fatal(err) + } + if res.Content != `{"a":1}` || len(res.ToolCalls) != 0 || res.FinishReason != FinishStop { + t.Errorf("stream = %+v", res) + } + if len(deltas) == 0 { + t.Error("expected streamed deltas") + } + for _, d := range deltas { + if d.Kind != DeltaContent { + t.Errorf("JSON mode must stream content deltas only, got %+v", d) + } + } +} + +// Quirk OFF on the same wire reply: no folding, native path. +func TestM4Quirk_OffDoesNotFold(t *testing.T) { + c, body := m4QuirkGateway(t, false, m4ToolReply) + res, err := c.Call(context.Background(), m4JSONReq()) + if err != nil { + t.Fatal(err) + } + if res.Content != "" || len(res.ToolCalls) != 1 { + t.Errorf("quirk off must not fold: %+v", res) + } + if (*body)["tool_choice"] != nil { + t.Errorf("quirk off must not force a tool: %v", (*body)["tool_choice"]) + } +} + +// Quirk ON: the four pre-change ConfigErrors (plus name collision). +func TestM4Quirk_EmulationConfigErrors(t *testing.T) { + rf := &ResponseFormat{Type: ResponseJSONSchema, Name: "f", Schema: testSchema} + msgs := []Message{{Role: RoleUser, Content: "x"}} + cases := []struct { + name string + req *ChatRequest + want string + }{ + {"tool name collision", &ChatRequest{Messages: msgs, Tools: []ToolDef{{Name: "f"}}, ResponseFormat: rf}, "Tools"}, + {"ToolChoice", &ChatRequest{Messages: msgs, ToolChoice: &ToolChoice{Mode: ToolChoiceAuto}, ResponseFormat: rf}, "ToolChoice"}, + {"Tools", &ChatRequest{Messages: msgs, Tools: []ToolDef{{Name: "g"}}, ResponseFormat: rf}, "Tools"}, + {"thinking budget", &ChatRequest{Messages: msgs, Thinking: "low", ThinkingBudget: 2000, + ResponseFormat: &ResponseFormat{Type: ResponseJSONObject}}, "extended thinking"}, + {"adaptive thinking", &ChatRequest{Messages: msgs, Thinking: "low", + ResponseFormat: &ResponseFormat{Type: ResponseJSONObject}}, "extended thinking"}, + {"non-object schema", &ChatRequest{Messages: msgs, + ResponseFormat: &ResponseFormat{Type: ResponseJSONSchema, Schema: json.RawMessage(`{"type":"array"}`)}}, "object"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + pc := newProviderClient(ProviderConfig{ID: "gw", Format: FormatAnthropic, BaseURL: "https://x", + Quirks: Quirks{AnthropicToolJSON: true}}, nil, nil) + _, _, err := pc.buildChatRequest(tc.req, "claude", false) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Fatalf("err = %v, want ConfigError", err) + } + if !strings.Contains(ce.Error(), tc.want) { + t.Errorf("err = %q, want mention of %q", ce.Error(), tc.want) + } + }) + } +} diff --git a/m4_thinking_test.go b/m4_thinking_test.go new file mode 100644 index 0000000..1ddbaa7 --- /dev/null +++ b/m4_thinking_test.go @@ -0,0 +1,187 @@ +package llm + +import ( + "context" + "encoding/json" + "errors" + "io" + "net/http" + "net/http/httptest" + "sync" + "testing" +) + +// anthropicWireBody sends req through a real ChatClient to an httptest server +// and returns the decoded Anthropic request body that hit the wire. +func anthropicWireBody(t *testing.T, req *ChatRequest, model string) map[string]any { + t.Helper() + var mu sync.Mutex + var raw []byte + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + b, _ := io.ReadAll(r.Body) + mu.Lock() + raw = b + mu.Unlock() + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"content":[{"type":"text","text":"ok"}],"stop_reason":"end_turn","usage":{"input_tokens":1,"output_tokens":1}}`)) + })) + defer srv.Close() + cc := newTestClient(t, ProviderConfig{ID: "anthropic", Format: FormatAnthropic, BaseURL: srv.URL, APIKey: "k"}, srv) + cc.model = model + if _, err := cc.Call(context.Background(), req); err != nil { + t.Fatalf("Call: %v", err) + } + mu.Lock() + defer mu.Unlock() + var m map[string]any + if err := json.Unmarshal(raw, &m); err != nil { + t.Fatalf("wire body %q: %v", raw, err) + } + return m +} + +func userMsgs() []Message { return []Message{{Role: RoleUser, Content: "x"}} } + +// AC-1: presets map to adaptive thinking + output_config.effort on the wire. +func TestM4_AdaptivePresetsOnWire(t *testing.T) { + cases := []struct { + level string + wantThink bool + wantEffort string + }{ + {"enabled", true, ""}, + {"low", true, "low"}, + {"medium", true, "medium"}, + {"high", true, "high"}, + {"max", true, "max"}, + {"", false, ""}, + {"disabled", false, ""}, + } + for _, tc := range cases { + t.Run("level="+tc.level, func(t *testing.T) { + m := anthropicWireBody(t, &ChatRequest{Messages: userMsgs(), Thinking: tc.level}, "claude-any") + th, has := m["thinking"].(map[string]any) + if has != tc.wantThink { + t.Fatalf("thinking present = %v, want %v (%v)", has, tc.wantThink, m["thinking"]) + } + if has { + if len(th) != 1 || th["type"] != "adaptive" { + t.Errorf("thinking = %v, want exactly {type:adaptive}", th) + } + } + oc, hasOC := m["output_config"].(map[string]any) + if tc.wantEffort == "" { + if hasOC { + t.Errorf("output_config = %v, want absent", oc) + } + } else if oc["effort"] != tc.wantEffort { + t.Errorf("effort = %v, want %q", oc["effort"], tc.wantEffort) + } + }) + } +} + +// AC-2: an explicit ThinkingBudget keeps the legacy path, with no effort. +func TestM4_ExplicitBudgetKeepsLegacyPath(t *testing.T) { + for _, level := range []string{"enabled", "low", "high", "max"} { + m := anthropicWireBody(t, &ChatRequest{Messages: userMsgs(), Thinking: level, ThinkingBudget: 4000}, "claude-any") + th := m["thinking"].(map[string]any) + if th["type"] != "enabled" || th["budget_tokens"] != float64(4000) { + t.Errorf("%s: thinking = %v, want enabled/4000", level, th) + } + if m["max_tokens"].(float64) <= 4000 { + t.Errorf("%s: max_tokens %v must exceed budget", level, m["max_tokens"]) + } + if _, ok := m["output_config"]; ok { + t.Errorf("%s: output_config must be absent on the budget path: %v", level, m["output_config"]) + } + } + // ThinkingBudget with thinking off is ignored. + for _, level := range []string{"", "disabled"} { + m := anthropicWireBody(t, &ChatRequest{Messages: userMsgs(), Thinking: level, ThinkingBudget: 4000}, "claude-any") + if _, ok := m["thinking"]; ok { + t.Errorf("level %q: thinking must be absent, got %v", level, m["thinking"]) + } + } + // Existing ConfigErrors still fire. + _, err := buildAnthropicRequest(&ChatRequest{Messages: userMsgs(), Thinking: "high", ThinkingBudget: 4000, MaxTokens: 4000}, "claude", false) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Errorf("budget >= MaxTokens: err = %v, want ConfigError", err) + } +} + +// AC-3: adaptive default max_tokens 16384, explicit as-is, sampling removed. +func TestM4_AdaptiveMaxTokensAndSampling(t *testing.T) { + cases := []struct { + name string + req ChatRequest + wantMax float64 + }{ + {"adaptive default", ChatRequest{Thinking: "medium"}, 16384}, + {"adaptive explicit small", ChatRequest{Thinking: "high", MaxTokens: 512}, 512}, + {"adaptive explicit large", ChatRequest{Thinking: "max", MaxTokens: 64000}, 64000}, + {"no thinking default", ChatRequest{}, 8192}, + {"disabled default", ChatRequest{Thinking: "disabled"}, 8192}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + tc.req.Messages = userMsgs() + m := anthropicWireBody(t, &tc.req, "claude-any") + if m["max_tokens"] != tc.wantMax { + t.Errorf("max_tokens = %v, want %v", m["max_tokens"], tc.wantMax) + } + }) + } + for _, req := range []ChatRequest{{Thinking: "medium"}, {Thinking: "medium", ThinkingBudget: 2048}} { + req.Messages, req.Temperature, req.TopP = userMsgs(), 0.4, 0.6 + m := anthropicWireBody(t, &req, "claude-any") + if _, ok := m["temperature"]; ok { + t.Errorf("budget=%d: temperature must be absent with thinking", req.ThinkingBudget) + } + if _, ok := m["top_p"]; ok { + t.Errorf("budget=%d: top_p must be absent with thinking", req.ThinkingBudget) + } + } + m := anthropicWireBody(t, &ChatRequest{Messages: userMsgs(), Temperature: 0.4, TopP: 0.6}, "claude-any") + if m["temperature"] != 0.4 || m["top_p"] != 0.6 { + t.Errorf("sampling must survive without thinking: %v %v", m["temperature"], m["top_p"]) + } +} + +// AC-6: forced tool choice passes through on adaptive thinking; the rejection +// fires only on the legacy budget path; the result never depends on the model name. +func TestM4_ForcedToolChoiceAdaptiveVsBudget(t *testing.T) { + tools := []ToolDef{{Name: "f", Parameters: json.RawMessage(`{"type":"object"}`)}} + choices := []struct { + name string + tc *ToolChoice + wantType string + }{ + {"required", &ToolChoice{Mode: ToolChoiceRequired}, "any"}, + {"named", ToolChoiceNamed("f"), "tool"}, + } + for _, model := range []string{"claude-opus-4-8", "claude-haiku-4-5", "totally-unknown-model"} { + for _, c := range choices { + t.Run(model+"/"+c.name, func(t *testing.T) { + m := anthropicWireBody(t, &ChatRequest{Messages: userMsgs(), Tools: tools, ToolChoice: c.tc, Thinking: "high"}, model) + tcm, _ := m["tool_choice"].(map[string]any) + if tcm["type"] != c.wantType { + t.Errorf("tool_choice = %v, want type %s", tcm, c.wantType) + } + if th := m["thinking"].(map[string]any); th["type"] != "adaptive" { + t.Errorf("thinking = %v, want adaptive", th) + } + _, err := buildAnthropicRequest(&ChatRequest{Messages: userMsgs(), Tools: tools, ToolChoice: c.tc, Thinking: "high", ThinkingBudget: 2000}, model, false) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Errorf("budget path: err = %v, want ConfigError", err) + } + }) + } + } + // auto + budget thinking stays accepted. + if _, err := buildAnthropicRequest(&ChatRequest{Messages: userMsgs(), Tools: tools, ToolChoice: &ToolChoice{Mode: ToolChoiceAuto}, Thinking: "low", ThinkingBudget: 2000}, "claude", false); err != nil { + t.Errorf("auto + budget thinking must pass: %v", err) + } +} diff --git a/m5_embed_test.go b/m5_embed_test.go new file mode 100644 index 0000000..d4a1f31 --- /dev/null +++ b/m5_embed_test.go @@ -0,0 +1,109 @@ +package llm + +import ( + "context" + "encoding/base64" + "encoding/binary" + "encoding/json" + "fmt" + "io" + "math" + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +// b64Vec encodes v as little-endian float32, base64 (OpenAI's +// encoding_format=base64 wire shape). +func b64Vec(v []float32) string { + buf := make([]byte, 4*len(v)) + for i, f := range v { + binary.LittleEndian.PutUint32(buf[4*i:], math.Float32bits(f)) + } + return base64.StdEncoding.EncodeToString(buf) +} + +// M5 bug 1: OpenAI-compatible Embed must request encoding_format=base64 (a +// 2048x1536 float-array response overflows the 50MB body cap) and decode +// little-endian float32 base64, while still accepting float arrays from +// providers that ignore the parameter. +func TestM5Embed_OpenAIBase64(t *testing.T) { + vecs := [][]float32{{0.5, -1.25, 3}, {8, 0.015625, -2}} + asArray := func(v []float32) string { + parts := make([]string, len(v)) + for i, f := range v { + parts[i] = fmt.Sprintf("%v", f) + } + return "[" + strings.Join(parts, ",") + "]" + } + cases := []struct { + name string + body func() string + }{ + {"base64 response", func() string { + return fmt.Sprintf(`{"data":[{"index":1,"embedding":%q},{"index":0,"embedding":%q}],"usage":{"prompt_tokens":4}}`, + b64Vec(vecs[1]), b64Vec(vecs[0])) + }}, + {"float-array fallback (provider ignores encoding_format)", func() string { + return fmt.Sprintf(`{"data":[{"index":1,"embedding":%s},{"index":0,"embedding":%s}],"usage":{"prompt_tokens":4}}`, + asArray(vecs[1]), asArray(vecs[0])) + }}, + {"mixed per item", func() string { + return fmt.Sprintf(`{"data":[{"index":0,"embedding":%q},{"index":1,"embedding":%s}],"usage":{"prompt_tokens":4}}`, + b64Vec(vecs[0]), asArray(vecs[1])) + }}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + var got map[string]any + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + b, _ := io.ReadAll(r.Body) + _ = json.Unmarshal(b, &got) + _, _ = fmt.Fprint(w, tc.body()) + })) + t.Cleanup(srv.Close) + sdk := New(WithProvider("openai", WithAPIKey("k"), WithBaseURL(srv.URL))) + res, err := sdk.Embed(context.Background(), "openai", "emb", EmbedRequest{Inputs: []string{"a", "b"}}) + if got["encoding_format"] != "base64" { + t.Errorf("wire encoding_format = %v, want base64", got["encoding_format"]) + } + if err != nil { + t.Fatalf("Embed: %v", err) + } + if len(res.Embeddings) != 2 { + t.Fatalf("got %d vectors", len(res.Embeddings)) + } + for i, want := range vecs { + if len(res.Embeddings[i]) != len(want) { + t.Fatalf("vector %d len = %d, want %d", i, len(res.Embeddings[i]), len(want)) + } + for j, f := range want { + if res.Embeddings[i][j] != float64(f) { + t.Errorf("vec[%d][%d] = %v, want %v", i, j, res.Embeddings[i][j], f) + } + } + } + if res.Usage.PromptTokens != 4 { + t.Errorf("usage = %+v", res.Usage) + } + }) + } +} + +// A base64 payload whose byte length is not a multiple of 4 is a protocol +// failure, never a silently truncated vector. +func TestM5Embed_OpenAIBase64BadLength(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = fmt.Fprintf(w, `{"data":[{"index":0,"embedding":%q}]}`, base64.StdEncoding.EncodeToString([]byte{1, 2, 3, 4, 5})) + })) + t.Cleanup(srv.Close) + sdk := New(WithProvider("openai", WithAPIKey("k"), WithBaseURL(srv.URL))) + res, err := sdk.Embed(context.Background(), "openai", "emb", EmbedRequest{Inputs: []string{"a"}}) + if err == nil { + t.Fatalf("want error for 5-byte base64 vector, got %v", res.Embeddings) + } + if strings.Contains(err.Error(), "json: cannot unmarshal") { + t.Errorf("want a base64-length error, got the float-array decode error: %v", err) + } +} diff --git a/m5_models_test.go b/m5_models_test.go new file mode 100644 index 0000000..452f427 --- /dev/null +++ b/m5_models_test.go @@ -0,0 +1,194 @@ +package llm + +import ( + "context" + "errors" + "fmt" + "net/http" + "net/http/httptest" + "sync/atomic" + "testing" + "time" +) + +func m5Provider(t *testing.T, id string, srv *httptest.Server, opts ...Option) *Provider { + t.Helper() + fastBackoff(t) + sdk := New(append(opts, WithProvider(id, WithAPIKey("k"), WithBaseURL(srv.URL)))...) + p, err := sdk.Provider(id) + if err != nil { + t.Fatal(err) + } + return p +} + +// M5 bug 2: a bare top-level JSON array (Together AI) is a valid listing. +func TestM5ListModels_OpenAIBareArray(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = fmt.Fprint(w, `[{"id":"a","created":1700000000,"context_length":8192},{"id":"b"}]`) + })) + t.Cleanup(srv.Close) + models, err := m5Provider(t, "openai", srv).ListModels(context.Background()) + if err != nil { + t.Fatalf("ListModels: %v", err) + } + if len(models) != 2 || models[0].ID != "a" || models[0].ContextWindow != 8192 || models[1].ID != "b" { + t.Fatalf("models = %+v", models) + } +} + +// M5 bug 3: has_more with no last_id cannot be paged; that is an error, +// never a silent partial success. +func TestM5ListModels_AnthropicHasMoreWithoutLastID(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = fmt.Fprint(w, `{"data":[{"id":"claude-x","display_name":"X","created_at":"2025-01-01T00:00:00Z"}],"has_more":true,"last_id":""}`) + })) + t.Cleanup(srv.Close) + models, err := m5Provider(t, "anthropic", srv).ListModels(context.Background()) + if err == nil { + t.Fatalf("want error for has_more without last_id, got partial %+v", models) + } + if models != nil { + t.Errorf("models = %+v alongside error", models) + } +} + +// M5 bug 4a: callers own what ListModels returns; mutating a Capabilities +// slice must never reach the cache or other callers (first and cached read). +func TestM5ListModels_CapabilitiesNotAliased(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + _, _ = fmt.Fprint(w, `{"models":[{"name":"models/g1","displayName":"G","supportedGenerationMethods":["generateContent","embedContent"]}]}`) + })) + t.Cleanup(srv.Close) + p := m5Provider(t, "gemini", srv) + ctx := context.Background() + first, err := p.ListModels(ctx) + if err != nil || len(first) != 1 || len(first[0].Capabilities) != 2 { + t.Fatalf("first = %+v, %v", first, err) + } + first[0].Capabilities[0] = "CORRUPTED" + second, err := p.ListModels(ctx) + if err != nil { + t.Fatal(err) + } + if second[0].Capabilities[0] != "generateContent" { + t.Errorf("mutation of first result leaked into cache: %v", second[0].Capabilities) + } + second[0].Capabilities[1] = "CORRUPTED" + third, _ := p.ListModels(ctx) + if third[0].Capabilities[1] != "embedContent" { + t.Errorf("mutation of cached result leaked into cache: %v", third[0].Capabilities) + } +} + +// M5 bug 4b: an empty successful listing is cached like any other, on every +// format (Anthropic/Gemini build the list from a nil slice). +func TestM5ListModels_EmptyListCached(t *testing.T) { + cases := []struct{ id, body string }{ + {"openai", `{"data":[]}`}, + {"anthropic", `{"data":[],"has_more":false,"last_id":""}`}, + {"gemini", `{"models":[]}`}, + } + for _, tc := range cases { + t.Run(tc.id, func(t *testing.T) { + var hits atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hits.Add(1) + _, _ = fmt.Fprint(w, tc.body) + })) + t.Cleanup(srv.Close) + p := m5Provider(t, tc.id, srv) + for i := 0; i < 3; i++ { + models, err := p.ListModels(context.Background()) + if err != nil || len(models) != 0 { + t.Fatalf("call %d: %v, %v", i, models, err) + } + } + if n := hits.Load(); n != 1 { + t.Errorf("upstream hits = %d, want 1 (empty list must be cached)", n) + } + }) + } +} + +// M5 bug 5: ListModels follows WithRetryPolicy; unconfigured it keeps the +// documented 3 attempts. +func TestM5ListModels_RetryPolicyAttempts(t *testing.T) { + cases := []struct { + name string + opts []Option + want int32 + }{ + {"default 3 attempts", nil, 3}, + {"MaxAttempts 2", []Option{WithRetryPolicy(RetryPolicy{MaxAttempts: 2})}, 2}, + {"MaxAttempts 5", []Option{WithRetryPolicy(RetryPolicy{MaxAttempts: 5})}, 5}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + srv, n := countingServer(t, http.StatusInternalServerError, nil) + _, err := m5Provider(t, "openai", srv, tc.opts...).ListModels(context.Background()) + if err == nil { + t.Fatal("want error") + } + if got := n.Load(); got != tc.want { + t.Errorf("attempts = %d, want %d", got, tc.want) + } + }) + } +} + +// M5 bug 5: Retry-After on a retryable status is honoured, not replaced by +// a computed backoff. +func TestM5ListModels_HonoursRetryAfter(t *testing.T) { + var hits atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if hits.Add(1) == 1 { + w.Header().Set("Retry-After", "1") + w.WriteHeader(http.StatusTooManyRequests) + _, _ = fmt.Fprint(w, `{"error":{"message":"slow down"}}`) + return + } + _, _ = fmt.Fprint(w, `{"data":[{"id":"a"}]}`) + })) + t.Cleanup(srv.Close) + p := m5Provider(t, "openai", srv) + start := time.Now() + models, err := p.ListModels(context.Background()) + if err != nil || len(models) != 1 { + t.Fatalf("models=%v err=%v", models, err) + } + if el := time.Since(start); el < 900*time.Millisecond { + t.Errorf("retried after %v, want >= ~1s per Retry-After", el) + } +} + +// M5 bug 5: WithRequestTimeout bounds the whole ListModels call, retries +// included. +func TestM5ListModels_RequestTimeoutBoundsCall(t *testing.T) { + var hits atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hits.Add(1) + tm := time.NewTimer(3 * time.Second) + defer tm.Stop() + select { + case <-r.Context().Done(): + case <-tm.C: + } + })) + t.Cleanup(srv.Close) + p := m5Provider(t, "openai", srv, + WithRequestTimeout(300*time.Millisecond), + WithRetryPolicy(RetryPolicy{MaxAttempts: 5})) + start := time.Now() + _, err := p.ListModels(context.Background()) + el := time.Since(start) + if err == nil { + t.Fatal("want timeout error") + } + if el > 1500*time.Millisecond { + t.Errorf("call took %v, want ~300ms whole-call budget", el) + } + if errors.Is(err, context.Canceled) { + t.Errorf("err = %v", err) + } +} diff --git a/m6_headers_followup_test.go b/m6_headers_followup_test.go new file mode 100644 index 0000000..d384f7a --- /dev/null +++ b/m6_headers_followup_test.go @@ -0,0 +1,95 @@ +package llm + +import ( + "errors" + "strings" + "testing" +) + +func hdrFollowEntries() []struct { + name string + call func(t *testing.T, s *SDK, cc *ChatClient, h map[string]string) error +} { + return []struct { + name string + call func(t *testing.T, s *SDK, cc *ChatClient, h map[string]string) error + }{ + {"chat", func(t *testing.T, _ *SDK, cc *ChatClient, h map[string]string) error { + req := m2Req() + req.Headers = h + _, err := cc.Call(t.Context(), req) + return err + }}, + {"stream", func(t *testing.T, _ *SDK, cc *ChatClient, h map[string]string) error { + req := m2Req() + req.Headers = h + _, err := cc.CallStream(t.Context(), req, func(Delta) error { return nil }) + return err + }}, + {"embed", func(t *testing.T, s *SDK, _ *ChatClient, h map[string]string) error { + _, err := s.Embed(t.Context(), "openai", "m", EmbedRequest{Inputs: []string{"a"}, Headers: h}) + return err + }}, + {"speak", func(t *testing.T, s *SDK, _ *ChatClient, h map[string]string) error { + _, err := s.Speak(t.Context(), "openai", "tts-1", SpeakRequest{Text: "hi", Voice: "alloy", Headers: h}) + return err + }}, + {"transcribe", func(t *testing.T, s *SDK, _ *ChatClient, h map[string]string) error { + _, err := s.Transcribe(t.Context(), "openai", "whisper-1", TranscribeRequest{Audio: []byte{1}, Headers: h}) + return err + }}, + } +} + +// Two keys equal case-insensitively are ambiguous and rejected; so are +// values with ASCII control bytes other than HTAB (including DEL). +func TestM6_HeadersFollowupRejected(t *testing.T) { + const secretVal = "SECRET-VALUE-77c1" + const key = "key-must-not-leak" + bads := []struct { + name string + h map[string]string + }{ + {"duplicate differing case", map[string]string{"X-Dup": secretVal, "x-dup": secretVal + "2"}}, + {"duplicate upper and lower", map[string]string{"X-TRACE": secretVal, "x-trace": ""}}, + {"SOH in value", map[string]string{"X-Ok": secretVal + "\x01"}}, + {"BEL in value", map[string]string{"X-Ok": secretVal + "\x07"}}, + {"US in value", map[string]string{"X-Ok": secretVal + "\x1f"}}, + {"DEL in value", map[string]string{"X-Ok": secretVal + "\x7f"}}, + } + for _, e := range hdrFollowEntries() { + for _, b := range bads { + t.Run(e.name+"/"+b.name, func(t *testing.T) { + srv, rec := hdrSrv(t, false, "application/json", hdrOpenAIBody) + cfg := ProviderConfig{ID: "openai", Format: FormatOpenAI, APIKey: key} + s := newTestSDK(t, cfg, srv) + cfg.BaseURL = srv.URL + cc := newTestClient(t, cfg, srv) + err := e.call(t, s, cc, b.h) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Fatalf("err = %v (%T), want *ConfigError", err, err) + } + if n := len(rec.all()); n != 0 { + t.Errorf("%d HTTP requests made, want 0", n) + } + if msg := err.Error(); strings.Contains(msg, secretVal) || strings.Contains(msg, key) { + t.Errorf("error leaks value or key: %q", msg) + } + }) + } + } +} + +// HTAB and printable/obs-text bytes remain valid in header values. +func TestM6_HeadersFollowupTabAllowed(t *testing.T) { + srv, rec := hdrSrv(t, false, "application/json", hdrOpenAIBody) + cc := newTestClient(t, ProviderConfig{ID: "openai", Format: FormatOpenAI, BaseURL: srv.URL, APIKey: "k"}, srv) + req := m2Req() + req.Headers = map[string]string{"X-Tab": "a\tb", "X-Other": "v"} + m2Run(t, cc, req, false) + got := rec.all() + if len(got) != 1 || got[0].Get("X-Tab") != "a\tb" || got[0].Get("X-Other") != "v" { + t.Errorf("headers = %v", got) + } +} diff --git a/m6_headers_test.go b/m6_headers_test.go new file mode 100644 index 0000000..2f9741d --- /dev/null +++ b/m6_headers_test.go @@ -0,0 +1,331 @@ +package llm + +import ( + "errors" + "net/http" + "net/http/httptest" + "strings" + "sync" + "sync/atomic" + "testing" +) + +// hdrRec records every request's headers; when failFirst is set the first +// request is a 500 so the per-attempt application of headers is observable. +type hdrRec struct { + mu sync.Mutex + seen []http.Header + hits atomic.Int32 +} + +func (r *hdrRec) all() []http.Header { + r.mu.Lock() + defer r.mu.Unlock() + return append([]http.Header(nil), r.seen...) +} + +func hdrSrv(t *testing.T, failFirst bool, ctype, body string) (*httptest.Server, *hdrRec) { + t.Helper() + rec := &hdrRec{} + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + rec.mu.Lock() + rec.seen = append(rec.seen, r.Header.Clone()) + rec.mu.Unlock() + if failFirst && rec.hits.Add(1) == 1 { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusInternalServerError) + _, _ = w.Write([]byte(`{"error":{"message":"boom"}}`)) + return + } + w.Header().Set("Content-Type", ctype) + _, _ = w.Write([]byte(body)) + })) + t.Cleanup(srv.Close) + return srv, rec +} + +const ( + hdrOpenAIBody = `{"id":"c1","choices":[{"message":{"role":"assistant","content":"hi"},"finish_reason":"stop"}]}` + hdrRespBody = `{"id":"resp_1","status":"completed","output":[{"type":"message","content":[{"type":"output_text","text":"hi"}]}]}` + hdrAnthBody = `{"id":"msg_1","type":"message","role":"assistant","content":[{"type":"text","text":"hi"}],"stop_reason":"end_turn","usage":{"input_tokens":1,"output_tokens":1}}` + hdrGemBody = `{"candidates":[{"content":{"parts":[{"text":"hi"}]},"finishReason":"STOP"}]}` +) + +func hdrSSE() string { + return metaSSE(`{"id":"c1","choices":[{"delta":{"content":"hi"},"finish_reason":"stop"}]}`, "[DONE]") +} + +// Per-request headers reach the server on every attempt, buffered, streaming +// and Responses-diverted, on every chat format. +func TestM6_HeadersChatEveryPathEveryAttempt(t *testing.T) { + cases := []struct { + name string + cfg ProviderConfig + model string + stream bool + ctype string + body string + }{ + {"openai buffered", ProviderConfig{ID: "openai", Format: FormatOpenAI}, "", false, "application/json", hdrOpenAIBody}, + {"openai stream", ProviderConfig{ID: "openai", Format: FormatOpenAI}, "", true, "text/event-stream", hdrSSE()}, + {"responses diverted buffered", ProviderConfig{ID: "openai", Format: FormatOpenAI}, "gpt-5.6-luna", false, "application/json", hdrRespBody}, + {"anthropic buffered", ProviderConfig{ID: "anthropic", Format: FormatAnthropic}, "", false, "application/json", hdrAnthBody}, + {"gemini buffered", ProviderConfig{ID: "gemini", Format: FormatGemini}, "", false, "application/json", hdrGemBody}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + srv, rec := hdrSrv(t, true, c.ctype, c.body) + cfg := c.cfg + cfg.BaseURL, cfg.APIKey = srv.URL, "k" + cc := newTestClient(t, cfg, srv) + if c.model != "" { + cc.model = c.model + } + req := m2Req() + req.Headers = map[string]string{"X-Trace": "t-1", "anthropic-beta": "beta-x"} + if _, _ = m2Run(t, cc, req, c.stream); false { + return + } + got := rec.all() + if len(got) < 2 { + t.Fatalf("attempts = %d, want >= 2", len(got)) + } + for i, h := range got { + if h.Get("X-Trace") != "t-1" || h.Get("Anthropic-Beta") != "beta-x" { + t.Errorf("attempt %d headers = %v, want X-Trace and anthropic-beta", i+1, h) + } + } + }) + } +} + +// Per-request headers win over provider headers; an empty value removes a +// non-reserved provider header; neither leaks into the next call. +func TestM6_HeadersMergeOrderAndNoLeak(t *testing.T) { + srv, rec := hdrSrv(t, false, "application/json", hdrOpenAIBody) + cfg := ProviderConfig{ID: "openai", Format: FormatOpenAI, BaseURL: srv.URL, APIKey: "k", + Headers: map[string]string{"X-Over": "provider", "X-Drop": "provider", "X-Keep": "provider"}} + cc := newTestClient(t, cfg, srv) + + req := m2Req() + req.Headers = map[string]string{"X-Over": "request", "X-Drop": "", "X-New": "n"} + m2Run(t, cc, req, false) + m2Run(t, cc, m2Req(), false) + + got := rec.all() + if len(got) != 2 { + t.Fatalf("requests = %d, want 2", len(got)) + } + h := got[0] + if h.Get("X-Over") != "request" || h.Get("X-New") != "n" || h.Get("X-Keep") != "provider" { + t.Errorf("first call headers = %v", h) + } + if _, ok := h["X-Drop"]; ok { + t.Errorf("empty value did not remove X-Drop: %v", h) + } + h = got[1] + if h.Get("X-Over") != "provider" || h.Get("X-Drop") != "provider" || h.Get("X-New") != "" { + t.Errorf("second call leaked per-request headers: %v", h) + } +} + +// Auth stays intact when per-request headers are present. +func TestM6_HeadersAuthIntact(t *testing.T) { + srv, rec := hdrSrv(t, false, "application/json", hdrOpenAIBody) + cc := newTestClient(t, ProviderConfig{ID: "openai", Format: FormatOpenAI, BaseURL: srv.URL, APIKey: "sekret"}, srv) + req := m2Req() + req.Headers = map[string]string{"X-A": "b"} + m2Run(t, cc, req, false) + got := rec.all() + if len(got) != 1 || got[0].Get("X-A") != "b" || got[0].Get("Authorization") != "Bearer sekret" { + t.Errorf("headers = %v", got) + } +} + +// Embed, Speak and Transcribe honour per-request headers on every attempt +// and do not leak them into the next call. +func TestM6_HeadersNonChatEntryPoints(t *testing.T) { + embedOpenAI := `{"data":[{"index":0,"embedding":[0.5,1]}],"usage":{"prompt_tokens":2}}` + embedGemini := `{"embeddings":[{"values":[0.5,1]}]}` + geminiAudio := `{"candidates":[{"content":{"parts":[{"inlineData":{"mimeType":"audio/L16;rate=24000","data":"AQID"}}]}}]}` + hd := map[string]string{"X-Trace": "t-2"} + runs := []struct { + name string + format Format + id string + ctype string + body string + call func(s *SDK, id string, h map[string]string) error + }{ + {"embed openai", FormatOpenAI, "openai", "application/json", embedOpenAI, func(s *SDK, id string, h map[string]string) error { + _, err := s.Embed(t.Context(), id, "m", EmbedRequest{Inputs: []string{"a"}, Headers: h}) + return err + }}, + {"embed gemini", FormatGemini, "gemini", "application/json", embedGemini, func(s *SDK, id string, h map[string]string) error { + _, err := s.Embed(t.Context(), id, "m", EmbedRequest{Inputs: []string{"a"}, Headers: h}) + return err + }}, + {"speak openai", FormatOpenAI, "openai", "audio/mpeg", "\x01\x02", func(s *SDK, id string, h map[string]string) error { + _, err := s.Speak(t.Context(), id, "tts-1", SpeakRequest{Text: "hi", Voice: "alloy", Headers: h}) + return err + }}, + {"speak gemini", FormatGemini, "gemini", "application/json", geminiAudio, func(s *SDK, id string, h map[string]string) error { + _, err := s.Speak(t.Context(), id, "gemini-tts", SpeakRequest{Text: "hi", Voice: "Kore", Headers: h}) + return err + }}, + {"transcribe openai", FormatOpenAI, "openai", "application/json", `{"text":"hi"}`, func(s *SDK, id string, h map[string]string) error { + _, err := s.Transcribe(t.Context(), id, "whisper-1", TranscribeRequest{Audio: []byte{1, 2}, Headers: h}) + return err + }}, + } + for _, r := range runs { + t.Run(r.name, func(t *testing.T) { + srv, rec := hdrSrv(t, true, r.ctype, r.body) + s := newTestSDK(t, ProviderConfig{ID: r.id, Format: r.format, APIKey: "k"}, srv) + if err := r.call(s, r.id, hd); err != nil { + t.Fatal(err) + } + got := rec.all() + if len(got) < 2 { + t.Fatalf("attempts = %d, want >= 2", len(got)) + } + for i, h := range got { + if h.Get("X-Trace") != "t-2" { + t.Errorf("attempt %d missing X-Trace: %v", i+1, h) + } + } + n := len(got) + if err := r.call(s, r.id, nil); err != nil { + t.Fatal(err) + } + if last := rec.all()[n]; last.Get("X-Trace") != "" { + t.Errorf("per-request header leaked into next call: %v", last) + } + }) + } +} + +// Reserved names (any case) and malformed names/values are rejected with a +// *ConfigError before any HTTP request; the error never carries the value +// or the API key. +func TestM6_HeadersRejectedBeforeNetwork(t *testing.T) { + const secretVal = "SECRET-VALUE-9f3a" + const key = "key-must-not-leak" + reserved := []string{"Authorization", "authorization", "AUTHORIZATION", "Proxy-Authorization", "X-Api-Key", "x-api-key", + "X-Goog-Api-Key", "x-goog-api-key", "Host", "Content-Length", "Content-Type", "content-type", "Transfer-Encoding"} + type bad struct { + name string + h map[string]string + namedInErr string // header name that must appear in the error ("" = not asserted) + } + var bads []bad + for _, n := range reserved { + bads = append(bads, bad{"reserved " + n, map[string]string{n: secretVal}, n}) + } + bads = append(bads, + bad{"reserved with empty value", map[string]string{"Authorization": ""}, "Authorization"}, + bad{"empty name", map[string]string{"": secretVal}, ""}, + bad{"space in name", map[string]string{"X Bad": secretVal}, ""}, + bad{"colon in name", map[string]string{"X:Bad": secretVal}, ""}, + bad{"CR in name", map[string]string{"X-Bad\r": secretVal}, ""}, + bad{"LF in name", map[string]string{"X-Bad\n": secretVal}, ""}, + bad{"NUL in name", map[string]string{"X-Bad\x00": secretVal}, ""}, + bad{"non-ASCII name", map[string]string{"X-Bäd": secretVal}, ""}, + bad{"CR in value", map[string]string{"X-Ok": secretVal + "\rX-Evil: 1"}, ""}, + bad{"LF in value", map[string]string{"X-Ok": secretVal + "\nX-Evil: 1"}, ""}, + bad{"CRLF in value", map[string]string{"X-Ok": secretVal + "\r\nX-Evil: 1"}, ""}, + bad{"NUL in value", map[string]string{"X-Ok": secretVal + "\x00"}, ""}, + ) + entries := []struct { + name string + call func(t *testing.T, s *SDK, cc *ChatClient, h map[string]string) error + }{ + {"chat", func(t *testing.T, _ *SDK, cc *ChatClient, h map[string]string) error { + req := m2Req() + req.Headers = h + _, err := cc.Call(t.Context(), req) + return err + }}, + {"stream", func(t *testing.T, _ *SDK, cc *ChatClient, h map[string]string) error { + req := m2Req() + req.Headers = h + _, err := cc.CallStream(t.Context(), req, func(Delta) error { return nil }) + return err + }}, + {"responses", func(t *testing.T, _ *SDK, cc *ChatClient, h map[string]string) error { + cc.model = "gpt-5.6-luna" + req := m2Req() + req.Headers = h + _, err := cc.Call(t.Context(), req) + return err + }}, + {"embed", func(t *testing.T, s *SDK, _ *ChatClient, h map[string]string) error { + _, err := s.Embed(t.Context(), "openai", "m", EmbedRequest{Inputs: []string{"a"}, Headers: h}) + return err + }}, + {"speak", func(t *testing.T, s *SDK, _ *ChatClient, h map[string]string) error { + _, err := s.Speak(t.Context(), "openai", "tts-1", SpeakRequest{Text: "hi", Voice: "alloy", Headers: h}) + return err + }}, + {"transcribe", func(t *testing.T, s *SDK, _ *ChatClient, h map[string]string) error { + _, err := s.Transcribe(t.Context(), "openai", "whisper-1", TranscribeRequest{Audio: []byte{1}, Headers: h}) + return err + }}, + } + for _, e := range entries { + for _, b := range bads { + t.Run(e.name+"/"+b.name, func(t *testing.T) { + srv, rec := hdrSrv(t, false, "application/json", hdrOpenAIBody) + cfg := ProviderConfig{ID: "openai", Format: FormatOpenAI, APIKey: key} + s := newTestSDK(t, cfg, srv) + cfg.BaseURL = srv.URL + cc := newTestClient(t, cfg, srv) + err := e.call(t, s, cc, b.h) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Fatalf("err = %v (%T), want *ConfigError", err, err) + } + if n := len(rec.all()); n != 0 { + t.Errorf("%d HTTP requests made, want 0", n) + } + msg := err.Error() + if strings.Contains(msg, secretVal) || strings.Contains(msg, key) { + t.Errorf("error text leaks value or key: %q", msg) + } + if b.namedInErr != "" && !strings.Contains(strings.ToLower(msg), strings.ToLower(b.namedInErr)) { + t.Errorf("error %q does not name header %q", msg, b.namedInErr) + } + }) + } + } +} + +// anthropic-version and anthropic-beta are not reserved; a provider header +// that is not reserved can be set by name in any case. +func TestM6_HeadersAnthropicVersionOverridable(t *testing.T) { + srv, rec := hdrSrv(t, false, "application/json", hdrAnthBody) + cc := newTestClient(t, ProviderConfig{ID: "anthropic", Format: FormatAnthropic, BaseURL: srv.URL, APIKey: "k"}, srv) + req := m2Req() + req.Headers = map[string]string{"Anthropic-Version": "2099-01-01"} + m2Run(t, cc, req, false) + got := rec.all() + if len(got) != 1 || got[0].Get("Anthropic-Version") != "2099-01-01" || got[0].Get("X-Api-Key") != "k" { + t.Errorf("headers = %v", got) + } +} + +// Nil and empty maps change nothing (AC-5). +func TestM6_HeadersNilEmptyNoChange(t *testing.T) { + for _, h := range []map[string]string{nil, {}} { + srv, rec := hdrSrv(t, false, "application/json", hdrOpenAIBody) + cc := newTestClient(t, ProviderConfig{ID: "openai", Format: FormatOpenAI, BaseURL: srv.URL, APIKey: "k", + Headers: map[string]string{"X-P": "p"}}, srv) + req := m2Req() + req.Headers = h + m2Run(t, cc, req, false) + got := rec.all() + if len(got) != 1 || got[0].Get("X-P") != "p" || got[0].Get("Authorization") != "Bearer k" { + t.Errorf("headers = %v", got) + } + } +} diff --git a/m6_meta_test.go b/m6_meta_test.go new file mode 100644 index 0000000..d60320b --- /dev/null +++ b/m6_meta_test.go @@ -0,0 +1,365 @@ +package llm + +import ( + "net/http" + "net/http/httptest" + "strings" + "sync/atomic" + "testing" + "time" +) + +// metaSrv answers with body and hdr. When failFirst is set the first request +// is a 500 carrying poisoned headers that must never reach the result. +func metaSrv(t *testing.T, failFirst bool, ctype string, hdr map[string]string, body string) *httptest.Server { + t.Helper() + var hits atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if failFirst && hits.Add(1) == 1 { + w.Header().Set("X-Request-Id", "req-poison") + w.Header().Set("X-Ratelimit-Remaining-Requests", "1") + w.Header().Set("X-Ratelimit-Remaining-Tokens", "2") + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusInternalServerError) + _, _ = w.Write([]byte(`{"error":{"message":"boom"}}`)) + return + } + for k, v := range hdr { + w.Header().Set(k, v) + } + w.Header().Set("Content-Type", ctype) + _, _ = w.Write([]byte(body)) + })) + t.Cleanup(srv.Close) + return srv +} + +var metaFullHdr = map[string]string{ + "X-Request-Id": "req-123", + "X-Ratelimit-Remaining-Requests": "99", + "X-Ratelimit-Remaining-Tokens": "4000", + "X-Ratelimit-Reset-Requests": "1s", + "X-Ratelimit-Reset-Tokens": "6m0s", +} + +var metaFullRL = RateLimit{Reported: true, RemainingRequests: 99, RemainingTokens: 4000, + ResetRequests: time.Second, ResetTokens: 6 * time.Minute} + +func metaSSE(events ...string) string { + var b strings.Builder + for _, e := range events { + b.WriteString("data: " + e + "\n\n") + } + return b.String() +} + +func TestM6_MetaChatEveryFormat(t *testing.T) { + anthStream := "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"id\":\"msg_1\",\"model\":\"claude-x\",\"usage\":{\"input_tokens\":1}}}\n\n" + + "event: content_block_start\ndata: {\"type\":\"content_block_start\",\"index\":0,\"content_block\":{\"type\":\"text\",\"text\":\"\"}}\n\n" + + "event: content_block_delta\ndata: {\"type\":\"content_block_delta\",\"index\":0,\"delta\":{\"type\":\"text_delta\",\"text\":\"hi\"}}\n\n" + + "event: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"end_turn\"},\"usage\":{\"output_tokens\":1}}\n\n" + + "event: message_stop\ndata: {\"type\":\"message_stop\"}\n\n" + cases := []struct { + name string + format Format + id, model string + stream bool + body string + wantID string + wantModel string + }{ + {"openai buffered", FormatOpenAI, "openai", "", false, + `{"id":"chatcmpl-1","model":"gpt-x","choices":[{"message":{"role":"assistant","content":"hi"},"finish_reason":"stop"}]}`, + "chatcmpl-1", "gpt-x"}, + {"openai stream", FormatOpenAI, "openai", "", true, + metaSSE(`{"id":"chatcmpl-2","model":"gpt-y","choices":[{"delta":{"content":"hi"}}]}`, + `{"id":"chatcmpl-2","model":"gpt-y","choices":[{"delta":{},"finish_reason":"stop"}]}`, "[DONE]"), + "chatcmpl-2", "gpt-y"}, + {"responses buffered", FormatOpenAI, "openai", "gpt-5.6-luna", false, + `{"id":"resp_1","model":"gpt-5.6-luna-2026","status":"completed","output":[{"type":"message","content":[{"type":"output_text","text":"hi"}]}]}`, + "resp_1", "gpt-5.6-luna-2026"}, + {"responses stream", FormatOpenAI, "openai", "gpt-5.6-luna", true, + metaSSE(`{"type":"response.created","response":{"id":"resp_2","model":"gpt-5.6-luna-2026"}}`, + `{"type":"response.output_text.delta","delta":"hi"}`, + `{"type":"response.completed","response":{"id":"resp_2","model":"gpt-5.6-luna-2026","status":"completed"}}`), + "resp_2", "gpt-5.6-luna-2026"}, + {"anthropic buffered", FormatAnthropic, "anthropic", "", false, + `{"id":"msg_1","model":"claude-x","content":[{"type":"text","text":"hi"}],"stop_reason":"end_turn","usage":{"input_tokens":1,"output_tokens":1}}`, + "msg_1", "claude-x"}, + {"anthropic stream", FormatAnthropic, "anthropic", "", true, anthStream, "msg_1", "claude-x"}, + {"gemini buffered", FormatGemini, "gemini", "", false, + `{"responseId":"gr-1","modelVersion":"gemini-2.5","candidates":[{"content":{"parts":[{"text":"hi"}]},"finishReason":"STOP"}]}`, + "gr-1", "gemini-2.5"}, + {"gemini stream", FormatGemini, "gemini", "", true, + metaSSE(`{"responseId":"gr-2","modelVersion":"gemini-2.5","candidates":[{"content":{"parts":[{"text":"hi"}]},"finishReason":"STOP"}]}`), + "gr-2", "gemini-2.5"}, + } + for _, c := range cases { + for _, failFirst := range []bool{false, true} { + t.Run(c.name+map[bool]string{false: "", true: " after failed attempt"}[failFirst], func(t *testing.T) { + ctype := "application/json" + if c.stream { + ctype = "text/event-stream" + } + srv := metaSrv(t, failFirst, ctype, metaFullHdr, c.body) + cc := newTestClient(t, ProviderConfig{ID: c.id, Format: c.format, BaseURL: srv.URL, APIKey: "k"}, srv) + if c.model != "" { + cc.model = c.model + } + res, _ := m2Run(t, cc, m2Req(), c.stream) + if res.Meta.ID != c.wantID || res.Meta.Model != c.wantModel { + t.Errorf("ID/Model = %q/%q, want %q/%q", res.Meta.ID, res.Meta.Model, c.wantID, c.wantModel) + } + if res.Meta.RequestID != "req-123" { + t.Errorf("RequestID = %q, want req-123", res.Meta.RequestID) + } + if res.Meta.RateLimit != metaFullRL { + t.Errorf("RateLimit = %+v, want %+v", res.Meta.RateLimit, metaFullRL) + } + }) + } + } +} + +// A failed attempt's headers never leak into a later success lacking them. +func TestM6_MetaFailedAttemptHeadersDoNotLeak(t *testing.T) { + for _, stream := range []bool{false, true} { + body := `{"id":"c1","choices":[{"message":{"role":"assistant","content":"hi"},"finish_reason":"stop"}]}` + ctype := "application/json" + if stream { + body = metaSSE(`{"id":"c1","choices":[{"delta":{"content":"hi"},"finish_reason":"stop"}]}`, "[DONE]") + ctype = "text/event-stream" + } + srv := metaSrv(t, true, ctype, nil, body) + cc := newTestClient(t, ProviderConfig{ID: "openai", Format: FormatOpenAI, BaseURL: srv.URL, APIKey: "k"}, srv) + res, _ := m2Run(t, cc, m2Req(), stream) + if res.Meta.ID != "c1" { + t.Errorf("stream=%v ID = %q, want c1", stream, res.Meta.ID) + } + if res.Meta.RequestID != "" || res.Meta.RateLimit != (RateLimit{}) { + t.Errorf("stream=%v leaked failed-attempt headers: %+v", stream, res.Meta) + } + } +} + +// Absent data stays zero on every format. +func TestM6_MetaAbsentIsZero(t *testing.T) { + for _, c := range []struct { + format Format + id string + body string + }{ + {FormatOpenAI, "openai", `{"choices":[{"message":{"role":"assistant","content":"hi"},"finish_reason":"stop"}]}`}, + {FormatAnthropic, "anthropic", `{"content":[{"type":"text","text":"hi"}],"stop_reason":"end_turn","usage":{"input_tokens":1,"output_tokens":1}}`}, + {FormatGemini, "gemini", `{"candidates":[{"content":{"parts":[{"text":"hi"}]},"finishReason":"STOP"}]}`}, + } { + srv := metaSrv(t, false, "application/json", nil, c.body) + cc := newTestClient(t, ProviderConfig{ID: c.id, Format: c.format, BaseURL: srv.URL, APIKey: "k"}, srv) + res, _ := m2Run(t, cc, m2Req(), false) + if res.Meta != (ResponseMeta{}) { + t.Errorf("%s: Meta = %+v, want zero", c.format, res.Meta) + } + } +} + +// A successful JSON body to a streaming request (non-SSE fallback) fills Meta. +func TestM6_MetaNonSSEStreamFallback(t *testing.T) { + srv := metaSrv(t, false, "application/json", metaFullHdr, + `{"id":"chatcmpl-9","model":"gpt-z","choices":[{"message":{"role":"assistant","content":"hi"},"finish_reason":"stop"}]}`) + cc := newTestClient(t, ProviderConfig{ID: "openai", Format: FormatOpenAI, BaseURL: srv.URL, APIKey: "k"}, srv) + res, _ := m2Run(t, cc, m2Req(), true) + want := ResponseMeta{ID: "chatcmpl-9", Model: "gpt-z", RequestID: "req-123", RateLimit: metaFullRL} + if res.Meta != want { + t.Errorf("Meta = %+v, want %+v", res.Meta, want) + } +} + +func TestM6_ParseResponseMeta(t *testing.T) { + future := time.Now().Add(90 * time.Second).UTC().Format(time.RFC3339) + past := time.Now().Add(-time.Hour).UTC().Format(time.RFC3339) + cases := []struct { + name string + h map[string]string + check func(t *testing.T, m ResponseMeta) + }{ + {"empty", nil, func(t *testing.T, m ResponseMeta) { + if m != (ResponseMeta{}) { + t.Errorf("got %+v", m) + } + }}, + {"openai form", metaFullHdr, func(t *testing.T, m ResponseMeta) { + if m.RequestID != "req-123" || m.RateLimit != metaFullRL { + t.Errorf("got %+v", m) + } + }}, + {"reset forms: ms and plain seconds", map[string]string{ + "x-ratelimit-reset-requests": "20ms", "x-ratelimit-reset-tokens": "30"}, func(t *testing.T, m ResponseMeta) { + rl := m.RateLimit + if !rl.Reported || rl.ResetRequests != 20*time.Millisecond || rl.ResetTokens != 30*time.Second { + t.Errorf("got %+v", rl) + } + }}, + {"anthropic form", map[string]string{ + "anthropic-ratelimit-requests-remaining": "7", "anthropic-ratelimit-tokens-remaining": "800", + "anthropic-ratelimit-requests-reset": future, "anthropic-ratelimit-tokens-reset": future, + "request-id": "req-anth"}, func(t *testing.T, m ResponseMeta) { + rl := m.RateLimit + if m.RequestID != "req-anth" || !rl.Reported || rl.RemainingRequests != 7 || rl.RemainingTokens != 800 { + t.Errorf("got %+v", m) + } + for _, d := range []time.Duration{rl.ResetRequests, rl.ResetTokens} { + if d <= 60*time.Second || d > 91*time.Second { + t.Errorf("reset = %v, want ~90s", d) + } + } + }}, + {"anthropic reset in the past floors at zero but is reported", map[string]string{ + "anthropic-ratelimit-requests-reset": past}, func(t *testing.T, m ResponseMeta) { + if !m.RateLimit.Reported || m.RateLimit.ResetRequests != 0 { + t.Errorf("got %+v", m.RateLimit) + } + }}, + {"openai wins per field, anthropic fills the rest", map[string]string{ + "x-ratelimit-remaining-requests": "5", + "anthropic-ratelimit-requests-remaining": "50", + "anthropic-ratelimit-tokens-remaining": "60"}, func(t *testing.T, m ResponseMeta) { + if m.RateLimit.RemainingRequests != 5 || m.RateLimit.RemainingTokens != 60 { + t.Errorf("got %+v", m.RateLimit) + } + }}, + {"malformed field stays zero, others parse", map[string]string{ + "x-ratelimit-remaining-requests": "abc", "x-ratelimit-remaining-tokens": "12", + "x-ratelimit-reset-requests": "soon"}, func(t *testing.T, m ResponseMeta) { + rl := m.RateLimit + if !rl.Reported || rl.RemainingRequests != 0 || rl.RemainingTokens != 12 || rl.ResetRequests != 0 { + t.Errorf("got %+v", rl) + } + }}, + {"all malformed is not reported", map[string]string{ + "x-ratelimit-remaining-requests": "abc", "x-ratelimit-reset-tokens": "??"}, func(t *testing.T, m ResponseMeta) { + if m.RateLimit != (RateLimit{}) { + t.Errorf("got %+v", m.RateLimit) + } + }}, + {"request id precedence", map[string]string{ + "x-request-id": "a", "request-id": "b", "x-goog-request-id": "c", "x-amzn-requestid": "d"}, func(t *testing.T, m ResponseMeta) { + if m.RequestID != "a" { + t.Errorf("got %q", m.RequestID) + } + }}, + {"request-id second", map[string]string{"request-id": "b", "x-goog-request-id": "c"}, func(t *testing.T, m ResponseMeta) { + if m.RequestID != "b" { + t.Errorf("got %q", m.RequestID) + } + }}, + {"goog third", map[string]string{"x-goog-request-id": "c", "x-amzn-requestid": "d"}, func(t *testing.T, m ResponseMeta) { + if m.RequestID != "c" { + t.Errorf("got %q", m.RequestID) + } + }}, + {"amzn fourth, cf-ray ignored", map[string]string{"x-amzn-requestid": "d", "cf-ray": "ray"}, func(t *testing.T, m ResponseMeta) { + if m.RequestID != "d" { + t.Errorf("got %q", m.RequestID) + } + }}, + {"cf-ray alone ignored", map[string]string{"cf-ray": "ray"}, func(t *testing.T, m ResponseMeta) { + if m.RequestID != "" { + t.Errorf("got %q", m.RequestID) + } + }}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + h := http.Header{} + for k, v := range c.h { + h.Set(k, v) + } + c.check(t, parseResponseMeta(h)) + }) + } +} + +// Embed / Speak / Transcribe carry RequestID + RateLimit; a failed first +// attempt's headers never appear. +func TestM6_MetaNonChatEntryPoints(t *testing.T) { + embedOpenAI := `{"data":[{"index":0,"embedding":[0.5,1]}],"usage":{"prompt_tokens":2}}` + embedGemini := `{"embeddings":[{"values":[0.5,1]}]}` + geminiAudio := `{"candidates":[{"content":{"parts":[{"inlineData":{"mimeType":"audio/L16;rate=24000","data":"AQID"}}]}}]}` + for _, failFirst := range []bool{false, true} { + suffix := map[bool]string{false: "", true: " after failed attempt"}[failFirst] + runs := []struct { + name string + format Format + id string + ctype string + body string + call func(t *testing.T, s *SDK, id string) ResponseMeta + }{ + {"embed openai", FormatOpenAI, "openai", "application/json", embedOpenAI, + func(t *testing.T, s *SDK, id string) ResponseMeta { + r, err := s.Embed(t.Context(), id, "m", EmbedRequest{Inputs: []string{"a"}}) + if err != nil { + t.Fatal(err) + } + return r.Meta + }}, + {"embed gemini", FormatGemini, "gemini", "application/json", embedGemini, + func(t *testing.T, s *SDK, id string) ResponseMeta { + r, err := s.Embed(t.Context(), id, "m", EmbedRequest{Inputs: []string{"a"}}) + if err != nil { + t.Fatal(err) + } + return r.Meta + }}, + {"speak openai", FormatOpenAI, "openai", "audio/mpeg", "\x01\x02", + func(t *testing.T, s *SDK, id string) ResponseMeta { + r, err := s.Speak(t.Context(), id, "tts-1", SpeakRequest{Text: "hi", Voice: "alloy"}) + if err != nil { + t.Fatal(err) + } + return r.Meta + }}, + {"speak gemini", FormatGemini, "gemini", "application/json", geminiAudio, + func(t *testing.T, s *SDK, id string) ResponseMeta { + r, err := s.Speak(t.Context(), id, "gemini-tts", SpeakRequest{Text: "hi", Voice: "Kore"}) + if err != nil { + t.Fatal(err) + } + return r.Meta + }}, + {"transcribe openai", FormatOpenAI, "openai", "application/json", `{"text":"hi"}`, + func(t *testing.T, s *SDK, id string) ResponseMeta { + r, err := s.Transcribe(t.Context(), id, "whisper-1", TranscribeRequest{Audio: []byte{1, 2}}) + if err != nil { + t.Fatal(err) + } + return r.Meta + }}, + } + for _, r := range runs { + t.Run(r.name+suffix, func(t *testing.T) { + srv := metaSrv(t, failFirst, r.ctype, metaFullHdr, r.body) + s := newTestSDK(t, ProviderConfig{ID: r.id, Format: r.format, APIKey: "k"}, srv) + m := r.call(t, s, r.id) + if m.RequestID != "req-123" { + t.Errorf("RequestID = %q, want req-123", m.RequestID) + } + if m.RateLimit != metaFullRL { + t.Errorf("RateLimit = %+v, want %+v", m.RateLimit, metaFullRL) + } + }) + } + } +} + +func TestM6_MetaNonChatAbsentIsZero(t *testing.T) { + srv := metaSrv(t, false, "application/json", nil, `{"text":"hi"}`) + s := newTestSDK(t, ProviderConfig{ID: "openai", Format: FormatOpenAI, APIKey: "k"}, srv) + r, err := s.Transcribe(t.Context(), "openai", "whisper-1", TranscribeRequest{Audio: []byte{1}}) + if err != nil { + t.Fatal(err) + } + if r.Meta != (ResponseMeta{}) { + t.Errorf("Meta = %+v, want zero", r.Meta) + } +} diff --git a/m6_observer_test.go b/m6_observer_test.go new file mode 100644 index 0000000..1911d3b --- /dev/null +++ b/m6_observer_test.go @@ -0,0 +1,358 @@ +package llm + +import ( + "context" + "errors" + "net/http" + "net/http/httptest" + "strings" + "sync" + "sync/atomic" + "testing" +) + +const obsKey = "sk-observer-secret-key" + +type obsCollector struct { + mu sync.Mutex + ev []AttemptEvent +} + +func (c *obsCollector) fn(e AttemptEvent) { + c.mu.Lock() + c.ev = append(c.ev, e) + c.mu.Unlock() +} + +func (c *obsCollector) all() []AttemptEvent { + c.mu.Lock() + defer c.mu.Unlock() + return append([]AttemptEvent(nil), c.ev...) +} + +// obsStep is one scripted server response. +type obsStep struct { + status int + header map[string]string + ctype string + body string +} + +func obsServer(t *testing.T, steps ...obsStep) (*httptest.Server, *atomic.Int32) { + t.Helper() + var n atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + i := int(n.Add(1)) - 1 + if i >= len(steps) { + i = len(steps) - 1 + } + s := steps[i] + for k, v := range s.header { + w.Header().Set(k, v) + } + if s.ctype != "" { + w.Header().Set("Content-Type", s.ctype) + } + w.WriteHeader(s.status) + _, _ = w.Write([]byte(s.body)) + })) + t.Cleanup(srv.Close) + return srv, &n +} + +type obsEntry struct { + name string + op string + model string + ctype string + ok string + call func(s *SDK) error +} + +func obsEntries() []obsEntry { + ctx := context.Background() + chatOK := `{"id":"c1","choices":[{"message":{"role":"assistant","content":"hi"},"finish_reason":"stop"}]}` + return []obsEntry{ + {"chat", "chat", "m", "application/json", chatOK, func(s *SDK) error { + c, err := s.Chat("openai", "m") + if err != nil { + return err + } + _, err = c.Call(ctx, userMsg()) + return err + }}, + {"stream", "stream", "m", "text/event-stream", metaSSE(`{"id":"c1","choices":[{"delta":{"content":"hi"},"finish_reason":"stop"}]}`, "[DONE]"), func(s *SDK) error { + c, err := s.Chat("openai", "m") + if err != nil { + return err + } + _, err = c.CallStream(ctx, userMsg(), func(Delta) error { return nil }) + return err + }}, + {"embed", "embed", "m", "application/json", `{"data":[{"index":0,"embedding":[0.5,1]}],"usage":{"prompt_tokens":2}}`, func(s *SDK) error { + _, err := s.Embed(ctx, "openai", "m", EmbedRequest{Inputs: []string{"a"}}) + return err + }}, + {"speak", "speak", "tts-1", "audio/mpeg", "\x01\x02", func(s *SDK) error { + _, err := s.Speak(ctx, "openai", "tts-1", SpeakRequest{Text: "hi", Voice: "alloy"}) + return err + }}, + {"transcribe", "transcribe", "whisper-1", "application/json", `{"text":"hi"}`, func(s *SDK) error { + _, err := s.Transcribe(ctx, "openai", "whisper-1", TranscribeRequest{Audio: []byte{1, 2}}) + return err + }}, + {"models", "models", "", "application/json", `{"data":[{"id":"m1"}]}`, func(s *SDK) error { + p, err := s.Provider("openai") + if err != nil { + return err + } + _, err = p.ListModels(ctx, ForceRefresh()) + return err + }}, + } +} + +func obsSDK(t *testing.T, srv *httptest.Server, col *obsCollector, extra ...Option) *SDK { + t.Helper() + fastBackoff(t) + opts := append([]Option{ + WithAttemptObserver(col.fn), + WithProvider("openai", WithAPIKey(obsKey), WithBaseURL(srv.URL)), + }, extra...) + return New(opts...) +} + +func obsNoKey(t *testing.T, ev AttemptEvent) { + t.Helper() + if ev.Err != nil && strings.Contains(ev.Err.Error(), obsKey) { + t.Errorf("event Err leaks API key: %v", ev.Err) + } +} + +// A first-try success yields exactly one fully populated event. +func TestM6_ObserverSuccessFirstTry(t *testing.T) { + for _, e := range obsEntries() { + t.Run(e.name, func(t *testing.T) { + srv, _ := obsServer(t, obsStep{200, map[string]string{"X-Request-Id": "req-1"}, e.ctype, e.ok}) + col := &obsCollector{} + if err := e.call(obsSDK(t, srv, col)); err != nil { + t.Fatal(err) + } + evs := col.all() + if len(evs) != 1 { + t.Fatalf("events = %d, want 1: %+v", len(evs), evs) + } + ev := evs[0] + if ev.Provider != "openai" || ev.Operation != e.op || ev.Model != e.model { + t.Errorf("identity = %q/%q/%q, want openai/%s/%q", ev.Provider, ev.Operation, ev.Model, e.op, e.model) + } + if ev.Attempt != 1 || ev.Status != 200 || ev.Err != nil { + t.Errorf("attempt/status/err = %d/%d/%v", ev.Attempt, ev.Status, ev.Err) + } + if ev.Latency <= 0 { + t.Errorf("Latency = %v, want > 0", ev.Latency) + } + if ev.RetryDelay != 0 { + t.Errorf("RetryDelay = %v, want 0", ev.RetryDelay) + } + if ev.RequestID != "req-1" { + t.Errorf("RequestID = %q, want req-1", ev.RequestID) + } + }) + } +} + +// A retried call produces one event per HTTP attempt; only non-final +// retryable attempts carry a RetryDelay. +func TestM6_ObserverRetryProducesNEvents(t *testing.T) { + for _, e := range obsEntries() { + t.Run(e.name, func(t *testing.T) { + srv, n := obsServer(t, + obsStep{500, map[string]string{"X-Request-Id": "bad-1"}, "application/json", `{"error":{"message":"boom"}}`}, + obsStep{429, map[string]string{"X-Request-Id": "bad-2"}, "application/json", `{"error":{"message":"slow"}}`}, + obsStep{200, map[string]string{"X-Request-Id": "good"}, e.ctype, e.ok}) + col := &obsCollector{} + if err := e.call(obsSDK(t, srv, col)); err != nil { + t.Fatal(err) + } + evs := col.all() + if int(n.Load()) != 3 || len(evs) != 3 { + t.Fatalf("requests = %d, events = %d, want 3/3", n.Load(), len(evs)) + } + wantStatus := []int{500, 429, 200} + wantID := []string{"bad-1", "bad-2", "good"} + for i, ev := range evs { + if ev.Attempt != i+1 || ev.Status != wantStatus[i] || ev.RequestID != wantID[i] { + t.Errorf("event %d = attempt %d status %d id %q", i, ev.Attempt, ev.Status, ev.RequestID) + } + if ev.Latency <= 0 { + t.Errorf("event %d Latency = %v", i, ev.Latency) + } + if i < 2 { + var ae *APIError + if !errors.As(ev.Err, &ae) || ae.Status != wantStatus[i] { + t.Errorf("event %d Err = %v, want *APIError %d", i, ev.Err, wantStatus[i]) + } + if ev.RetryDelay <= 0 { + t.Errorf("event %d RetryDelay = %v, want > 0", i, ev.RetryDelay) + } + } else if ev.Err != nil || ev.RetryDelay != 0 { + t.Errorf("final event err=%v delay=%v, want nil/0", ev.Err, ev.RetryDelay) + } + obsNoKey(t, ev) + } + }) + } +} + +// A terminal (non-retryable) failure fires one event with the error and no +// RetryDelay. +func TestM6_ObserverTerminalFailure(t *testing.T) { + for _, e := range obsEntries() { + t.Run(e.name, func(t *testing.T) { + srv, n := obsServer(t, obsStep{401, nil, "application/json", `{"error":{"message":"nope"}}`}) + col := &obsCollector{} + if err := e.call(obsSDK(t, srv, col)); err == nil { + t.Fatal("want error") + } + evs := col.all() + if len(evs) != int(n.Load()) || len(evs) != 1 { + t.Fatalf("events = %d, requests = %d, want 1/1", len(evs), n.Load()) + } + var ae *APIError + if evs[0].Status != 401 || !errors.As(evs[0].Err, &ae) || evs[0].RetryDelay != 0 { + t.Errorf("event = %+v", evs[0]) + } + obsNoKey(t, evs[0]) + }) + } +} + +// A Retry-After hint is reflected in RetryDelay. +func TestM6_ObserverRetryAfterHonoured(t *testing.T) { + e := obsEntries()[0] + srv, _ := obsServer(t, + obsStep{429, map[string]string{"Retry-After": "1"}, "application/json", `{"error":{"message":"slow"}}`}, + obsStep{200, nil, e.ctype, e.ok}) + col := &obsCollector{} + if err := e.call(obsSDK(t, srv, col)); err != nil { + t.Fatal(err) + } + evs := col.all() + if len(evs) != 2 || evs[0].RetryDelay < 900*1e6 { + t.Fatalf("events = %+v, want 2 with first RetryDelay >= ~1s", evs) + } +} + +type obsFailRT struct{ n atomic.Int32 } + +func (f *obsFailRT) RoundTrip(*http.Request) (*http.Response, error) { + f.n.Add(1) + return nil, errors.New("dial failed using " + obsKey) +} + +// A transport error reports Status 0, retries, and never leaks the key. +func TestM6_ObserverTransportError(t *testing.T) { + for _, e := range obsEntries() { + t.Run(e.name, func(t *testing.T) { + fastBackoff(t) + rt := &obsFailRT{} + col := &obsCollector{} + s := New(WithAttemptObserver(col.fn), WithTransport(rt), WithRetryPolicy(RetryPolicy{MaxAttempts: 2}), + WithProvider("openai", WithAPIKey(obsKey), WithBaseURL("http://127.0.0.1:1"))) + err := e.call(s) + if err == nil { + t.Fatal("want error") + } + evs := col.all() + if len(evs) != 2 || int(rt.n.Load()) != 2 { + t.Fatalf("events = %d, round trips = %d, want 2/2", len(evs), rt.n.Load()) + } + for i, ev := range evs { + if ev.Status != 0 || ev.Err == nil || ev.Attempt != i+1 { + t.Errorf("event %d = %+v", i, ev) + } + obsNoKey(t, ev) + } + if evs[0].RetryDelay <= 0 || evs[1].RetryDelay != 0 { + t.Errorf("RetryDelay = %v then %v, want >0 then 0", evs[0].RetryDelay, evs[1].RetryDelay) + } + }) + } +} + +// The learn-once immediate retry counts as an attempt (RetryDelay 0 though +// not final). +func TestM6_ObserverLearnOnceRetryCounts(t *testing.T) { + e := obsEntries()[1] + srv, _ := obsServer(t, + obsStep{400, nil, "application/json", `{"error":{"message":"Unknown parameter: stream_options"}}`}, + obsStep{200, nil, e.ctype, e.ok}) + col := &obsCollector{} + if err := e.call(obsSDK(t, srv, col)); err != nil { + t.Fatal(err) + } + evs := col.all() + if len(evs) != 2 { + t.Fatalf("events = %d, want 2: %+v", len(evs), evs) + } + if evs[0].Status != 400 || evs[0].Attempt != 1 || evs[0].Err == nil || evs[0].RetryDelay != 0 { + t.Errorf("learn attempt = %+v", evs[0]) + } + if evs[1].Status != 200 || evs[1].Attempt != 2 || evs[1].Err != nil { + t.Errorf("retry attempt = %+v", evs[1]) + } +} + +// A stream that fails after headers reports Status 200 with the error on +// that attempt's single event. +func TestM6_ObserverStreamMidFailure(t *testing.T) { + srv, _ := obsServer(t, obsStep{200, nil, "text/event-stream", + metaSSE(`{"id":"c1","choices":[{"delta":{"content":"par"}}]}`)}) + col := &obsCollector{} + s := obsSDK(t, srv, col) + c, _ := s.Chat("openai", "m") + if _, err := c.CallStream(context.Background(), userMsg(), func(Delta) error { return nil }); err == nil { + t.Fatal("want premature-close error") + } + evs := col.all() + if len(evs) != 1 || evs[0].Status != 200 || evs[0].Err == nil || evs[0].Operation != "stream" { + t.Fatalf("events = %+v", evs) + } +} + +// A panicking observer is recovered: the call still succeeds on every entry +// point, streams complete, and the observer is still invoked per attempt. +func TestM6_ObserverPanicRecovered(t *testing.T) { + for _, e := range obsEntries() { + t.Run(e.name, func(t *testing.T) { + fastBackoff(t) + srv, _ := obsServer(t, + obsStep{500, nil, "application/json", `{"error":{"message":"boom"}}`}, + obsStep{200, nil, e.ctype, e.ok}) + var calls atomic.Int32 + s := New(WithAttemptObserver(func(AttemptEvent) { calls.Add(1); panic("observer boom") }), + WithProvider("openai", WithAPIKey(obsKey), WithBaseURL(srv.URL))) + if err := e.call(s); err != nil { + t.Fatalf("call failed despite observer panic: %v", err) + } + if calls.Load() != 2 { + t.Errorf("observer calls = %d, want 2", calls.Load()) + } + }) + } +} + +// Without an observer nothing changes (AC-5). +func TestM6_ObserverNilIsNoop(t *testing.T) { + for _, e := range obsEntries() { + t.Run(e.name, func(t *testing.T) { + fastBackoff(t) + srv, _ := obsServer(t, obsStep{200, nil, e.ctype, e.ok}) + s := New(WithAttemptObserver(nil), WithProvider("openai", WithAPIKey(obsKey), WithBaseURL(srv.URL))) + if err := e.call(s); err != nil { + t.Fatal(err) + } + }) + } +} diff --git a/m6_refusal_test.go b/m6_refusal_test.go new file mode 100644 index 0000000..382b6b7 --- /dev/null +++ b/m6_refusal_test.go @@ -0,0 +1,135 @@ +package llm + +import ( + "strings" + "testing" +) + +func m6Sse(events ...string) string { + var b strings.Builder + for _, e := range events { + b.WriteString("data: " + e + "\n\n") + } + return b.String() +} + +func m6AnthropicSSE(stopDetails string) string { + return "event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"id\":\"m\",\"model\":\"c\",\"usage\":{\"input_tokens\":1}}}\n\n" + + "event: message_delta\ndata: {\"type\":\"message_delta\",\"delta\":{\"stop_reason\":\"refusal\"" + stopDetails + "},\"usage\":{\"output_tokens\":1}}\n\n" + + "event: message_stop\ndata: {\"type\":\"message_stop\"}\n\n" +} + +func TestM6_RefusalSurfacedOnEveryFormat(t *testing.T) { + cases := []struct { + name string + format Format + id string + model string + stream bool + body string + wantRef string + wantFinish string + wantContnt string + wantDeltas int // number of DeltaRefusal deltas, -1 = at least one + }{ + {"openai buffered", FormatOpenAI, "openai", "", false, + `{"choices":[{"message":{"role":"assistant","content":null,"refusal":"no way"},"finish_reason":"stop"}]}`, + "no way", FinishContentFilter, "", 0}, + {"openai buffered empty finish", FormatOpenAI, "openai", "", false, + `{"choices":[{"message":{"role":"assistant","refusal":"nope"},"finish_reason":""}]}`, + "nope", FinishContentFilter, "", 0}, + {"openai buffered length kept", FormatOpenAI, "openai", "", false, + `{"choices":[{"message":{"role":"assistant","content":"par","refusal":"nope"},"finish_reason":"length"}]}`, + "nope", FinishLength, "par", 0}, + {"openai buffered normal", FormatOpenAI, "openai", "", false, + `{"choices":[{"message":{"role":"assistant","content":"hi"},"finish_reason":"stop"}]}`, + "", FinishStop, "hi", 0}, + {"openai stream", FormatOpenAI, "openai", "", true, + m6Sse(`{"choices":[{"delta":{"refusal":"no "}}]}`, `{"choices":[{"delta":{"refusal":"way"}}]}`, + `{"choices":[{"delta":{},"finish_reason":"stop"}]}`, "[DONE]"), + "no way", FinishContentFilter, "", -1}, + {"openai stream normal", FormatOpenAI, "openai", "", true, + m6Sse(`{"choices":[{"delta":{"content":"hi"}}]}`, `{"choices":[{"delta":{},"finish_reason":"stop"}]}`, "[DONE]"), + "", FinishStop, "hi", 0}, + {"responses buffered", FormatOpenAI, "openai", "gpt-5.6-luna", false, + `{"status":"completed","output":[{"type":"message","content":[{"type":"refusal","refusal":"no "},{"type":"refusal","refusal":"way"}]}]}`, + "no way", FinishContentFilter, "", 0}, + {"responses buffered normal", FormatOpenAI, "openai", "gpt-5.6-luna", false, + `{"status":"completed","output":[{"type":"message","content":[{"type":"output_text","text":"hi"}]}]}`, + "", FinishStop, "hi", 0}, + {"responses stream", FormatOpenAI, "openai", "gpt-5.6-luna", true, + m6Sse(`{"type":"response.refusal.delta","delta":"no "}`, `{"type":"response.refusal.delta","delta":"way"}`, + `{"type":"response.refusal.done","refusal":"no way"}`, + `{"type":"response.completed","response":{"status":"completed"}}`), + "no way", FinishContentFilter, "", -1}, + {"anthropic buffered explanation", FormatAnthropic, "anthropic", "", false, + `{"content":[],"stop_reason":"refusal","stop_details":{"explanation":"policy"},"usage":{"input_tokens":1,"output_tokens":1}}`, + "policy", FinishContentFilter, "", 0}, + {"anthropic buffered no explanation", FormatAnthropic, "anthropic", "", false, + `{"content":[],"stop_reason":"refusal","usage":{"input_tokens":1,"output_tokens":1}}`, + "", FinishContentFilter, "", 0}, + {"anthropic buffered normal", FormatAnthropic, "anthropic", "", false, + `{"content":[{"type":"text","text":"hi"}],"stop_reason":"end_turn","usage":{"input_tokens":1,"output_tokens":1}}`, + "", FinishStop, "hi", 0}, + {"anthropic stream explanation", FormatAnthropic, "anthropic", "", true, + m6AnthropicSSE(`,"stop_details":{"explanation":"policy"}`), + "policy", FinishContentFilter, "", 1}, + {"anthropic stream no explanation", FormatAnthropic, "anthropic", "", true, + m6AnthropicSSE(``), + "", FinishContentFilter, "", 0}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + cc := m2Client(t, c.format, c.id, c.body, c.stream) + if c.model != "" { + cc.model = c.model + } + req := m2Req() + res, ds := m2Run(t, cc, req, c.stream) + if res.Refusal != c.wantRef { + t.Errorf("Refusal = %q, want %q", res.Refusal, c.wantRef) + } + if res.FinishReason != c.wantFinish { + t.Errorf("FinishReason = %q, want %q", res.FinishReason, c.wantFinish) + } + if res.Content != c.wantContnt { + t.Errorf("Content = %q, want %q", res.Content, c.wantContnt) + } + if !c.stream { + return + } + var cat strings.Builder + n := 0 + for _, d := range ds { + if d.Kind == DeltaRefusal { + n++ + cat.WriteString(d.Text) + } + } + if cat.String() != c.wantRef { + t.Errorf("concatenated DeltaRefusal = %q, want %q", cat.String(), c.wantRef) + } + if c.wantDeltas >= 0 && n != c.wantDeltas { + t.Errorf("DeltaRefusal count = %d, want %d", n, c.wantDeltas) + } + if c.wantDeltas < 0 && n < 1 { + t.Errorf("want at least one DeltaRefusal, got %d", n) + } + }) + } +} + +// DeltaRefusal is appended after DeltaToolArgs so existing numbers hold. +func TestM6_DeltaRefusalAppended(t *testing.T) { + if DeltaReasoning != 0 || DeltaContent != 1 || DeltaToolArgs != 2 || DeltaRefusal != 3 { + t.Errorf("DeltaKind numbering: %d %d %d %d", DeltaReasoning, DeltaContent, DeltaToolArgs, DeltaRefusal) + } +} + +// The refusal never replays into the next turn. +func TestM6_AssistantMessageDoesNotReplayRefusal(t *testing.T) { + m := (&ChatResult{Content: "c", Refusal: "no"}).AssistantMessage() + if strings.Contains(m.Content, "no") || m.Content != "c" { + t.Errorf("content = %q", m.Content) + } +} diff --git a/m6b_closeout_test.go b/m6b_closeout_test.go new file mode 100644 index 0000000..57f9b8c --- /dev/null +++ b/m6b_closeout_test.go @@ -0,0 +1,119 @@ +package llm + +import ( + "context" + "errors" + "net/http" + "sync/atomic" + "testing" +) + +// m6b SL05: close-out tests (RunTools per-handler ctx check, coverage gaps). + +// The context is checked before EACH handler: once a handler cancels it, the +// remaining handlers of the turn do not run and RunTools returns the ctx error. +func TestRunTools_CtxCheckedBeforeEachHandler(t *testing.T) { + f := toolFormats[0] // openai + srv, _ := newToolServer(t, f.call([2]string{"c1", "a"}, [2]string{"c2", "b"}), f.stop) + cc := f.client(t, srv) + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + var ranA, ranB atomic.Int32 + _, _, err := RunTools(ctx, cc, f.req(), map[string]ToolHandler{ + "a": func(context.Context, ToolCall) (string, error) { ranA.Add(1); cancel(); return "ok", nil }, + "b": func(context.Context, ToolCall) (string, error) { ranB.Add(1); return "ok", nil }, + }, ToolLoopOptions{}) + if !errors.Is(err, context.Canceled) { + t.Fatalf("err = %v, want context.Canceled", err) + } + if ranA.Load() != 1 { + t.Errorf("handler a ran %d times, want 1", ranA.Load()) + } + if ranB.Load() != 0 { + t.Errorf("handler b ran %d times after cancellation, want 0", ranB.Load()) + } +} + +// A Gemini tool result carrying an image URL without a MIME type is rejected. +func TestGemini_ToolResultImageURLWithoutMIME(t *testing.T) { + req := &ChatRequest{Messages: []Message{ + {Role: RoleUser, Content: "go"}, + {Role: RoleAssistant, ToolCalls: []ToolCall{{ID: "c1", Name: "wx", Arguments: "{}"}}}, + {Role: RoleTool, ToolCallID: "c1", ToolName: "wx", Parts: []ContentPart{ImageURLPart("https://x/y.png")}}, + }} + _, err := buildGeminiRequest(req, "gemini-x", false) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Fatalf("err = %v, want *ConfigError", err) + } +} + +// CountTokens with a reserved header fails before any I/O. +func TestCountTokens_ReservedHeaderBeforeIO(t *testing.T) { + srv, rec := countServer(t, obsStep{status: 200, body: anCountOK}) + cc := countClient(t, "anthropic", FormatAnthropic, "claude-x", srv) + req := countReq() + req.Headers = map[string]string{"x-api-key": "evil"} + _, err := cc.CountTokens(context.Background(), req) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Fatalf("err = %v, want *ConfigError", err) + } + if rec.n() != 0 { + t.Errorf("requests = %d, want 0", rec.n()) + } +} + +// A request only the Anthropic builder rejects errors before any I/O. +func TestCountTokens_AnthropicBuilderRejection(t *testing.T) { + srv, rec := countServer(t, obsStep{status: 200, body: anCountOK}) + cc := countClient(t, "anthropic", FormatAnthropic, "claude-x", srv) + req := countReq() + req.Thinking = "enabled" + req.ThinkingBudget = 8000 + req.MaxTokens = 2000 + _, err := cc.CountTokens(context.Background(), req) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Fatalf("err = %v, want *ConfigError", err) + } + if rec.n() != 0 { + t.Errorf("requests = %d, want 0", rec.n()) + } +} + +type closeoutSentinel struct{} + +func (closeoutSentinel) Error() string { return "dial failed using " + obsKey } + +type closeoutRT struct{ err error } + +func (r closeoutRT) RoundTrip(*http.Request) (*http.Response, error) { return nil, r.err } + +// A masked AttemptEvent.Err still unwraps to the original error. +func TestAttemptEvent_MaskedErrUnwraps(t *testing.T) { + fastBackoff(t) + orig := closeoutSentinel{} + col := &obsCollector{} + s := New(WithAttemptObserver(col.fn), WithTransport(closeoutRT{err: orig}), + WithRetryPolicy(RetryPolicy{MaxAttempts: 1}), + WithProvider("openai", WithAPIKey(obsKey), WithBaseURL("http://127.0.0.1:1"))) + cc, err := s.Chat("openai", "gpt-4o") + if err != nil { + t.Fatal(err) + } + _, _ = cc.Call(context.Background(), &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "hi"}}}) + evs := col.all() + if len(evs) == 0 || evs[0].Err == nil { + t.Fatalf("events = %+v", evs) + } + e := evs[0].Err + obsNoKey(t, evs[0]) + if !errors.Is(e, orig) { + t.Errorf("errors.Is(event err, original) = false: %v", e) + } + var cs closeoutSentinel + if !errors.As(e, &cs) { + t.Errorf("errors.As to original type failed: %v", e) + } +} diff --git a/m6b_count_test.go b/m6b_count_test.go new file mode 100644 index 0000000..3a8d378 --- /dev/null +++ b/m6b_count_test.go @@ -0,0 +1,485 @@ +package llm + +import ( + "context" + "encoding/json" + "errors" + "io" + "net/http" + "net/http/httptest" + "strings" + "sync" + "sync/atomic" + "testing" + "time" +) + +// m6b SL03: CountTokens (AC-4). + +type countRec struct { + mu sync.Mutex + paths []string + bodies []map[string]any + headers []http.Header +} + +func (r *countRec) n() int { + r.mu.Lock() + defer r.mu.Unlock() + return len(r.paths) +} + +// countServer answers request i with steps[i] (last repeats) and records it. +func countServer(t *testing.T, steps ...obsStep) (*httptest.Server, *countRec) { + t.Helper() + rec := &countRec{} + var n atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + raw, _ := io.ReadAll(r.Body) + var b map[string]any + _ = json.Unmarshal(raw, &b) + rec.mu.Lock() + rec.paths = append(rec.paths, r.URL.Path) + rec.bodies = append(rec.bodies, b) + rec.headers = append(rec.headers, r.Header.Clone()) + rec.mu.Unlock() + i := min(int(n.Add(1))-1, len(steps)-1) + s := steps[i] + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(s.status) + _, _ = w.Write([]byte(s.body)) + })) + t.Cleanup(srv.Close) + return srv, rec +} + +const ( + anCountOK = `{"input_tokens":42}` + gmCountOK = `{"totalTokens":17}` +) + +func countClient(t *testing.T, id string, f Format, model string, srv *httptest.Server) *ChatClient { + t.Helper() + cc := newTestClient(t, ProviderConfig{ID: id, Format: f, BaseURL: srv.URL, APIKey: "countkey"}, srv) + cc.model = model + return cc +} + +func countReq() *ChatRequest { + return &ChatRequest{ + Messages: []Message{{Role: RoleUser, Content: "hello"}}, + System: []SystemBlock{{Text: "be brief"}}, + Tools: []ToolDef{{Name: "f", Description: "d", Parameters: json.RawMessage(`{"type":"object"}`)}}, + MaxTokens: 99, + Temperature: 0.5, + TopP: 0.9, + } +} + +func TestCountTokens_AnthropicWire(t *testing.T) { + srv, rec := countServer(t, obsStep{status: 200, body: anCountOK}) + cc := countClient(t, "anthropic", FormatAnthropic, "claude-x", srv) + req := countReq() + req.Thinking = "enabled" + req.ThinkingBudget = 2048 + req.MaxTokens = 8192 + n, err := cc.CountTokens(context.Background(), req) + if err != nil { + t.Fatalf("CountTokens: %v", err) + } + if n != 42 { + t.Errorf("count = %d, want 42", n) + } + if rec.n() != 1 || rec.paths[0] != "/v1/messages/count_tokens" { + t.Fatalf("paths = %v", rec.paths) + } + b := rec.bodies[0] + for _, k := range []string{"stream", "max_tokens", "temperature", "top_p", "top_k"} { + if _, ok := b[k]; ok { + t.Errorf("body has %q: %v", k, b) + } + } + for _, k := range []string{"model", "messages", "system", "tools", "thinking"} { + if _, ok := b[k]; !ok { + t.Errorf("body missing %q: %v", k, b) + } + } + if b["model"] != "claude-x" { + t.Errorf("model = %v", b["model"]) + } + for k := range b { + switch k { + case "model", "messages", "system", "tools", "tool_choice", "thinking": + default: + t.Errorf("unexpected key %q in count body", k) + } + } + if got := rec.headers[0].Get("x-api-key"); got != "countkey" { + t.Errorf("x-api-key = %q", got) + } +} + +func TestCountTokens_AnthropicToolChoiceKept(t *testing.T) { + srv, rec := countServer(t, obsStep{status: 200, body: anCountOK}) + cc := countClient(t, "anthropic", FormatAnthropic, "claude-x", srv) + req := countReq() + req.ToolChoice = &ToolChoice{Mode: ToolChoiceRequired} + if _, err := cc.CountTokens(context.Background(), req); err != nil { + t.Fatal(err) + } + if rec.n() != 1 { + t.Fatalf("requests = %d, want 1", rec.n()) + } + if _, ok := rec.bodies[0]["tool_choice"]; !ok { + t.Errorf("tool_choice missing: %v", rec.bodies[0]) + } +} + +func TestCountTokens_GeminiWire(t *testing.T) { + srv, rec := countServer(t, obsStep{status: 200, body: gmCountOK}) + cc := countClient(t, "gemini", FormatGemini, "gemini-x", srv) + n, err := cc.CountTokens(context.Background(), countReq()) + if err != nil { + t.Fatalf("CountTokens: %v", err) + } + if n != 17 { + t.Errorf("count = %d, want 17", n) + } + if rec.n() != 1 || rec.paths[0] != "/v1beta/models/gemini-x:countTokens" { + t.Fatalf("paths = %v", rec.paths) + } + b := rec.bodies[0] + g, ok := b["generateContentRequest"].(map[string]any) + if !ok || len(b) != 1 { + t.Fatalf("body = %v, want only generateContentRequest", b) + } + if g["model"] != "models/gemini-x" { + t.Errorf("model = %v", g["model"]) + } + for _, k := range []string{"contents", "systemInstruction", "tools"} { + if _, ok := g[k]; !ok { + t.Errorf("generateContentRequest missing %q: %v", k, g) + } + } + if _, ok := g["generationConfig"]; ok { + t.Errorf("generationConfig must not be sent: %v", g) + } + if got := rec.headers[0].Get("x-goog-api-key"); got != "countkey" { + t.Errorf("x-goog-api-key = %q", got) + } +} + +func TestCountTokens_UnsupportedNoNetwork(t *testing.T) { + cases := []struct{ name, model string }{ + {"chat", "gpt-4o"}, + {"responses", "gpt-5.6"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + srv, rec := countServer(t, obsStep{status: 200, body: `{}`}) + cc := countClient(t, "openai", FormatOpenAI, tc.model, srv) + n, err := cc.CountTokens(context.Background(), countReq()) + if !errors.Is(err, ErrUnsupported) { + t.Fatalf("err = %v, want ErrUnsupported", err) + } + var ue *UnsupportedError + if !errors.As(err, &ue) || ue.Op != "CountTokens" { + t.Errorf("err = %#v, want *UnsupportedError{Op:CountTokens}", err) + } + if n != 0 { + t.Errorf("count = %d, want 0 (never an estimate)", n) + } + if rec.n() != 0 { + t.Errorf("made %d HTTP requests, want 0", rec.n()) + } + if strings.Contains(err.Error(), "countkey") { + t.Error("error leaks API key") + } + }) + } +} + +func TestUnsupportedError_Text(t *testing.T) { + e := &UnsupportedError{Op: "CountTokens", Provider: "openai", Reason: "no token counting endpoint"} + want := "llm: CountTokens unsupported on openai: no token counting endpoint" + if e.Error() != want { + t.Errorf("Error() = %q, want %q", e.Error(), want) + } + if !errors.Is(e, ErrUnsupported) { + t.Error("Unwrap must reach ErrUnsupported") + } +} + +func TestCountTokens_ValidationBeforeNetwork(t *testing.T) { + for _, f := range []struct { + id string + format Format + model string + ok string + }{ + {"anthropic", FormatAnthropic, "claude-x", anCountOK}, + {"gemini", FormatGemini, "gemini-x", gmCountOK}, + } { + t.Run(f.id, func(t *testing.T) { + srv, rec := countServer(t, obsStep{status: 200, body: f.ok}) + cc := countClient(t, f.id, f.format, f.model, srv) + bad := []*ChatRequest{ + nil, + {Messages: []Message{{Role: "bogus", Content: "x"}}}, + {Messages: []Message{{Role: RoleUser, Content: "x"}}, Headers: map[string]string{"Bad Name": "v"}}, + } + for i, r := range bad { + _, err := cc.CountTokens(context.Background(), r) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Errorf("case %d: err = %v, want *ConfigError", i, err) + } + } + cc.model = "" + _, err := cc.CountTokens(context.Background(), &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}}) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Errorf("no model: err = %v, want *ConfigError", err) + } + if rec.n() != 0 { + t.Errorf("made %d HTTP requests, want 0", rec.n()) + } + }) + } +} + +func TestCountTokens_ModelFromRequestFallback(t *testing.T) { + srv, rec := countServer(t, obsStep{status: 200, body: gmCountOK}) + cc := countClient(t, "gemini", FormatGemini, "", srv) + req := countReq() + req.Model = "gemini-req" + if _, err := cc.CountTokens(context.Background(), req); err != nil { + t.Fatal(err) + } + if rec.n() != 1 || rec.paths[0] != "/v1beta/models/gemini-req:countTokens" { + t.Errorf("paths = %v", rec.paths) + } +} + +func TestCountTokens_RetriesServerError(t *testing.T) { + for _, f := range []struct { + id string + format Format + model string + ok string + want int + }{ + {"anthropic", FormatAnthropic, "claude-x", anCountOK, 42}, + {"gemini", FormatGemini, "gemini-x", gmCountOK, 17}, + } { + t.Run(f.id, func(t *testing.T) { + srv, rec := countServer(t, obsStep{status: 500, body: `{"error":"boom"}`}, obsStep{status: 200, body: f.ok}) + cc := countClient(t, f.id, f.format, f.model, srv) + n, err := cc.CountTokens(context.Background(), countReq()) + if err != nil { + t.Fatalf("CountTokens: %v", err) + } + if n != f.want || rec.n() != 2 { + t.Errorf("count = %d attempts = %d, want %d / 2", n, rec.n(), f.want) + } + }) + } +} + +func TestCountTokens_RetryPolicyBounded(t *testing.T) { + fastBackoff(t) + srv, rec := countServer(t, obsStep{status: 503, body: `{}`}) + sdk := New(WithRetryPolicy(RetryPolicy{MaxAttempts: 2}), + WithProvider("anthropic", WithAPIKey("k"), WithBaseURL(srv.URL))) + c, err := sdk.Chat("anthropic", "claude-x") + if err != nil { + t.Fatal(err) + } + if _, err := c.CountTokens(context.Background(), countReq()); err == nil { + t.Fatal("want error") + } + if rec.n() != 2 { + t.Errorf("attempts = %d, want 2", rec.n()) + } +} + +func TestCountTokens_WholeCallTimeoutBoundsRetries(t *testing.T) { + srv, rec := countServer(t, obsStep{status: 500, body: `{}`}) + cc := countClient(t, "anthropic", FormatAnthropic, "claude-x", srv) + old := backoffUnit + backoffUnit = 40 * time.Millisecond + t.Cleanup(func() { backoffUnit = old }) + cc.SetRequestTimeout(150 * time.Millisecond) + start := time.Now() + _, err := cc.CountTokens(context.Background(), countReq()) + if err == nil { + t.Fatal("want error") + } + if d := time.Since(start); d > 2*time.Second { + t.Errorf("took %v, whole-call budget not enforced", d) + } + if rec.n() >= 8 { + t.Errorf("attempts = %d, budget should cut retries short", rec.n()) + } +} + +func TestCountTokens_HangingServerHitsBudget(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + select { + case <-r.Context().Done(): + case <-time.After(5 * time.Second): + } + })) + t.Cleanup(srv.Close) + cc := countClient(t, "gemini", FormatGemini, "gemini-x", srv) + cc.SetRequestTimeout(120 * time.Millisecond) + start := time.Now() + if _, err := cc.CountTokens(context.Background(), countReq()); err == nil { + t.Fatal("want deadline error") + } + if time.Since(start) > 3*time.Second { + t.Error("timeout not applied") + } +} + +func TestCountTokens_PerRequestHeaders(t *testing.T) { + for _, f := range []struct { + id string + format Format + model string + ok string + }{ + {"anthropic", FormatAnthropic, "claude-x", anCountOK}, + {"gemini", FormatGemini, "gemini-x", gmCountOK}, + } { + t.Run(f.id, func(t *testing.T) { + srv, rec := countServer(t, obsStep{status: 500, body: `{}`}, obsStep{status: 200, body: f.ok}) + cc := countClient(t, f.id, f.format, f.model, srv) + req := countReq() + req.Headers = map[string]string{"X-Trace": "t-1"} + if _, err := cc.CountTokens(context.Background(), req); err != nil { + t.Fatal(err) + } + if rec.n() != 2 { + t.Fatalf("attempts = %d", rec.n()) + } + for i, h := range rec.headers { + if h.Get("X-Trace") != "t-1" { + t.Errorf("attempt %d: X-Trace = %q", i, h.Get("X-Trace")) + } + } + }) + } +} + +func TestCountTokens_AttemptObserverOperation(t *testing.T) { + for _, f := range []struct { + id string + format Format + model string + ok string + }{ + {"anthropic", FormatAnthropic, "claude-x", anCountOK}, + {"gemini", FormatGemini, "gemini-x", gmCountOK}, + } { + t.Run(f.id, func(t *testing.T) { + fastBackoff(t) + srv, _ := countServer(t, obsStep{status: 500, body: `{}`}, obsStep{status: 200, body: f.ok}) + col := &obsCollector{} + sdk := New(WithAttemptObserver(col.fn), + WithProvider(f.id, WithAPIKey(obsKey), WithBaseURL(srv.URL))) + c, err := sdk.Chat(f.id, f.model) + if err != nil { + t.Fatal(err) + } + if _, err := c.CountTokens(context.Background(), countReq()); err != nil { + t.Fatal(err) + } + evs := col.all() + if len(evs) != 2 { + t.Fatalf("events = %d, want 2", len(evs)) + } + for i, e := range evs { + if e.Operation != "count_tokens" || e.Model != f.model || e.Attempt != i+1 || e.Provider != f.id { + t.Errorf("event %d = %+v", i, e) + } + obsNoKey(t, e) + } + if evs[0].Status != 500 || evs[1].Status != 200 { + t.Errorf("statuses = %d, %d", evs[0].Status, evs[1].Status) + } + }) + } +} + +func TestCountTokens_UnusableSuccessBody(t *testing.T) { + cases := []struct { + name string + f Format + id string + body string + }{ + {"anthropic missing", FormatAnthropic, "anthropic", `{}`}, + {"anthropic negative", FormatAnthropic, "anthropic", `{"input_tokens":-3}`}, + {"anthropic bad json", FormatAnthropic, "anthropic", `not json`}, + {"gemini missing", FormatGemini, "gemini", `{"cachedContentTokenCount":1}`}, + {"gemini negative", FormatGemini, "gemini", `{"totalTokens":-1}`}, + {"gemini bad json", FormatGemini, "gemini", `{`}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + srv, rec := countServer(t, obsStep{status: 200, body: tc.body}) + cc := countClient(t, tc.id, tc.f, "m", srv) + n, err := cc.CountTokens(context.Background(), countReq()) + var ae *APIError + if !errors.As(err, &ae) || ae.Status != 200 { + t.Fatalf("err = %v, want *APIError{Status:200}", err) + } + if n != 0 { + t.Errorf("count = %d, want 0", n) + } + if rec.n() != 1 { + t.Errorf("attempts = %d, want 1 (terminal, not retried)", rec.n()) + } + if strings.Contains(err.Error(), "countkey") { + t.Error("error leaks API key") + } + }) + } +} + +func TestCountTokens_ZeroCountIsValid(t *testing.T) { + srv, _ := countServer(t, obsStep{status: 200, body: `{"input_tokens":0}`}) + cc := countClient(t, "anthropic", FormatAnthropic, "m", srv) + n, err := cc.CountTokens(context.Background(), countReq()) + if err != nil || n != 0 { + t.Errorf("n = %d err = %v, want 0 / nil for an explicit zero", n, err) + } +} + +func TestCountTokens_ErrorNeverLeaksKey(t *testing.T) { + srv, _ := countServer(t, obsStep{status: 401, body: `{"error":{"message":"bad key countkey"}}`}) + cc := countClient(t, "anthropic", FormatAnthropic, "m", srv) + _, err := cc.CountTokens(context.Background(), countReq()) + if err == nil { + t.Fatal("want error") + } + var ae *APIError + if !errors.As(err, &ae) || ae.Status != 401 { + t.Errorf("err = %v, want 401 APIError", err) + } +} + +func TestCountTokens_RequestNotMutated(t *testing.T) { + srv, _ := countServer(t, obsStep{status: 200, body: anCountOK}) + cc := countClient(t, "anthropic", FormatAnthropic, "claude-x", srv) + req := countReq() + before, _ := json.Marshal(req) + if _, err := cc.CountTokens(context.Background(), req); err != nil { + t.Fatal(err) + } + after, _ := json.Marshal(req) + if string(before) != string(after) { + t.Errorf("request mutated: %s -> %s", before, after) + } +} diff --git a/m6b_mime_test.go b/m6b_mime_test.go new file mode 100644 index 0000000..2b97bc0 --- /dev/null +++ b/m6b_mime_test.go @@ -0,0 +1,77 @@ +package llm + +import "testing" + +// MIME types are case-insensitive (RFC 2045): any case is accepted at the +// boundary and the canonical lower-case form goes on the wire. + +func TestM6bDocumentMIMECaseInsensitive(t *testing.T) { + for _, mt := range []string{"APPLICATION/PDF", "Application/Pdf"} { + msg := m6bUser(DocumentPart(mt, m6bPDF, "a.pdf")) + + m, _, err := m6bBuild(FormatOpenAI, "gpt-4o", msg) + if err != nil { + t.Errorf("openai %q: %v", mt, err) + } else { + jsonEq(t, "openai "+mt, m6bGet(t, m, "messages", 0, "content", 0), + `{"type":"file","file":{"filename":"a.pdf","file_data":"data:application/pdf;base64,AQID"}}`) + } + + m, _, err = m6bBuild(FormatOpenAI, "gpt-5.6", msg) + if err != nil { + t.Errorf("responses %q: %v", mt, err) + } else { + jsonEq(t, "responses "+mt, m6bGet(t, m, "input", 0, "content", 0), + `{"type":"input_file","filename":"a.pdf","file_data":"data:application/pdf;base64,AQID"}`) + } + + m, _, err = m6bBuild(FormatAnthropic, "claude", msg) + if err != nil { + t.Errorf("anthropic %q: %v", mt, err) + } else { + jsonEq(t, "anthropic "+mt, m6bGet(t, m, "messages", 0, "content", 0), + `{"type":"document","source":{"type":"base64","media_type":"application/pdf","data":"AQID"},"title":"a.pdf"}`) + } + + m, _, err = m6bBuild(FormatGemini, "gemini", msg) + if err != nil { + t.Errorf("gemini %q: %v", mt, err) + } else { + jsonEq(t, "gemini "+mt, m6bGet(t, m, "contents", 0, "parts", 0), + `{"inlineData":{"mimeType":"application/pdf","data":"AQID"}}`) + } + } +} + +func TestM6bDocumentTextPlainMIMECaseInsensitive(t *testing.T) { + msg := m6bUser(DocumentPart("TEXT/Plain", []byte("hello"), "n.txt")) + m, _, err := m6bBuild(FormatAnthropic, "claude", msg) + if err != nil { + t.Fatalf("anthropic: %v", err) + } + jsonEq(t, "anthropic", m6bGet(t, m, "messages", 0, "content", 0), + `{"type":"document","source":{"type":"text","media_type":"text/plain","data":"hello"},"title":"n.txt"}`) + m, _, err = m6bBuild(FormatGemini, "gemini", msg) + if err != nil { + t.Fatalf("gemini: %v", err) + } + jsonEq(t, "gemini", m6bGet(t, m, "contents", 0, "parts", 0), `{"inlineData":{"mimeType":"text/plain","data":"aGVsbG8="}}`) +} + +func TestM6bImageURLMIMECaseInsensitive(t *testing.T) { + const u = "https://x/y.png" + cases := map[string]string{"IMAGE/PNG": "image/png", "Image/Jpeg": "image/jpeg", "IMAGE/JPG": "image/jpeg"} + for in, want := range cases { + for _, f := range m6bFormats { + if _, _, err := m6bBuild(f.format, f.model, m6bUser(ImageURLPartMIME(u, in))); err != nil { + t.Errorf("%s %q: %v", f.name, in, err) + } + } + m, _, err := m6bBuild(FormatGemini, "gemini", m6bUser(ImageURLPartMIME(u, in))) + if err != nil { + continue + } + jsonEq(t, "gemini "+in, m6bGet(t, m, "contents", 0, "parts", 0), + `{"fileData":{"fileUri":"https://x/y.png","mimeType":"`+want+`"}}`) + } +} diff --git a/m6b_parts_test.go b/m6b_parts_test.go new file mode 100644 index 0000000..d131b16 --- /dev/null +++ b/m6b_parts_test.go @@ -0,0 +1,463 @@ +package llm + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "net/http" + "net/http/httptest" + "strings" + "sync/atomic" + "testing" +) + +// m6b: DocumentPart and ImageURLPart on every format (AC-1, AC-2). + +var m6bFormats = []struct { + name string + format Format + model string +}{ + {"openai", FormatOpenAI, "gpt-4o"}, + {"responses", FormatOpenAI, "gpt-5.6"}, + {"anthropic", FormatAnthropic, "claude"}, + {"gemini", FormatGemini, "gemini"}, +} + +var m6bTools = []ToolDef{{Name: "wx", Parameters: json.RawMessage(`{"type":"object"}`)}} + +// m6bBuild renders req through the real boundary + builder path. Tools are +// attached so gpt-5.6 routes to /responses. +func m6bBuild(format Format, model string, msgs ...Message) (map[string]any, string, error) { + pc := newProviderClient(ProviderConfig{ID: string(format), Format: format, BaseURL: "https://x"}, nil, nil) + b, url, err := pc.buildChatRequest(&ChatRequest{Messages: msgs, Tools: m6bTools}, model, false) + if err != nil { + return nil, url, err + } + var m map[string]any + if jerr := json.Unmarshal(b, &m); jerr != nil { + return nil, url, jerr + } + return m, url, nil +} + +func m6bMustBuild(t *testing.T, format Format, model string, msgs ...Message) map[string]any { + t.Helper() + m, _, err := m6bBuild(format, model, msgs...) + if err != nil { + t.Fatalf("%s: build: %v", format, err) + } + return m +} + +// m6bWantConfigErr requires a *ConfigError that is a deliberate rejection: +// never the generic unknown-part-type error a not-yet-supported type hits. +func m6bWantConfigErr(t *testing.T, name string, err error, contains ...string) { + t.Helper() + if err == nil { + t.Errorf("%s: accepted, want *ConfigError", name) + return + } + var ce *ConfigError + if !errors.As(err, &ce) { + t.Errorf("%s: err = %T %v, want *ConfigError", name, err, err) + return + } + if strings.Contains(err.Error(), "unknown content part type") { + t.Errorf("%s: rejected as unknown part type, want a specific rejection: %v", name, err) + } + for _, c := range contains { + if !strings.Contains(err.Error(), c) { + t.Errorf("%s: error %q lacks %q", name, err.Error(), c) + } + } +} + +func m6bGet(t *testing.T, v any, path ...any) any { + t.Helper() + cur := v + for _, p := range path { + switch k := p.(type) { + case string: + m, ok := cur.(map[string]any) + if !ok { + t.Fatalf("path %v: %T is not an object at %q", path, cur, k) + } + cur = m[k] + case int: + a, ok := cur.([]any) + if !ok || k >= len(a) { + t.Fatalf("path %v: no index %d in %T", path, k, cur) + } + cur = a[k] + } + } + return cur +} + +var ( + m6bPDF = []byte{1, 2, 3} // base64 "AQID" + m6bUser = func(parts ...ContentPart) Message { return Message{Role: RoleUser, Parts: parts} } +) + +func TestM6bConstructors(t *testing.T) { + d := DocumentPart("application/pdf", m6bPDF, "a.pdf") + if d.Type != ContentPartDocument || d.MIMEType != "application/pdf" || !bytes.Equal(d.Data, m6bPDF) || d.Name != "a.pdf" || d.Text != "" || d.Image != nil || d.URL != "" { + t.Errorf("DocumentPart = %+v", d) + } + u := ImageURLPart("https://x/y.png") + if u.Type != ContentPartImageURL || u.URL != "https://x/y.png" || u.MIMEType != "" || u.Image != nil { + t.Errorf("ImageURLPart = %+v", u) + } + um := ImageURLPartMIME("https://x/y.png", "image/png") + if um.Type != ContentPartImageURL || um.URL != "https://x/y.png" || um.MIMEType != "image/png" { + t.Errorf("ImageURLPartMIME = %+v", um) + } + w := u.WithMIME("image/webp") + if w.Type != ContentPartImageURL || w.URL != u.URL || w.MIMEType != "image/webp" { + t.Errorf("WithMIME = %+v", w) + } + if u.MIMEType != "" { + t.Errorf("WithMIME mutated the receiver: %+v", u) + } + if ContentPartDocument != "document" || ContentPartImageURL != "image_url" { + t.Errorf("part type consts = %q %q", ContentPartDocument, ContentPartImageURL) + } + if MaxDocumentBytes != 20<<20 { + t.Errorf("MaxDocumentBytes = %d", MaxDocumentBytes) + } +} + +// ── AC-1: documents on every format ────────────────────────────────────── + +func TestM6bDocumentWirePDF(t *testing.T) { + msg := m6bUser(TextPart("read"), DocumentPart("application/pdf", m6bPDF, "a.pdf")) + + m := m6bMustBuild(t, FormatOpenAI, "gpt-4o", msg) + jsonEq(t, "openai content", m6bGet(t, m, "messages", 0, "content"), + `[{"type":"text","text":"read"},{"type":"file","file":{"filename":"a.pdf","file_data":"data:application/pdf;base64,AQID"}}]`) + + m = m6bMustBuild(t, FormatOpenAI, "gpt-5.6", msg) + jsonEq(t, "responses content", m6bGet(t, m, "input", 0, "content"), + `[{"type":"input_text","text":"read"},{"type":"input_file","filename":"a.pdf","file_data":"data:application/pdf;base64,AQID"}]`) + + m = m6bMustBuild(t, FormatAnthropic, "claude", msg) + jsonEq(t, "anthropic content", m6bGet(t, m, "messages", 0, "content"), + `[{"type":"text","text":"read"},{"type":"document","source":{"type":"base64","media_type":"application/pdf","data":"AQID"},"title":"a.pdf"}]`) + + m = m6bMustBuild(t, FormatGemini, "gemini", msg) + jsonEq(t, "gemini parts", m6bGet(t, m, "contents", 0, "parts"), + `[{"text":"read"},{"inlineData":{"mimeType":"application/pdf","data":"AQID"}}]`) +} + +func TestM6bDocumentDefaultNameOpenAIFormats(t *testing.T) { + msg := m6bUser(DocumentPart("application/pdf", m6bPDF, "")) + m := m6bMustBuild(t, FormatOpenAI, "gpt-4o", msg) + jsonEq(t, "openai", m6bGet(t, m, "messages", 0, "content", 0), + `{"type":"file","file":{"filename":"document.pdf","file_data":"data:application/pdf;base64,AQID"}}`) + m = m6bMustBuild(t, FormatOpenAI, "gpt-5.6", msg) + jsonEq(t, "responses", m6bGet(t, m, "input", 0, "content", 0), + `{"type":"input_file","filename":"document.pdf","file_data":"data:application/pdf;base64,AQID"}`) + // Anthropic title is omitted when no name is given. + m = m6bMustBuild(t, FormatAnthropic, "claude", msg) + jsonEq(t, "anthropic", m6bGet(t, m, "messages", 0, "content", 0), + `{"type":"document","source":{"type":"base64","media_type":"application/pdf","data":"AQID"}}`) +} + +func TestM6bDocumentTextPlain(t *testing.T) { + msg := m6bUser(DocumentPart("text/plain", []byte("hello"), "n.txt")) + + m := m6bMustBuild(t, FormatAnthropic, "claude", msg) + jsonEq(t, "anthropic", m6bGet(t, m, "messages", 0, "content", 0), + `{"type":"document","source":{"type":"text","media_type":"text/plain","data":"hello"},"title":"n.txt"}`) + + m = m6bMustBuild(t, FormatGemini, "gemini", msg) + jsonEq(t, "gemini", m6bGet(t, m, "contents", 0, "parts", 0), + `{"inlineData":{"mimeType":"text/plain","data":"aGVsbG8="}}`) + + _, _, err := m6bBuild(FormatOpenAI, "gpt-4o", msg) + m6bWantConfigErr(t, "openai text/plain", err, "application/pdf only") + _, _, err = m6bBuild(FormatOpenAI, "gpt-5.6", msg) + m6bWantConfigErr(t, "responses text/plain", err, "application/pdf only") +} + +func TestM6bDocumentBoundaryErrors(t *testing.T) { + big := make([]byte, MaxDocumentBytes+1) + cases := []struct { + name string + part ContentPart + }{ + {"image mime", ContentPart{Type: ContentPartDocument, MIMEType: "image/png", Data: m6bPDF}}, + {"empty mime", ContentPart{Type: ContentPartDocument, Data: m6bPDF}}, + {"mime with params", ContentPart{Type: ContentPartDocument, MIMEType: "application/pdf; charset=x", Data: m6bPDF}}, + {"unsupported mime", ContentPart{Type: ContentPartDocument, MIMEType: "application/zip", Data: m6bPDF}}, + {"empty data", ContentPart{Type: ContentPartDocument, MIMEType: "application/pdf"}}, + {"oversized", ContentPart{Type: ContentPartDocument, MIMEType: "application/pdf", Data: big}}, + {"text set", ContentPart{Type: ContentPartDocument, MIMEType: "application/pdf", Data: m6bPDF, Text: "x"}}, + {"image set", ContentPart{Type: ContentPartDocument, MIMEType: "application/pdf", Data: m6bPDF, Image: []byte{1}}}, + {"url set", ContentPart{Type: ContentPartDocument, MIMEType: "application/pdf", Data: m6bPDF, URL: "https://x/a.pdf"}}, + } + for _, f := range m6bFormats { + for _, c := range cases { + _, _, err := m6bBuild(f.format, f.model, m6bUser(c.part)) + m6bWantConfigErr(t, f.name+"/"+c.name, err, "message 0 part 0") + } + } +} + +func TestM6bDocumentAtLimitAccepted(t *testing.T) { + msg := m6bUser(DocumentPart("application/pdf", make([]byte, MaxDocumentBytes), "big.pdf")) + m := m6bMustBuild(t, FormatAnthropic, "claude", msg) + if got := m6bGet(t, m, "messages", 0, "content", 0, "type"); got != "document" { + t.Errorf("type = %v", got) + } +} + +func TestM6bDocumentRoleRules(t *testing.T) { + doc := DocumentPart("application/pdf", m6bPDF, "a.pdf") + for _, role := range []Role{RoleAssistant, RoleSystem} { + _, _, err := m6bBuild(FormatAnthropic, "claude", Message{Role: role, Parts: []ContentPart{doc}}) + m6bWantConfigErr(t, string(role), err, "require the user or tool role") + } + // Content and Parts together stay rejected. + _, _, err := m6bBuild(FormatAnthropic, "claude", Message{Role: RoleUser, Content: "x", Parts: []ContentPart{doc}}) + m6bWantConfigErr(t, "content+parts", err, "cannot both be set") +} + +func TestM6bTextAndImagePartsRejectForeignFields(t *testing.T) { + cases := []struct { + name string + part ContentPart + }{ + {"text+url", ContentPart{Type: ContentPartText, Text: "x", URL: "https://x/y.png"}}, + {"text+data", ContentPart{Type: ContentPartText, Text: "x", Data: m6bPDF}}, + {"text+name", ContentPart{Type: ContentPartText, Text: "x", Name: "n"}}, + {"image+data", ContentPart{Type: ContentPartImage, MIMEType: "image/png", Image: []byte{1}, Data: m6bPDF}}, + {"image+url", ContentPart{Type: ContentPartImage, MIMEType: "image/png", Image: []byte{1}, URL: "https://x/y.png"}}, + {"image+name", ContentPart{Type: ContentPartImage, MIMEType: "image/png", Image: []byte{1}, Name: "n"}}, + } + for _, c := range cases { + _, _, err := m6bBuild(FormatOpenAI, "gpt-4o", m6bUser(c.part)) + m6bWantConfigErr(t, c.name, err, "message 0 part 0") + } +} + +func TestM6bAggregateInlineCapCountsDocuments(t *testing.T) { + doc := func(n int) ContentPart { return DocumentPart("application/pdf", make([]byte, n), "d.pdf") } + cases := []struct { + name string + parts []ContentPart + }{ + {"documents only", []ContentPart{doc(12 << 20), doc(12 << 20), doc(12 << 20)}}, + {"images plus document", []ContentPart{ + ImagePart("image/png", make([]byte, 10<<20)), ImagePart("image/png", make([]byte, 10<<20)), doc(13 << 20)}}, + } + for _, f := range m6bFormats { + for _, c := range cases { + _, _, err := m6bBuild(f.format, f.model, m6bUser(c.parts...)) + m6bWantConfigErr(t, f.name+"/"+c.name, err, "exceed") + } + } + // Exactly at the cap is accepted. + half := (MaxRequestImageBytes - MaxDocumentBytes) / 2 + ok := []ContentPart{doc(MaxDocumentBytes), ImagePart("image/png", make([]byte, half)), ImagePart("image/png", make([]byte, half))} + if _, _, err := m6bBuild(FormatAnthropic, "claude", m6bUser(ok...)); err != nil { + t.Errorf("at-cap request rejected: %v", err) + } +} + +// ── AC-2: image URLs on every format ───────────────────────────────────── + +func TestM6bImageURLWire(t *testing.T) { + const u = "https://x/y.png" + plain := m6bUser(TextPart("see"), ImageURLPart(u)) + hinted := m6bUser(TextPart("see"), ImageURLPartMIME(u, "image/png")) + + m := m6bMustBuild(t, FormatOpenAI, "gpt-4o", plain) + jsonEq(t, "openai", m6bGet(t, m, "messages", 0, "content", 1), `{"type":"image_url","image_url":{"url":"https://x/y.png"}}`) + + m = m6bMustBuild(t, FormatOpenAI, "gpt-5.6", plain) + jsonEq(t, "responses", m6bGet(t, m, "input", 0, "content", 1), `{"type":"input_image","image_url":"https://x/y.png"}`) + + for _, msg := range []Message{plain, hinted} { // Anthropic ignores a MIME hint + m = m6bMustBuild(t, FormatAnthropic, "claude", msg) + jsonEq(t, "anthropic", m6bGet(t, m, "messages", 0, "content", 1), `{"type":"image","source":{"type":"url","url":"https://x/y.png"}}`) + } + + for name, p := range map[string]ContentPart{ + "ImageURLPartMIME": ImageURLPartMIME(u, "image/png"), + "WithMIME": ImageURLPart(u).WithMIME("image/png"), + } { + m = m6bMustBuild(t, FormatGemini, "gemini", m6bUser(TextPart("see"), p)) + jsonEq(t, "gemini "+name, m6bGet(t, m, "contents", 0, "parts", 1), `{"fileData":{"fileUri":"https://x/y.png","mimeType":"image/png"}}`) + } + + m = m6bMustBuild(t, FormatGemini, "gemini", m6bUser(ImageURLPartMIME(u, "image/jpg"))) + jsonEq(t, "gemini jpg alias", m6bGet(t, m, "contents", 0, "parts", 0), `{"fileData":{"fileUri":"https://x/y.png","mimeType":"image/jpeg"}}`) +} + +func TestM6bImageURLSchemeCaseInsensitive(t *testing.T) { + for _, f := range m6bFormats { + msg := m6bUser(ImageURLPartMIME("HTTP://x/y.png", "image/png")) + if _, _, err := m6bBuild(f.format, f.model, msg); err != nil { + t.Errorf("%s: uppercase http scheme rejected: %v", f.name, err) + } + } +} + +func TestM6bGeminiImageURLRequiresMIME(t *testing.T) { + _, _, err := m6bBuild(FormatGemini, "gemini", m6bUser(ImageURLPart("https://x/y.png"))) + m6bWantConfigErr(t, "gemini no mime", err, "require a MIME type") + // The other formats do not need a MIME for a URL image. + for _, f := range m6bFormats[:3] { + if _, _, err := m6bBuild(f.format, f.model, m6bUser(ImageURLPart("https://x/y.png"))); err != nil { + t.Errorf("%s: URL image without MIME rejected: %v", f.name, err) + } + } +} + +func TestM6bImageURLBoundaryErrors(t *testing.T) { + cases := []struct { + name string + part ContentPart + }{ + {"data url", m6bURLPart("data:image/png;base64,AAAA")}, + {"ftp", m6bURLPart("ftp://x/y.png")}, + {"file", m6bURLPart("file:///etc/passwd")}, + {"empty host", m6bURLPart("https:///y.png")}, + {"malformed", m6bURLPart("http://[::1")}, + {"empty url", m6bURLPart("")}, + {"relative", m6bURLPart("/y.png")}, + {"too long", m6bURLPart("https://x/" + strings.Repeat("a", 9<<10))}, + {"image bytes set", ContentPart{Type: ContentPartImageURL, URL: "https://x/y.png", MIMEType: "image/png", Image: []byte{1}}}, + {"text set", ContentPart{Type: ContentPartImageURL, URL: "https://x/y.png", MIMEType: "image/png", Text: "x"}}, + {"data set", ContentPart{Type: ContentPartImageURL, URL: "https://x/y.png", MIMEType: "image/png", Data: m6bPDF}}, + {"non-image mime hint", m6bURLPart("https://x/y.png").WithMIME("text/plain")}, + {"pdf mime hint", m6bURLPart("https://x/y.png").WithMIME("application/pdf")}, + } + for _, f := range m6bFormats { + for _, c := range cases { + _, _, err := m6bBuild(f.format, f.model, m6bUser(c.part)) + m6bWantConfigErr(t, f.name+"/"+c.name, err, "message 0 part 0") + } + } +} + +// m6bURLPart builds an image_url part directly (not via the constructor) +// so boundary cases do not depend on the constructors under test. +func m6bURLPart(u string) ContentPart { return ContentPart{Type: ContentPartImageURL, URL: u} } + +// The SDK never fetches an image URL itself: it only forwards the string. +func TestM6bImageURLNeverFetchedBySDK(t *testing.T) { + var imageHits atomic.Int32 + imgSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + imageHits.Add(1) + w.WriteHeader(200) + })) + defer imgSrv.Close() + imgURL := imgSrv.URL + "/y.png" + + var gotBody atomic.Value + api := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + b := new(bytes.Buffer) + _, _ = b.ReadFrom(r.Body) + gotBody.Store(b.String()) + fmt.Fprint(w, `{"choices":[{"message":{"role":"assistant","content":"ok"},"finish_reason":"stop"}]}`) + })) + defer api.Close() + + cc := newTestClient(t, ProviderConfig{ID: "openai", Format: FormatOpenAI, BaseURL: api.URL, APIKey: "k"}, api) + res, err := cc.Call(context.Background(), &ChatRequest{Messages: []Message{m6bUser(TextPart("see"), ImageURLPart(imgURL))}}) + if err != nil { + t.Fatalf("Call: %v", err) + } + if res.Content != "ok" { + t.Fatalf("content = %q", res.Content) + } + if body, _ := gotBody.Load().(string); !strings.Contains(body, imgURL) { + t.Errorf("request body does not carry the URL verbatim: %s", body) + } + if n := imageHits.Load(); n != 0 { + t.Errorf("image URL fetched %d times by the SDK, want 0", n) + } +} + +// Invalid parts fail before any HTTP request is made. +func TestM6bInvalidPartsFailBeforeNetwork(t *testing.T) { + var hits atomic.Int32 + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hits.Add(1) + w.WriteHeader(200) + })) + defer srv.Close() + bad := []ContentPart{ + {Type: ContentPartDocument, MIMEType: "image/png", Data: m6bPDF}, + m6bURLPart("ftp://x/y.png"), + } + for _, f := range []Format{FormatOpenAI, FormatAnthropic, FormatGemini} { + cc := newTestClient(t, ProviderConfig{ID: string(f), Format: f, BaseURL: srv.URL, APIKey: "k"}, srv) + for i, p := range bad { + _, err := cc.Call(context.Background(), &ChatRequest{Messages: []Message{m6bUser(p)}}) + m6bWantConfigErr(t, fmt.Sprintf("%s/%d", f, i), err, "message 0 part 0") + } + } + if n := hits.Load(); n != 0 { + t.Errorf("%d HTTP requests made for invalid parts", n) + } +} + +// ── tool-result placement ──────────────────────────────────────────────── + +func m6bToolTurn(parts ...ContentPart) []Message { + return []Message{ + {Role: RoleUser, Content: "go"}, + {Role: RoleAssistant, ToolCalls: []ToolCall{{ID: "c1", Name: "wx", Arguments: `{}`}}}, + {Role: RoleTool, ToolCallID: "c1", ToolName: "wx", Parts: parts}, + } +} + +func TestM6bToolResultAnthropic(t *testing.T) { + m := m6bMustBuild(t, FormatAnthropic, "claude", m6bToolTurn( + TextPart("r"), DocumentPart("application/pdf", m6bPDF, "a.pdf"), ImageURLPart("https://x/y.png"))...) + jsonEq(t, "tool_result", m6bGet(t, m, "messages", 2, "content", 0, "content"), + `[{"type":"text","text":"r"},`+ + `{"type":"document","source":{"type":"base64","media_type":"application/pdf","data":"AQID"},"title":"a.pdf"},`+ + `{"type":"image","source":{"type":"url","url":"https://x/y.png"}}]`) +} + +func TestM6bToolResultGemini(t *testing.T) { + m := m6bMustBuild(t, FormatGemini, "gemini", m6bToolTurn( + TextPart("r"), DocumentPart("application/pdf", m6bPDF, "a.pdf"), ImageURLPartMIME("https://x/y.png", "image/png"))...) + parts := m6bGet(t, m, "contents", 2, "parts") + jsonEq(t, "functionResponse", m6bGet(t, parts, 0), `{"functionResponse":{"name":"wx","response":{"result":"r"}}}`) + jsonEq(t, "document", m6bGet(t, parts, 1), `{"inlineData":{"mimeType":"application/pdf","data":"AQID"}}`) + jsonEq(t, "image url", m6bGet(t, parts, 2), `{"fileData":{"fileUri":"https://x/y.png","mimeType":"image/png"}}`) +} + +func TestM6bToolResultOpenAIChat(t *testing.T) { + m := m6bMustBuild(t, FormatOpenAI, "gpt-4o", m6bToolTurn(TextPart("chart"), ImageURLPart("https://x/y.png"))...) + msgs := m6bGet(t, m, "messages").([]any) + if len(msgs) != 4 { + t.Fatalf("messages = %d, want user, assistant, tool, user", len(msgs)) + } + jsonEq(t, "tool message", m6bGet(t, msgs, 2, "content"), `"chart"`) + if role := m6bGet(t, msgs, 3, "role"); role != "user" { + t.Errorf("post-tool message role = %v, want user", role) + } + content := m6bGet(t, msgs, 3, "content").([]any) + jsonEq(t, "image rides user message", content[len(content)-1], `{"type":"image_url","image_url":{"url":"https://x/y.png"}}`) + + _, _, err := m6bBuild(FormatOpenAI, "gpt-4o", m6bToolTurn(TextPart("r"), DocumentPart("application/pdf", m6bPDF, "a.pdf"))...) + m6bWantConfigErr(t, "openai tool doc", err, "tool results cannot carry documents on OpenAI-format providers") +} + +func TestM6bToolResultResponses(t *testing.T) { + m := m6bMustBuild(t, FormatOpenAI, "gpt-5.6", m6bToolTurn(TextPart("chart"), ImageURLPart("https://x/y.png"))...) + jsonEq(t, "function_call_output", m6bGet(t, m, "input", 2), + `{"type":"function_call_output","call_id":"c1","output":[{"type":"input_text","text":"chart"},{"type":"input_image","image_url":"https://x/y.png"}]}`) + + _, _, err := m6bBuild(FormatOpenAI, "gpt-5.6", m6bToolTurn(TextPart("r"), DocumentPart("application/pdf", m6bPDF, "a.pdf"))...) + m6bWantConfigErr(t, "responses tool doc", err, "tool results cannot carry documents on the Responses API") +} diff --git a/m6b_tools_test.go b/m6b_tools_test.go new file mode 100644 index 0000000..bae64dc --- /dev/null +++ b/m6b_tools_test.go @@ -0,0 +1,541 @@ +package llm + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "net/http/httptest" + "reflect" + "strings" + "sync" + "sync/atomic" + "testing" + "time" +) + +// m6b SL02: RunTools (AC-3). + +// toolScript is a scripted multi-turn provider: response i answers request i; +// every request body is recorded. +type toolScript struct { + mu sync.Mutex + bodies []map[string]any + paths []string + resp []string +} + +func newToolServer(t *testing.T, resp ...string) (*httptest.Server, *toolScript) { + t.Helper() + s := &toolScript{resp: resp} + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + raw, _ := io.ReadAll(r.Body) + var m map[string]any + _ = json.Unmarshal(raw, &m) + s.mu.Lock() + i := len(s.bodies) + s.bodies = append(s.bodies, m) + s.paths = append(s.paths, r.URL.Path) + s.mu.Unlock() + if i >= len(s.resp) { + w.WriteHeader(400) + fmt.Fprint(w, `{"error":{"message":"script exhausted"}}`) + return + } + w.Header().Set("Content-Type", "application/json") + fmt.Fprint(w, s.resp[i]) + })) + t.Cleanup(srv.Close) + return srv, s +} + +func (s *toolScript) count() int { + s.mu.Lock() + defer s.mu.Unlock() + return len(s.bodies) +} + +func (s *toolScript) body(t *testing.T, i int) map[string]any { + t.Helper() + s.mu.Lock() + defer s.mu.Unlock() + if i >= len(s.bodies) { + t.Fatalf("only %d requests recorded, want index %d", len(s.bodies), i) + } + return s.bodies[i] +} + +// toolFormat describes one wire format's scripted responses. +type toolFormat struct { + name string + format Format + model string + // callTurn answers with the given calls; stop/length answer without calls. + call func(calls ...[2]string) string // {id,name} + stop string + length string + // replay checks the second request body against the first assistant turn. + replay func(t *testing.T, body map[string]any) +} + +func openaiCalls(calls ...[2]string) string { + var cs []string + for _, c := range calls { + cs = append(cs, fmt.Sprintf(`{"id":%q,"type":"function","function":{"name":%q,"arguments":"{}"}}`, c[0], c[1])) + } + return `{"choices":[{"message":{"role":"assistant","content":null,"tool_calls":[` + strings.Join(cs, ",") + `]},"finish_reason":"tool_calls"}]}` +} + +func responsesCalls(calls ...[2]string) string { + var cs []string + for _, c := range calls { + cs = append(cs, fmt.Sprintf(`{"type":"function_call","call_id":%q,"name":%q,"arguments":"{}"}`, c[0], c[1])) + } + return `{"status":"completed","output":[` + strings.Join(cs, ",") + `]}` +} + +func anthropicCalls(calls ...[2]string) string { + cs := []string{`{"type":"thinking","thinking":"plan A","signature":"SIG-A"}`, `{"type":"redacted_thinking","data":"ENC-B"}`} + for _, c := range calls { + cs = append(cs, fmt.Sprintf(`{"type":"tool_use","id":%q,"name":%q,"input":{}}`, c[0], c[1])) + } + return `{"role":"assistant","content":[` + strings.Join(cs, ",") + `],"stop_reason":"tool_use","usage":{"input_tokens":1,"output_tokens":1}}` +} + +func geminiCalls(calls ...[2]string) string { + var cs []string + for i, c := range calls { + sig := "" + if i == 0 { + sig = `,"thoughtSignature":"GSIG-1"` + } + cs = append(cs, fmt.Sprintf(`{"functionCall":{"name":%q,"args":{}}%s}`, c[1], sig)) + } + return `{"candidates":[{"content":{"role":"model","parts":[` + strings.Join(cs, ",") + `]},"finishReason":"STOP"}]}` +} + +var toolFormats = []toolFormat{ + { + name: "openai", format: FormatOpenAI, model: "gpt-4o", call: openaiCalls, + stop: `{"choices":[{"message":{"role":"assistant","content":"done"},"finish_reason":"stop"}]}`, + length: `{"choices":[{"message":{"role":"assistant","content":"cut"},"finish_reason":"length"}]}`, + replay: func(t *testing.T, b map[string]any) { + msgs := b["messages"].([]any) + asst := msgs[1].(map[string]any) + tcs, _ := asst["tool_calls"].([]any) + if len(tcs) == 0 { + t.Fatalf("assistant tool_calls not replayed: %v", asst) + } + if first := tcs[0].(map[string]any); first["id"] != "c1" { + t.Errorf("first tool call id = %v", first["id"]) + } + }, + }, + { + name: "responses", format: FormatOpenAI, model: "gpt-5.6", call: responsesCalls, + stop: `{"status":"completed","output":[{"type":"message","role":"assistant","content":[{"type":"output_text","text":"done"}]}]}`, + length: `{"status":"incomplete","incomplete_details":{"reason":"max_output_tokens"},"output":[{"type":"message","role":"assistant","content":[{"type":"output_text","text":"cut"}]}]}`, + replay: func(t *testing.T, b map[string]any) { + var types []string + var ids []string + for _, it := range b["input"].([]any) { + m := it.(map[string]any) + types = append(types, fmt.Sprint(m["type"])) + if m["type"] == "function_call_output" { + ids = append(ids, fmt.Sprint(m["call_id"])) + } + } + joined := strings.Join(types, ",") + if !strings.Contains(joined, "function_call,") && !strings.HasSuffix(joined, "function_call") && !strings.Contains(joined, "function_call,function_call_output") { + t.Errorf("input item types = %s, want function_call then function_call_output", joined) + } + if len(ids) == 0 || ids[0] != "c1" { + t.Errorf("function_call_output ids = %v", ids) + } + }, + }, + { + name: "anthropic", format: FormatAnthropic, model: "claude-x", call: anthropicCalls, + stop: `{"role":"assistant","content":[{"type":"text","text":"done"}],"stop_reason":"end_turn"}`, + length: `{"role":"assistant","content":[{"type":"text","text":"cut"}],"stop_reason":"max_tokens"}`, + replay: func(t *testing.T, b map[string]any) { + msgs := b["messages"].([]any) + asst := msgs[1].(map[string]any) + blocks := asst["content"].([]any) + if len(blocks) < 3 { + t.Fatalf("assistant blocks = %v", blocks) + } + b0, b1, b2 := blocks[0].(map[string]any), blocks[1].(map[string]any), blocks[2].(map[string]any) + if b0["type"] != "thinking" || b0["signature"] != "SIG-A" || b0["thinking"] != "plan A" { + t.Errorf("block 0 = %v, want signed thinking first", b0) + } + if b1["type"] != "redacted_thinking" || b1["data"] != "ENC-B" { + t.Errorf("block 1 = %v, want redacted_thinking", b1) + } + if b2["type"] != "tool_use" || b2["id"] != "c1" { + t.Errorf("block 2 = %v, want tool_use c1", b2) + } + // tool results ride one user message, in call order. + user := msgs[2].(map[string]any) + if user["role"] != "user" { + t.Errorf("results role = %v", user["role"]) + } + }, + }, + { + name: "gemini", format: FormatGemini, model: "gemini-x", call: geminiCalls, + stop: `{"candidates":[{"content":{"role":"model","parts":[{"text":"done"}]},"finishReason":"STOP"}]}`, + length: `{"candidates":[{"content":{"role":"model","parts":[{"text":"cut"}]},"finishReason":"MAX_TOKENS"}]}`, + replay: func(t *testing.T, b map[string]any) { + contents := b["contents"].([]any) + model := contents[1].(map[string]any) + parts := model["parts"].([]any) + p0 := parts[0].(map[string]any) + if _, ok := p0["functionCall"]; !ok || p0["thoughtSignature"] != "GSIG-1" { + t.Errorf("first model part = %v, want functionCall with thoughtSignature GSIG-1", p0) + } + }, + }, +} + +func (f toolFormat) client(t *testing.T, srv *httptest.Server) *ChatClient { + t.Helper() + cc := newTestClient(t, ProviderConfig{ID: string(f.format), Format: f.format, BaseURL: srv.URL, APIKey: "k"}, srv) + cc.model = f.model + return cc +} + +func (f toolFormat) req() ChatRequest { + r := ChatRequest{ + Model: f.model, + Messages: []Message{{Role: RoleUser, Content: "go"}}, + Tools: []ToolDef{{Name: "a", Parameters: json.RawMessage(`{"type":"object"}`)}, {Name: "b", Parameters: json.RawMessage(`{"type":"object"}`)}}, + } + if f.format == FormatAnthropic { + r.Thinking = "enabled" + r.MaxTokens = 4096 + } + return r +} + +func mustLoop(t *testing.T, res *ChatResult, msgs []Message, err error) { + t.Helper() + if err != nil { + t.Fatalf("RunTools: %v", err) + } + if res == nil { + t.Fatal("RunTools returned a nil result") + } + if len(msgs) == 0 { + t.Fatal("RunTools returned an empty transcript") + } +} + +func echoHandler(_ context.Context, c ToolCall) (string, error) { return "ok-" + c.Name, nil } + +// idOK compares a tool-call id; Gemini has no call ids, the SDK synthesises +// them, so only the other formats pin the scripted id. +func (f toolFormat) idOK(got, want string) bool { return f.format == FormatGemini || got == want } + +func TestRunTools_TwoTurnPerFormat(t *testing.T) { + for _, f := range toolFormats { + t.Run(f.name, func(t *testing.T) { + srv, sc := newToolServer(t, f.call([2]string{"c1", "a"}, [2]string{"c2", "b"}), f.stop) + cc := f.client(t, srv) + req := f.req() + orig := append([]Message(nil), req.Messages...) + var turns []int + res, msgs, err := RunTools(context.Background(), cc, req, + map[string]ToolHandler{"a": echoHandler, "b": echoHandler}, + ToolLoopOptions{OnTurn: func(n int, r *ChatResult) { + if r == nil { + t.Error("OnTurn got nil result") + } + turns = append(turns, n) + }}) + mustLoop(t, res, msgs, err) + if res.Content != "done" || res.FinishReason != FinishStop { + t.Errorf("final = %q / %q", res.Content, res.FinishReason) + } + if sc.count() != 2 { + t.Fatalf("requests = %d, want 2", sc.count()) + } + f.replay(t, sc.body(t, 1)) + if !reflect.DeepEqual(turns, []int{1, 2}) { + t.Errorf("OnTurn turns = %v, want [1 2]", turns) + } + // transcript = original + [assistant, tool, tool, assistant] + if len(msgs) != 5 { + t.Fatalf("transcript len = %d: %+v", len(msgs), msgs) + } + if !reflect.DeepEqual(msgs[0], orig[0]) { + t.Errorf("transcript[0] = %+v", msgs[0]) + } + if msgs[1].Role != RoleAssistant || len(msgs[1].ToolCalls) != 2 { + t.Errorf("transcript[1] = %+v", msgs[1]) + } + for i, id := range []string{"c1", "c2"} { + m := msgs[2+i] + if m.Role != RoleTool || !f.idOK(m.ToolCallID, id) || m.Content != "ok-"+m.ToolName || m.IsError { + t.Errorf("transcript[%d] = %+v", 2+i, m) + } + } + if msgs[2].ToolName != "a" || msgs[3].ToolName != "b" { + t.Errorf("tool names = %q %q", msgs[2].ToolName, msgs[3].ToolName) + } + if msgs[4].Role != RoleAssistant || msgs[4].Content != "done" { + t.Errorf("transcript[4] = %+v", msgs[4]) + } + // request value and caller slice are never mutated + if !reflect.DeepEqual(req.Messages, orig) || len(req.Messages) != 1 { + t.Errorf("caller messages mutated: %+v", req.Messages) + } + }) + } +} + +func TestRunTools_AnthropicReplaysBlocksFromTranscript(t *testing.T) { + f := toolFormats[2] + srv, _ := newToolServer(t, f.call([2]string{"c1", "a"}), f.stop) + res, msgs, err := RunTools(context.Background(), f.client(t, srv), f.req(), + map[string]ToolHandler{"a": echoHandler}, ToolLoopOptions{}) + mustLoop(t, res, msgs, err) + want := []ThinkingBlock{{Text: "plan A", Signature: "SIG-A"}, {Redacted: "ENC-B"}} + if !reflect.DeepEqual(msgs[1].ThinkingBlocks, want) { + t.Errorf("transcript thinking blocks = %+v, want %+v", msgs[1].ThinkingBlocks, want) + } +} + +func TestRunTools_HandlerErrorAndUnknownTool(t *testing.T) { + for _, f := range toolFormats { + t.Run(f.name, func(t *testing.T) { + srv, sc := newToolServer(t, f.call([2]string{"c1", "a"}, [2]string{"c2", "nope"}, [2]string{"c3", "b"}), f.stop) + handlers := map[string]ToolHandler{ + "a": func(context.Context, ToolCall) (string, error) { return "partial", errors.New("boom") }, + "b": nil, // nil handler counts as unknown + } + res, msgs, err := RunTools(context.Background(), f.client(t, srv), f.req(), handlers, ToolLoopOptions{}) + mustLoop(t, res, msgs, err) + if sc.count() != 2 { + t.Fatalf("loop must continue after tool errors: requests = %d", sc.count()) + } + if len(msgs) != 6 { + t.Fatalf("transcript len = %d: %+v", len(msgs), msgs) + } + if m := msgs[2]; !m.IsError || m.Content != "boom" || !f.idOK(m.ToolCallID, "c1") { + t.Errorf("handler error result = %+v", m) + } + for i, id := range []string{"c2", "c3"} { + if m := msgs[3+i]; !m.IsError || m.Content != "unknown tool" || !f.idOK(m.ToolCallID, id) { + t.Errorf("unknown tool result %d = %+v", i, m) + } + } + }) + } +} + +func TestRunTools_HandlerPanicBecomesErrorResult(t *testing.T) { + f := toolFormats[0] + srv, _ := newToolServer(t, f.call([2]string{"c1", "a"}), f.stop) + res, msgs, err := RunTools(context.Background(), f.client(t, srv), f.req(), + map[string]ToolHandler{"a": func(context.Context, ToolCall) (string, error) { panic("kaboom") }}, ToolLoopOptions{}) + mustLoop(t, res, msgs, err) + if m := msgs[2]; !m.IsError || !strings.Contains(m.Content, "tool a panicked") || !strings.Contains(m.Content, "kaboom") { + t.Errorf("panic result = %+v", m) + } +} + +func TestRunTools_ParallelRunsConcurrentlyInCallOrder(t *testing.T) { + for _, f := range toolFormats { + t.Run(f.name, func(t *testing.T) { + srv, _ := newToolServer(t, f.call([2]string{"c1", "a"}, [2]string{"c2", "b"}), f.stop) + var started sync.WaitGroup + started.Add(2) + release := make(chan struct{}) + go func() { + started.Wait() + close(release) + }() + h := func(_ context.Context, c ToolCall) (string, error) { + started.Done() + select { + case <-release: // both handlers started before either returns + case <-time.After(3 * time.Second): + return "", errors.New("handlers did not overlap") + } + if c.Name == "a" { + time.Sleep(20 * time.Millisecond) // finish out of call order + } + return "r-" + c.Name, nil + } + res, msgs, err := RunTools(context.Background(), f.client(t, srv), f.req(), + map[string]ToolHandler{"a": h, "b": h}, ToolLoopOptions{Parallel: true}) + mustLoop(t, res, msgs, err) + if len(msgs) != 5 { + t.Fatalf("transcript len = %d", len(msgs)) + } + for i, id := range []string{"c1", "c2"} { + m := msgs[2+i] + if !f.idOK(m.ToolCallID, id) || m.Content != "r-"+m.ToolName || m.IsError { + t.Errorf("result %d = %+v", i, m) + } + } + }) + } +} + +func TestRunTools_SequentialByDefault(t *testing.T) { + f := toolFormats[0] + srv, _ := newToolServer(t, f.call([2]string{"c1", "a"}, [2]string{"c2", "b"}), f.stop) + var cur, peak int32 + h := func(_ context.Context, c ToolCall) (string, error) { + n := atomic.AddInt32(&cur, 1) + for { + p := atomic.LoadInt32(&peak) + if n <= p || atomic.CompareAndSwapInt32(&peak, p, n) { + break + } + } + time.Sleep(10 * time.Millisecond) + atomic.AddInt32(&cur, -1) + return "x", nil + } + res, msgs, err := RunTools(context.Background(), f.client(t, srv), f.req(), + map[string]ToolHandler{"a": h, "b": h}, ToolLoopOptions{}) + mustLoop(t, res, msgs, err) + if peak != 1 { + t.Errorf("peak concurrency = %d, want 1", peak) + } +} + +func TestRunTools_NonToolFinishEndsLoop(t *testing.T) { + for _, f := range toolFormats { + for _, fin := range []struct{ name, body, want string }{ + {"stop", f.stop, FinishStop}, {"length", f.length, FinishLength}, + } { + t.Run(f.name+"/"+fin.name, func(t *testing.T) { + srv, sc := newToolServer(t, fin.body) + var calls int32 + h := func(context.Context, ToolCall) (string, error) { atomic.AddInt32(&calls, 1); return "x", nil } + var turns int + res, msgs, err := RunTools(context.Background(), f.client(t, srv), f.req(), + map[string]ToolHandler{"a": h, "b": h}, ToolLoopOptions{OnTurn: func(int, *ChatResult) { turns++ }}) + mustLoop(t, res, msgs, err) + if res.FinishReason != fin.want { + t.Errorf("finish = %q, want %q", res.FinishReason, fin.want) + } + if sc.count() != 1 || turns != 1 || atomic.LoadInt32(&calls) != 0 { + t.Errorf("requests=%d turns=%d handler calls=%d, want 1/1/0", sc.count(), turns, calls) + } + if len(msgs) != 2 || msgs[1].Role != RoleAssistant || msgs[1].Content != res.Content { + t.Errorf("transcript = %+v", msgs) + } + }) + } + } +} + +func TestRunTools_MaxTurnsExceeded(t *testing.T) { + for _, f := range toolFormats { + t.Run(f.name, func(t *testing.T) { + srv, sc := newToolServer(t, f.call([2]string{"c1", "a"}), f.call([2]string{"c2", "a"}), f.stop) + var turns int + res, msgs, err := RunTools(context.Background(), f.client(t, srv), f.req(), + map[string]ToolHandler{"a": echoHandler}, + ToolLoopOptions{MaxTurns: 2, OnTurn: func(int, *ChatResult) { turns++ }}) + if !errors.Is(err, ErrMaxToolTurns) { + t.Fatalf("err = %v, want ErrMaxToolTurns", err) + } + if res == nil || len(res.ToolCalls) != 1 || !f.idOK(res.ToolCalls[0].ID, "c2") { + t.Fatalf("last result = %+v", res) + } + if sc.count() != 2 || turns != 2 { + t.Errorf("requests=%d turns=%d, want 2/2", sc.count(), turns) + } + // original + [assistant c1, tool c1]; the final tool-call assistant turn is dropped. + if len(msgs) != 3 { + t.Fatalf("transcript len = %d: %+v", len(msgs), msgs) + } + if last := msgs[len(msgs)-1]; last.Role != RoleTool || !f.idOK(last.ToolCallID, "c1") { + t.Errorf("last message = %+v, want tool result c1 (no orphan assistant)", last) + } + }) + } +} + +func TestRunTools_DefaultMaxTurnsIs8(t *testing.T) { + f := toolFormats[0] + var resp []string + for i := 0; i < 12; i++ { + resp = append(resp, f.call([2]string{fmt.Sprintf("c%d", i), "a"})) + } + srv, sc := newToolServer(t, resp...) + _, _, err := RunTools(context.Background(), f.client(t, srv), f.req(), + map[string]ToolHandler{"a": echoHandler}, ToolLoopOptions{}) + if !errors.Is(err, ErrMaxToolTurns) { + t.Fatalf("err = %v, want ErrMaxToolTurns", err) + } + if sc.count() != 8 { + t.Errorf("requests = %d, want 8", sc.count()) + } +} + +func TestRunTools_ChatErrorReturnsTranscript(t *testing.T) { + f := toolFormats[0] + srv, _ := newToolServer(t, f.call([2]string{"c1", "a"})) // second request: 400 + res, msgs, err := RunTools(context.Background(), f.client(t, srv), f.req(), + map[string]ToolHandler{"a": echoHandler}, ToolLoopOptions{}) + if err == nil { + t.Fatal("want chat error") + } + if errors.Is(err, ErrMaxToolTurns) { + t.Errorf("chat error must pass through, got %v", err) + } + if res == nil || len(res.ToolCalls) != 1 { + t.Errorf("last successful result = %+v", res) + } + if len(msgs) != 3 { + t.Errorf("transcript len = %d: %+v", len(msgs), msgs) + } +} + +func TestRunTools_NilClient(t *testing.T) { + _, _, err := RunTools(context.Background(), nil, ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}}, nil, ToolLoopOptions{}) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Fatalf("err = %T %v, want *ConfigError", err, err) + } +} + +func TestRunTools_ContextCancel(t *testing.T) { + f := toolFormats[0] + srv, sc := newToolServer(t, f.call([2]string{"c1", "a"}), f.stop) + ctx, cancel := context.WithCancel(context.Background()) + h := func(context.Context, ToolCall) (string, error) { cancel(); return "x", nil } + res, msgs, err := RunTools(ctx, f.client(t, srv), f.req(), map[string]ToolHandler{"a": h}, ToolLoopOptions{}) + if !errors.Is(err, context.Canceled) { + t.Fatalf("err = %v, want context.Canceled", err) + } + if res == nil || len(msgs) == 0 { + t.Errorf("want last result and transcript, got %v / %d", res, len(msgs)) + } + if sc.count() != 1 { + t.Errorf("requests = %d, want 1 (no chat after cancel)", sc.count()) + } +} + +func TestRunTools_ContextCancelledBeforeStart(t *testing.T) { + f := toolFormats[0] + srv, sc := newToolServer(t, f.stop) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + _, _, err := RunTools(ctx, f.client(t, srv), f.req(), nil, ToolLoopOptions{}) + if !errors.Is(err, context.Canceled) { + t.Fatalf("err = %v, want context.Canceled", err) + } + if sc.count() != 0 { + t.Errorf("requests = %d, want 0", sc.count()) + } +} diff --git a/m6b_topk_test.go b/m6b_topk_test.go new file mode 100644 index 0000000..aff690d --- /dev/null +++ b/m6b_topk_test.go @@ -0,0 +1,297 @@ +package llm + +import ( + "context" + "encoding/json" + "errors" + "net/http/httptest" + "testing" +) + +// m6b SL04: TopK, Gemini generationConfig deep-merge, embedding TaskType (AC-5). + +const ( + tkOpenAIOK = `{"choices":[{"message":{"role":"assistant","content":"ok"},"finish_reason":"stop"}]}` + tkRespOK = `{"id":"r1","status":"completed","output":[{"type":"message","content":[{"type":"output_text","text":"ok"}]}]}` + tkAnthOK = `{"id":"m1","type":"message","role":"assistant","content":[{"type":"text","text":"ok"}],"stop_reason":"end_turn","usage":{"input_tokens":1,"output_tokens":1}}` + tkGemOK = `{"candidates":[{"content":{"parts":[{"text":"ok"}]},"finishReason":"STOP"}]}` +) + +func tkClient(t *testing.T, id string, f Format, model, body string) (*ChatClient, *countRec, *httptest.Server) { + t.Helper() + srv, rec := countServer(t, obsStep{status: 200, body: body}) + cc := newTestClient(t, ProviderConfig{ID: id, Format: f, BaseURL: srv.URL, APIKey: "k"}, srv) + cc.model = model + return cc, rec, srv +} + +func tkReq() *ChatRequest { + return &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "hi"}}, MaxTokens: 100} +} + +func tkSent(t *testing.T, rec *countRec) map[string]any { + t.Helper() + if rec.n() != 1 { + t.Fatalf("requests = %d, want 1", rec.n()) + } + return rec.bodies[0] +} + +func tkGenCfg(t *testing.T, b map[string]any) map[string]any { + t.Helper() + g, ok := b["generationConfig"].(map[string]any) + if !ok { + t.Fatalf("generationConfig missing or not an object: %v", b) + } + return g +} + +func TestTopK_PerFormatWire(t *testing.T) { + t.Run("anthropic", func(t *testing.T) { + cc, rec, _ := tkClient(t, "anthropic", FormatAnthropic, "claude-x", tkAnthOK) + req := tkReq() + req.TopK = 40 + if _, err := cc.Call(context.Background(), req); err != nil { + t.Fatal(err) + } + if got := tkSent(t, rec)["top_k"]; got != float64(40) { + t.Errorf("top_k = %v, want 40", got) + } + }) + t.Run("anthropic thinking omits", func(t *testing.T) { + cc, rec, _ := tkClient(t, "anthropic", FormatAnthropic, "claude-x", tkAnthOK) + req := tkReq() + req.TopK = 40 + req.Thinking = "enabled" + req.ThinkingBudget = 2048 + req.MaxTokens = 8192 + if _, err := cc.Call(context.Background(), req); err != nil { + t.Fatal(err) + } + b := tkSent(t, rec) + if _, ok := b["thinking"]; !ok { + t.Fatalf("thinking not enabled in body: %v", b) + } + if v, ok := b["top_k"]; ok { + t.Errorf("top_k = %v with thinking on, want absent", v) + } + }) + t.Run("gemini", func(t *testing.T) { + cc, rec, _ := tkClient(t, "gemini", FormatGemini, "gemini-x", tkGemOK) + req := tkReq() + req.TopK = 40 + if _, err := cc.Call(context.Background(), req); err != nil { + t.Fatal(err) + } + if got := tkGenCfg(t, tkSent(t, rec))["topK"]; got != float64(40) { + t.Errorf("generationConfig.topK = %v, want 40", got) + } + }) + t.Run("gemini without TopK emits none", func(t *testing.T) { + cc, rec, _ := tkClient(t, "gemini", FormatGemini, "gemini-x", tkGemOK) + if _, err := cc.Call(context.Background(), tkReq()); err != nil { + t.Fatal(err) + } + if _, ok := tkGenCfg(t, tkSent(t, rec))["topK"]; ok { + t.Error("topK present without TopK") + } + }) + t.Run("openai chat ignores", func(t *testing.T) { + cc, rec, _ := tkClient(t, "openai", FormatOpenAI, "gpt-4o", tkOpenAIOK) + req := tkReq() + req.TopK = 40 + if _, err := cc.Call(context.Background(), req); err != nil { + t.Fatal(err) + } + tkNoTopK(t, tkSent(t, rec)) + }) + t.Run("responses ignores", func(t *testing.T) { + cc, rec, _ := tkClient(t, "openai", FormatOpenAI, "gpt-5.6", tkRespOK) + req := tkReq() + req.TopK = 40 + req.Tools = []ToolDef{{Name: "f", Description: "d", Parameters: json.RawMessage(`{"type":"object"}`)}} + if _, err := cc.Call(context.Background(), req); err != nil { + t.Fatal(err) + } + if rec.paths[0] != "/responses" && rec.paths[0] != "/v1/responses" { + t.Fatalf("path = %q, want a responses endpoint", rec.paths[0]) + } + tkNoTopK(t, tkSent(t, rec)) + }) +} + +func tkNoTopK(t *testing.T, b map[string]any) { + t.Helper() + for _, k := range []string{"top_k", "topK"} { + if v, ok := b[k]; ok { + t.Errorf("body has %s = %v", k, v) + } + } +} + +func TestTopK_NegativeIsConfigError(t *testing.T) { + for _, f := range []struct { + id string + format Format + model string + body string + }{ + {"openai", FormatOpenAI, "gpt-4o", tkOpenAIOK}, + {"anthropic", FormatAnthropic, "claude-x", tkAnthOK}, + {"gemini", FormatGemini, "gemini-x", tkGemOK}, + } { + t.Run(f.id, func(t *testing.T) { + cc, rec, _ := tkClient(t, f.id, f.format, f.model, f.body) + req := tkReq() + req.TopK = -1 + _, err := cc.Call(context.Background(), req) + var ce *ConfigError + if !errors.As(err, &ce) { + t.Fatalf("err = %v, want *ConfigError", err) + } + if rec.n() != 0 { + t.Errorf("requests = %d, want 0", rec.n()) + } + }) + } +} + +func TestGeminiExtraGenerationConfigDeepMerge(t *testing.T) { + t.Run("extra keys merged, sdk keys kept", func(t *testing.T) { + cc, rec, _ := tkClient(t, "gemini", FormatGemini, "gemini-x", tkGemOK) + req := tkReq() + req.Temperature = 0.5 + req.Extra = map[string]any{"generationConfig": map[string]any{"topK": 5, "candidateCount": 2}} + if _, err := cc.Call(context.Background(), req); err != nil { + t.Fatal(err) + } + g := tkGenCfg(t, tkSent(t, rec)) + want := map[string]any{"maxOutputTokens": float64(100), "temperature": 0.5, "topK": float64(5), "candidateCount": float64(2)} + for k, v := range want { + if g[k] != v { + t.Errorf("generationConfig[%s] = %v, want %v (got %v)", k, g[k], v, g) + } + } + }) + t.Run("extra overrides sdk key", func(t *testing.T) { + cc, rec, _ := tkClient(t, "gemini", FormatGemini, "gemini-x", tkGemOK) + req := tkReq() + req.TopK = 40 + req.Extra = map[string]any{"generationConfig": map[string]any{"maxOutputTokens": 7, "topK": 5}} + if _, err := cc.Call(context.Background(), req); err != nil { + t.Fatal(err) + } + g := tkGenCfg(t, tkSent(t, rec)) + if g["maxOutputTokens"] != float64(7) || g["topK"] != float64(5) { + t.Errorf("generationConfig = %v, want Extra to win", g) + } + }) + t.Run("no sdk generationConfig uses extra as is", func(t *testing.T) { + cc, rec, _ := tkClient(t, "gemini", FormatGemini, "gemini-x", tkGemOK) + req := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "hi"}}, + Extra: map[string]any{"generationConfig": map[string]any{"candidateCount": 3}}} + if _, err := cc.Call(context.Background(), req); err != nil { + t.Fatal(err) + } + g := tkGenCfg(t, tkSent(t, rec)) + if len(g) != 1 || g["candidateCount"] != float64(3) { + t.Errorf("generationConfig = %v", g) + } + }) + t.Run("nested objects replaced whole", func(t *testing.T) { + cc, rec, _ := tkClient(t, "gemini", FormatGemini, "gemini-x", tkGemOK) + req := tkReq() + req.Thinking = "high" + req.Extra = map[string]any{"generationConfig": map[string]any{"thinkingConfig": map[string]any{"includeThoughts": false}}} + if _, err := cc.Call(context.Background(), req); err != nil { + t.Fatal(err) + } + g := tkGenCfg(t, tkSent(t, rec)) + tc, _ := g["thinkingConfig"].(map[string]any) + if len(tc) != 1 || tc["includeThoughts"] != false { + t.Errorf("thinkingConfig = %v, want exactly the Extra object", tc) + } + if g["maxOutputTokens"] != float64(100) { + t.Errorf("maxOutputTokens dropped: %v", g) + } + }) + t.Run("other extra keys stay shallow", func(t *testing.T) { + cc, rec, _ := tkClient(t, "gemini", FormatGemini, "gemini-x", tkGemOK) + req := tkReq() + req.Extra = map[string]any{"safetySettings": []any{map[string]any{"category": "X"}}} + if _, err := cc.Call(context.Background(), req); err != nil { + t.Fatal(err) + } + b := tkSent(t, rec) + if _, ok := b["safetySettings"]; !ok { + t.Errorf("safetySettings missing: %v", b) + } + if tkGenCfg(t, b)["maxOutputTokens"] != float64(100) { + t.Errorf("generationConfig changed: %v", b["generationConfig"]) + } + }) + t.Run("non-object extra replaces", func(t *testing.T) { + cc, rec, _ := tkClient(t, "gemini", FormatGemini, "gemini-x", tkGemOK) + req := tkReq() + req.Extra = map[string]any{"generationConfig": "raw"} + if _, err := cc.Call(context.Background(), req); err != nil { + t.Fatal(err) + } + if got := tkSent(t, rec)["generationConfig"]; got != "raw" { + t.Errorf("generationConfig = %v, want raw replacement", got) + } + }) + t.Run("anthropic stays shallow", func(t *testing.T) { + cc, rec, _ := tkClient(t, "anthropic", FormatAnthropic, "claude-x", tkAnthOK) + req := tkReq() + req.Extra = map[string]any{"metadata": map[string]any{"user_id": "u"}} + if _, err := cc.Call(context.Background(), req); err != nil { + t.Fatal(err) + } + if m, _ := tkSent(t, rec)["metadata"].(map[string]any); m["user_id"] != "u" { + t.Errorf("metadata = %v", m) + } + }) +} + +func TestEmbedTaskType(t *testing.T) { + embedOpenAI := `{"data":[{"index":0,"embedding":[0.5,1]},{"index":1,"embedding":[1,2]}],"usage":{"prompt_tokens":2}}` + embedGemini := `{"embeddings":[{"values":[0.5,1]},{"values":[1,2]}]}` + run := func(t *testing.T, f Format, id, body, taskType string) map[string]any { + t.Helper() + srv, rec := countServer(t, obsStep{status: 200, body: body}) + s := newTestSDK(t, ProviderConfig{ID: id, Format: f, APIKey: "k"}, srv) + if _, err := s.Embed(context.Background(), id, "m", EmbedRequest{Inputs: []string{"a", "b"}, TaskType: taskType}); err != nil { + t.Fatal(err) + } + return tkSent(t, rec) + } + t.Run("gemini every item", func(t *testing.T) { + b := run(t, FormatGemini, "gemini", embedGemini, " RETRIEVAL_QUERY ") + reqs, _ := b["requests"].([]any) + if len(reqs) != 2 { + t.Fatalf("requests = %v", b) + } + for i, r := range reqs { + if got := r.(map[string]any)["taskType"]; got != "RETRIEVAL_QUERY" { + t.Errorf("item %d taskType = %v", i, got) + } + } + }) + t.Run("gemini unset omits", func(t *testing.T) { + b := run(t, FormatGemini, "gemini", embedGemini, "") + for i, r := range b["requests"].([]any) { + if _, ok := r.(map[string]any)["taskType"]; ok { + t.Errorf("item %d has taskType", i) + } + } + }) + t.Run("openai ignores", func(t *testing.T) { + b := run(t, FormatOpenAI, "openai", embedOpenAI, "RETRIEVAL_QUERY") + for _, k := range []string{"task_type", "taskType"} { + if _, ok := b[k]; ok { + t.Errorf("body has %s: %v", k, b) + } + } + }) +} diff --git a/message.go b/message.go index 37ad2c0..edbfff7 100644 --- a/message.go +++ b/message.go @@ -28,6 +28,10 @@ type ContentPartType string const ( ContentPartText ContentPartType = "text" ContentPartImage ContentPartType = "image" + // ContentPartDocument is an inline document (PDF or plain text). + ContentPartDocument ContentPartType = "document" + // ContentPartImageURL is an image the provider fetches by URL. + ContentPartImageURL ContentPartType = "image_url" ) // ContentPart is one ordered text or inline-image segment in a message. @@ -39,16 +43,41 @@ type ContentPart struct { Text string Image []byte MIMEType string + // Data is the document payload (ContentPartDocument). + Data []byte + // Name is the optional document filename. + Name string + // URL is the remote image location (ContentPartImageURL). + URL string } const MaxImageBytes = 10 << 20 -// TextPart creates an ordered text content part. -func TextPart(text string) ContentPart { return ContentPart{Type: ContentPartText, Text: text} } +// MaxDocumentBytes bounds one inline document part. +const MaxDocumentBytes = 20 << 20 -// ImagePart creates an ordered inline-image content part. -func ImagePart(mimeType string, data []byte) ContentPart { - return ContentPart{Type: ContentPartImage, Image: data, MIMEType: mimeType} +// DocumentPart creates an inline document part (application/pdf or +// text/plain). Name is an optional filename. +func DocumentPart(mimeType string, data []byte, name string) ContentPart { + return ContentPart{Type: ContentPartDocument, MIMEType: mimeType, Data: data, Name: name} +} + +// ImageURLPart creates a part referencing a remote http(s) image. The SDK +// never fetches it; the provider does. Gemini needs a MIME type: use +// ImageURLPartMIME or WithMIME. +func ImageURLPart(url string) ContentPart { + return ContentPart{Type: ContentPartImageURL, URL: url} +} + +// ImageURLPartMIME creates a remote-image part with a MIME type hint. +func ImageURLPartMIME(url, mime string) ContentPart { + return ContentPart{Type: ContentPartImageURL, URL: url, MIMEType: mime} +} + +// WithMIME returns a copy of p with MIMEType set. +func (p ContentPart) WithMIME(mime string) ContentPart { + p.MIMEType = mime + return p } const ( @@ -231,6 +260,8 @@ type ResponseFormat struct { // OpenAI Responses API has none, so it is not sent on requests diverted // there). type ChatRequest struct { + // Headers are per-request HTTP headers (stub; behaviour lands with SL03). + Headers map[string]string Model string Messages []Message System []SystemBlock @@ -240,7 +271,12 @@ type ChatRequest struct { MaxTokens int Temperature float64 TopP float64 - Stop []string + // TopK limits sampling to the K most likely tokens (0 = provider + // default; negative is a ConfigError). Anthropic top_k (omitted when + // thinking is on) and Gemini generationConfig.topK; OpenAI chat and + // Responses ignore it. + TopK int + Stop []string // ToolChoice constrains tool use (nil = provider default). It requires // Tools. ToolChoice *ToolChoice @@ -256,8 +292,9 @@ type ChatRequest struct { // Extra holds provider-specific top-level body fields merged into the // request JSON after the SDK's own (so they override them): an escape // hatch for parameters the canonical shape does not model yet. The merge - // is shallow: an object value replaces the SDK's whole object (e.g. - // Gemini generationConfig). Keys go to whichever endpoint the request + // is shallow: an object value replaces the SDK's whole object, except + // Gemini generationConfig, whose keys are set one by one over the SDK's + // (nested objects such as thinkingConfig are replaced whole). Keys go to whichever endpoint the request // is sent to, including /responses when an OpenAI request is diverted // there. Reserved (ConfigError): "stream" (use Call vs CallStream) and // the keys the SDK validates — model, messages/contents/input, @@ -279,6 +316,11 @@ type ChatResult struct { ToolCalls []ToolCall FinishReason string Usage Usage + // Refusal is the provider's refusal text when it signals one; empty + // otherwise. Content is never altered. + Refusal string + // Meta is provider-reported response metadata (zero when unreported). + Meta ResponseMeta } // DeltaKind discriminates streamed fragments. @@ -292,6 +334,8 @@ const ( // DeltaToolArgs is a tool-call argument fragment (partial JSON). ToolIndex // and, on the first fragment of a call, ToolID/ToolName identify the call. DeltaToolArgs + // DeltaRefusal is a refusal-text fragment carried in Delta.Text. + DeltaRefusal ) // Delta is one streamed fragment. Text is the fragment for this event, not @@ -335,3 +379,11 @@ func (r *ChatResult) AssistantMessage() Message { } return m } + +// TextPart creates an ordered text content part. +func TextPart(text string) ContentPart { return ContentPart{Type: ContentPartText, Text: text} } + +// ImagePart creates an ordered inline-image content part. +func ImagePart(mimeType string, data []byte) ContentPart { + return ContentPart{Type: ContentPartImage, Image: data, MIMEType: mimeType} +} diff --git a/meta.go b/meta.go new file mode 100644 index 0000000..4830b11 --- /dev/null +++ b/meta.go @@ -0,0 +1,173 @@ +package llm + +import ( + "context" + "fmt" + "net/http" + "sort" + "strconv" + "strings" + "time" +) + +// ResponseMeta is provider-reported response metadata. Every field is +// provider-reported or zero — never guessed. +type ResponseMeta struct { + ID string // provider response/message id; "" when not reported + Model string // model reported by the provider; "" when not reported + RequestID string // from the x-request-id family of response headers + RateLimit RateLimit +} + +// RateLimit is the provider-reported rate-limit headroom. +type RateLimit struct { + Reported bool // true when at least one field below was parsed + RemainingRequests int + RemainingTokens int + ResetRequests time.Duration // time until the window resets; 0 = unknown + ResetTokens time.Duration +} + +var requestIDHeaders = []string{"X-Request-Id", "Request-Id", "X-Goog-Request-Id", "X-Amzn-Requestid"} + +// parseResponseMeta extracts RequestID and RateLimit from response headers +// (ID and Model come from the body). +func parseResponseMeta(h http.Header) ResponseMeta { + var m ResponseMeta + for _, k := range requestIDHeaders { + if v := strings.TrimSpace(h.Get(k)); v != "" { + m.RequestID = v + break + } + } + rl := &m.RateLimit + rl.RemainingRequests = headerInt(h, rl, "X-Ratelimit-Remaining-Requests", "Anthropic-Ratelimit-Requests-Remaining") + rl.RemainingTokens = headerInt(h, rl, "X-Ratelimit-Remaining-Tokens", "Anthropic-Ratelimit-Tokens-Remaining") + rl.ResetRequests = headerReset(h, rl, "X-Ratelimit-Reset-Requests", "Anthropic-Ratelimit-Requests-Reset") + rl.ResetTokens = headerReset(h, rl, "X-Ratelimit-Reset-Tokens", "Anthropic-Ratelimit-Tokens-Reset") + return m +} + +// headerInt parses the first parseable of the named headers. +func headerInt(h http.Header, rl *RateLimit, names ...string) int { + for _, n := range names { + if v, err := strconv.Atoi(strings.TrimSpace(h.Get(n))); err == nil && v >= 0 { + rl.Reported = true + return v + } + } + return 0 +} + +// headerReset parses a Go-style duration, plain seconds, or an RFC3339 +// timestamp (duration from now, floored at zero). +func headerReset(h http.Header, rl *RateLimit, names ...string) time.Duration { + for _, n := range names { + v := strings.TrimSpace(h.Get(n)) + if v == "" { + continue + } + if d, err := time.ParseDuration(v); err == nil && d >= 0 { + rl.Reported = true + return d + } + if f, err := strconv.ParseFloat(v, 64); err == nil && f >= 0 && f < 1e9 { + rl.Reported = true + return time.Duration(f * float64(time.Second)) + } + if t, err := time.Parse(time.RFC3339, v); err == nil { + rl.Reported = true + return max(time.Until(t), 0) + } + } + return 0 +} + +// withHeaderMeta overlays header-derived metadata onto r (nil-safe). +func (r *ChatResult) withHeaderMeta(h http.Header) { + if r == nil { + return + } + hm := parseResponseMeta(h) + r.Meta.RequestID, r.Meta.RateLimit = hm.RequestID, hm.RateLimit +} + +// reservedHeaders are never settable per request (authentication and framing). +var reservedHeaders = map[string]bool{ + "Authorization": true, "Proxy-Authorization": true, "X-Api-Key": true, + "X-Goog-Api-Key": true, "Host": true, "Content-Length": true, + "Content-Type": true, "Transfer-Encoding": true, +} + +type reqHeadersKey struct{} + +// validateReqHeaders checks per-request headers before any network I/O. +// Errors name a reserved header but never echo values or malformed names. +func validateReqHeaders(h map[string]string) error { + names := make([]string, 0, len(h)) + for k := range h { + names = append(names, k) + } + sort.Strings(names) + seen := make(map[string]bool, len(names)) + for _, k := range names { + if !validHeaderToken(k) { + return &ConfigError{Msg: "Headers: invalid header name"} + } + if reservedHeaders[http.CanonicalHeaderKey(k)] { + return &ConfigError{Msg: fmt.Sprintf("Headers: %q is a reserved header", http.CanonicalHeaderKey(k))} + } + ck := http.CanonicalHeaderKey(k) + if seen[ck] { + return &ConfigError{Msg: fmt.Sprintf("Headers: duplicate header %q (names are case-insensitive)", ck)} + } + seen[ck] = true + if badHeaderValue(h[k]) { + return &ConfigError{Msg: fmt.Sprintf("Headers: invalid value for header %q", ck)} + } + } + return nil +} + +// badHeaderValue reports any ASCII control byte other than HTAB (CR, LF, +// NUL, DEL included). +func badHeaderValue(v string) bool { + for i := 0; i < len(v); i++ { + if c := v[i]; (c < 0x20 && c != '\t') || c == 0x7f { + return true + } + } + return false +} + +func validHeaderToken(s string) bool { + if s == "" { + return false + } + for i := 0; i < len(s); i++ { + c := s[i] + switch { + case c >= 'a' && c <= 'z', c >= 'A' && c <= 'Z', c >= '0' && c <= '9': + case strings.IndexByte("!#$%&'*+-.^_`|~", c) >= 0: + default: + return false + } + } + return true +} + +// withReqHeaders validates h and attaches it to ctx for this call only. +func withReqHeaders(ctx context.Context, h map[string]string) (context.Context, error) { + if len(h) == 0 { + return ctx, nil + } + if err := validateReqHeaders(h); err != nil { + return ctx, err + } + return context.WithValue(ctx, reqHeadersKey{}, h), nil +} + +func reqHeadersFrom(ctx context.Context) map[string]string { + h, _ := ctx.Value(reqHeadersKey{}).(map[string]string) + return h +} diff --git a/models.go b/models.go index 170d044..51e67bc 100644 --- a/models.go +++ b/models.go @@ -1,6 +1,7 @@ package llm import ( + "bytes" "context" "encoding/json" "errors" @@ -15,21 +16,26 @@ import ( // back to static model tables. Fields the provider does not report stay // zero (unknown). -// listModels runs discovery for one provider with light retries (model -// listings are less critical than chat: 3 attempts). +// defaultListAttempts is the attempt budget when no RetryPolicy is +// configured (model listings are less critical than chat). +const defaultListAttempts = 3 + +// listModels runs discovery for one provider through the shared retry +// ladder. WithRetryPolicy sets the attempt count and backoff cap (3 attempts +// when unconfigured); the request timeout is a whole-call budget; Retry-After +// is honored. func (pc *providerClient) listModels(ctx context.Context) ([]Model, error) { - ctx, cancel := context.WithTimeout(ctx, 30*time.Second) + ctx = pc.withAttempts(ctx, "models", "") + ctx, cancel := context.WithTimeout(ctx, pc.requestTimeout()) defer cancel() - var lastErr error - for attempt := 0; attempt < 3; attempt++ { - if err := ctx.Err(); err != nil { - return nil, err - } - var ( - models []Model - err error - ) + n := pc.opts.retry.MaxAttempts + if n <= 0 { + n = defaultListAttempts + } + var models []Model + err := pc.withRetryN(ctx, n, func() (time.Duration, error) { + var err error switch pc.cfg.Format { case FormatAnthropic: models, err = listModelsAnthropic(ctx, pc) @@ -39,22 +45,49 @@ func (pc *providerClient) listModels(ctx context.Context) ([]Model, error) { models, err = listModelsOpenAI(ctx, pc) } if err == nil { - return models, nil + return 0, nil + } + var re *retryAfterError + var ra time.Duration + if errors.As(err, &re) { + ra = re.ra } - var te *terminalError - if errors.As(err, &te) { - return nil, te.err // malformed body: retrying cannot fix it + if errors.Is(err, ErrModelListTruncated) { + err = terminal(err) } + return ra, err + }, nil) + if err != nil { var apiErr *APIError - if (errors.As(err, &apiErr) && !apiErr.Retryable) || errors.Is(err, ErrModelListTruncated) { - return nil, err + var rl *RateLimitError + if (errors.As(err, &apiErr) && apiErr.Retryable) || errors.As(err, &rl) { + return nil, fmt.Errorf("llm: list models failed: %w", err) // exhausted retries } - lastErr = err - if !retrySleep(ctx, pc.retryDelay(0, attempt)) { - break + return nil, err + } + return models, nil +} + +// retryAfterError carries a response's Retry-After hint alongside its error +// out of the per-format listing functions. +type retryAfterError struct { + err error + ra time.Duration +} + +func (e *retryAfterError) Error() string { return e.err.Error() } +func (e *retryAfterError) Unwrap() error { return e.err } + +// cloneModels deep-copies a listing so callers never alias cached state. +func cloneModels(in []Model) []Model { + out := make([]Model, len(in)) + copy(out, in) + for i := range out { + if out[i].Capabilities != nil { + out[i].Capabilities = append([]string(nil), out[i].Capabilities...) } } - return nil, fmt.Errorf("llm: list models failed: %w", lastErr) + return out } // get performs a GET with auth headers and a bounded body read. @@ -64,9 +97,9 @@ func (pc *providerClient) get(ctx context.Context, url string) ([]byte, time.Dur return nil, 0, &ConfigError{Msg: "build request: " + err.Error()} } req.Header.Set("Accept", "application/json") - pc.setHeaders(req.Header) + pc.setHeaders(req.Header, nil) - resp, err := pc.buffered().Do(req) + resp, err := doObserved(ctx, pc.buffered(), req) if err != nil { return nil, 0, err } @@ -76,11 +109,11 @@ func (pc *providerClient) get(ctx context.Context, url string) ([]byte, time.Dur return nil, 0, err } if len(data) > maxModelsResponseSize { - return nil, 0, fmt.Errorf("llm: models response exceeds %d bytes", maxModelsResponseSize) + return nil, 0, terminal(fmt.Errorf("llm: models response exceeds %d bytes", maxModelsResponseSize)) } ra := parseRetryAfter(resp.Header.Get("Retry-After"), time.Now()) if resp.StatusCode < 200 || resp.StatusCode >= 300 { - return data, ra, pc.httpError(resp.StatusCode, data) + return data, ra, &retryAfterError{err: pc.httpError(resp.StatusCode, data), ra: ra} } return data, ra, nil } @@ -106,12 +139,20 @@ func listModelsOpenAI(ctx context.Context, pc *providerClient) ([]Model, error) return nil, err } var r oaModelsResponse - if err := json.Unmarshal(data, &r); err != nil { - return nil, terminal(fmt.Errorf("llm: parse models response: %w", err)) - } - entries := r.Data - if len(entries) == 0 { - entries = r.Models + var entries []oaModelEntry + if trimmed := bytes.TrimSpace(data); len(trimmed) > 0 && trimmed[0] == '[' { + // Bare top-level array (Together AI). + if err := json.Unmarshal(trimmed, &entries); err != nil { + return nil, terminal(fmt.Errorf("llm: parse models response: %w", err)) + } + } else { + if err := json.Unmarshal(data, &r); err != nil { + return nil, terminal(fmt.Errorf("llm: parse models response: %w", err)) + } + entries = r.Data + if len(entries) == 0 { + entries = r.Models + } } out := make([]Model, 0, len(entries)) for _, m := range entries { diff --git a/observe.go b/observe.go new file mode 100644 index 0000000..821e0e5 --- /dev/null +++ b/observe.go @@ -0,0 +1,120 @@ +package llm + +import ( + "context" + "net/http" + "strings" + "time" +) + +// AttemptEvent describes one HTTP attempt of an SDK call. See +// WithAttemptObserver. +type AttemptEvent struct { + Provider string + Operation string // "chat" | "stream" | "embed" | "speak" | "transcribe" | "models" | "count_tokens" + Model string // "" for models + Attempt int // 1-based, per entry-point call + Status int // HTTP status; 0 on transport error + Err error // nil on success; API keys are never included + Latency time.Duration // request start -> response headers received + RetryDelay time.Duration // delay before the next attempt; 0 when final + RequestID string // provider request id header, when reported +} + +// WithAttemptObserver registers fn to be called once per HTTP attempt on +// every entry point (chat, stream, embed, speak, transcribe, models), +// including retries and learn-once immediate retries. fn runs on the +// request goroutine: keep it fast and never call back into the SDK. fn must +// not panic, but a panic is recovered and discarded. A nil fn disables +// observation. +func WithAttemptObserver(fn func(AttemptEvent)) Option { + return func(s *SDK) { s.attemptObs = fn } +} + +type attemptKey struct{} + +// attemptRec is the per-call scratch the HTTP helpers fill and the retry +// ladders drain into AttemptEvents. It is only touched on the request +// goroutine. +type attemptRec struct { + op, model string + n int + sent bool // an HTTP attempt happened since the last emit + status int + latency time.Duration + reqID string +} + +func attemptRecFrom(ctx context.Context) *attemptRec { + r, _ := ctx.Value(attemptKey{}).(*attemptRec) + return r +} + +// withAttempts attaches an attempt record to ctx when an observer is set. +// An existing record is kept so a stream downgraded to buffered mid-call +// keeps one numbering. +func (pc *providerClient) withAttempts(ctx context.Context, op, model string) context.Context { + if pc.opts.attempt == nil || attemptRecFrom(ctx) != nil { + return ctx + } + return context.WithValue(ctx, attemptKey{}, &attemptRec{op: op, model: model}) +} + +// doObserved is client.Do that records status, latency and request id for +// the attempt observer. +func doObserved(ctx context.Context, c *http.Client, req *http.Request) (*http.Response, error) { + r := attemptRecFrom(ctx) + if r == nil { + return c.Do(req) + } + t0 := time.Now() + resp, err := c.Do(req) + r.sent = true + r.latency = max(time.Since(t0), 1) + r.status, r.reqID = 0, "" + if resp != nil { + r.status = resp.StatusCode + r.reqID = parseResponseMeta(resp.Header).RequestID + } + return resp, err +} + +// emitAttempt delivers the event for the attempt just finished (if any) +// with its final error and the delay before the next attempt. +func (pc *providerClient) emitAttempt(ctx context.Context, err error, delay time.Duration) { + r := attemptRecFrom(ctx) + if r == nil || !r.sent { + return + } + r.sent = false + r.n++ + ev := AttemptEvent{ + Provider: pc.cfg.ID, Operation: r.op, Model: r.model, Attempt: r.n, + Status: r.status, Err: err, Latency: r.latency, RetryDelay: delay, RequestID: r.reqID, + } + ev.Err = pc.redactErr(err) + defer func() { _ = recover() }() + pc.opts.attempt(ev) +} + +// redactedError carries err's text with the API key masked; the original +// stays reachable through errors.Is/As. +type redactedError struct { + msg string + err error +} + +func (e *redactedError) Error() string { return e.msg } +func (e *redactedError) Unwrap() error { return e.err } + +// redactErr masks the provider API key if a transport error echoed it. +func (pc *providerClient) redactErr(err error) error { + k := pc.cfg.APIKey + if err == nil || k == "" { + return err + } + if m := err.Error(); strings.Contains(m, k) { + return &redactedError{msg: strings.ReplaceAll(m, k, "[redacted]"), err: err} + } + return err +} diff --git a/openai.go b/openai.go index 63d08b5..4a9b8c0 100644 --- a/openai.go +++ b/openai.go @@ -60,15 +60,26 @@ type oaContentPart struct { Type string `json:"type"` Text string `json:"text,omitempty"` ImageURL *oaImageURL `json:"image_url,omitempty"` + File *oaFile `json:"file,omitempty"` +} + +type oaFile struct { + Filename string `json:"filename"` + FileData string `json:"file_data"` } // openAIContent renders ordered content parts as chat-completions parts. func openAIContent(m Message) []oaContentPart { parts := make([]oaContentPart, 0, len(m.Parts)) for _, p := range m.Parts { - if p.Type == ContentPartImage { + switch p.Type { + case ContentPartImage: parts = append(parts, oaContentPart{Type: "image_url", ImageURL: &oaImageURL{URL: "data:" + wireMIME(p.MIMEType) + ";base64," + base64.StdEncoding.EncodeToString(p.Image)}}) - } else { + case ContentPartImageURL: + parts = append(parts, oaContentPart{Type: "image_url", ImageURL: &oaImageURL{URL: p.URL}}) + case ContentPartDocument: + parts = append(parts, oaContentPart{Type: "file", File: &oaFile{Filename: documentName(p), FileData: "data:application/pdf;base64," + base64.StdEncoding.EncodeToString(p.Data)}}) + default: parts = append(parts, oaContentPart{Type: "text", Text: p.Text}) } } @@ -165,7 +176,7 @@ func buildOpenAIRequest(cfg ProviderConfig, req *ChatRequest, model string, stre c = "(image result attached below)" // some providers reject empty tool content } msgs = append(msgs, oaMessage{Role: "tool", Content: &c, ToolCallID: m.ToolCallID}) - if imgs := imageParts(m); len(imgs) > 0 { + if imgs := imageLikeParts(m); len(imgs) > 0 { toolImages = append(toolImages, TextPart("Images returned by tool call "+m.ToolCallID+":")) toolImages = append(toolImages, imgs...) } @@ -349,6 +360,7 @@ func (s *oaFlexString) UnmarshalJSON(b []byte) error { type oaRespMessage struct { Role string `json:"role"` Content string `json:"content"` + Refusal string `json:"refusal"` ReasoningContent oaFlexString `json:"reasoning_content"` Reasoning oaFlexString `json:"reasoning"` ReasoningDetails []oaReasoningDetail `json:"reasoning_details"` @@ -475,6 +487,8 @@ func usageFromOpenAI(u *oaRespUsage) Usage { } type oaResponse struct { + ID string `json:"id"` + Model string `json:"model"` Choices []oaRespChoice `json:"choices"` Usage *oaRespUsage `json:"usage"` Error *oaErrorBody `json:"error"` @@ -522,14 +536,25 @@ func parseOpenAIResponse(body []byte) (*ChatResult, error) { Content: ch.Message.Content, ReasoningContent: foldReasoning(string(ch.Message.ReasoningContent), string(ch.Message.Reasoning), ch.Message.ReasoningDetails), FinishReason: mapOpenAIFinishReason(ch.FinishReason), + Refusal: ch.Message.Refusal, + Meta: ResponseMeta{ID: r.ID, Model: r.Model}, } for _, tc := range ch.Message.ToolCalls { + args := tc.Function.Arguments + if args == "" { + args = "{}" + } res.ToolCalls = append(res.ToolCalls, ToolCall{ ID: tc.ID, Name: tc.Function.Name, - Arguments: tc.Function.Arguments, + Arguments: args, }) } + // Named tool_choice (and many gateways) report "stop" for a tool turn. + if res.FinishReason == FinishStop && len(res.ToolCalls) > 0 { + res.FinishReason = FinishToolCalls + } + res.FinishReason = refusalFinish(res.FinishReason, res.Refusal) if r.Usage != nil { res.Usage = usageFromOpenAI(r.Usage) } @@ -540,6 +565,7 @@ func parseOpenAIResponse(body []byte) (*ChatResult, error) { type oaStreamDelta struct { Content string `json:"content"` + Refusal string `json:"refusal"` ReasoningContent oaFlexString `json:"reasoning_content"` Reasoning oaFlexString `json:"reasoning"` ReasoningDetails []oaReasoningDetail `json:"reasoning_details"` @@ -561,8 +587,11 @@ type oaStreamChoice struct { } type oaStreamChunk struct { + ID string `json:"id"` + Model string `json:"model"` Choices []oaStreamChoice `json:"choices"` Usage *oaRespUsage `json:"usage"` + Error json.RawMessage `json:"error"` // see streamErrorMessage } // mapOpenAIStreamEvent folds one SSE data payload into the accumulator and @@ -572,6 +601,10 @@ func mapOpenAIStreamEvent(data []byte, acc *streamAccum) (deltas []Delta, done b if err := json.Unmarshal(data, &c); err != nil { return nil, false, fmt.Errorf("llm: parse stream chunk: %w", err) } + if msg, ok := streamErrorMessage(c.Error); ok { + return nil, false, fmt.Errorf("llm: provider error: %s", msg) + } + acc.noteMeta(c.ID, c.Model) if c.Usage != nil { acc.usage = usageFromOpenAI(c.Usage) } @@ -585,11 +618,12 @@ func mapOpenAIStreamEvent(data []byte, acc *streamAccum) (deltas []Delta, done b acc.content.WriteString(d.Content) deltas = append(deltas, Delta{Kind: DeltaContent, Text: d.Content}) } + if d.Refusal != "" { + acc.refusal.WriteString(d.Refusal) + deltas = append(deltas, Delta{Kind: DeltaRefusal, Text: d.Refusal}) + } for _, tc := range d.ToolCalls { - idx := 0 - if tc.Index != nil { - idx = *tc.Index - } + idx := acc.openAIToolKey(tc.Index, tc.ID) call := acc.call(idx) if tc.ID != "" { call.id = tc.ID @@ -603,6 +637,9 @@ func mapOpenAIStreamEvent(data []byte, acc *streamAccum) (deltas []Delta, done b } if ch.FinishReason != "" { acc.finishReason = mapOpenAIFinishReason(ch.FinishReason) + if acc.finishReason == FinishStop && len(acc.calls) > 0 { + acc.finishReason = FinishToolCalls + } acc.sawFinish = true } } diff --git a/provider.go b/provider.go index f568a2d..4d3391f 100644 --- a/provider.go +++ b/provider.go @@ -33,6 +33,11 @@ type Quirks struct { // in the thinking mode must be passed back to the API" — which poisons // every later request in the tool loop, not just the offending one. EchoReasoningWithTools bool + // AnthropicToolJSON: the Anthropic-format endpoint lacks + // output_config.format, so JSON ResponseFormat is emulated with a forced + // synthetic tool folded back into Content. Off for the built-in anthropic + // provider (native structured outputs). + AnthropicToolJSON bool // ReasoningEffort: provider accepts reasoning_effort (OpenAI, GLM-5.3+). ReasoningEffort bool // IncludeReasoning: the provider needs the explicit reasoning opt-in diff --git a/request_options_test.go b/request_options_test.go index a9b783d..2f2f3e2 100644 --- a/request_options_test.go +++ b/request_options_test.go @@ -1,12 +1,8 @@ package llm import ( - "context" "encoding/json" "errors" - "fmt" - "net/http" - "net/http/httptest" "reflect" "strings" "testing" @@ -154,12 +150,18 @@ func TestParallelToolCallsAndSeed(t *testing.T) { func TestAnthropic_ForcedToolWithThinkingRejected(t *testing.T) { for _, choice := range []*ToolChoice{{Mode: ToolChoiceRequired}, ToolChoiceNamed("f")} { req := toolReq(choice) - req.Thinking = "low" + req.Thinking, req.ThinkingBudget = "low", 2000 // legacy budget path only (AC-6) _, err := buildAnthropicRequest(req, "claude", false) var ce *ConfigError if !errors.As(err, &ce) { t.Errorf("%+v: err = %v, want ConfigError", choice, err) } + // Adaptive thinking passes forced tool choice through unchanged. + req = toolReq(choice) + req.Thinking = "low" + if _, err := buildAnthropicRequest(req, "claude", false); err != nil { + t.Errorf("%+v adaptive: err = %v, want pass-through", choice, err) + } } req := toolReq(&ToolChoice{Mode: ToolChoiceAuto}) req.Thinking = "low" @@ -205,20 +207,14 @@ func TestResponseFormat_PerFormat(t *testing.T) { t.Errorf("gemini json_object = %v", gc) } - // Anthropic: a forced synthetic tool carries the schema. + // Anthropic: native output_config.format, no synthetic tool (M4 SL02: Anthropic now supports structured outputs natively). m, _ = buildBody(t, FormatAnthropic, "claude", schemaReq) - jsonEq(t, "anthropic tool_choice", m["tool_choice"], `{"type":"tool","name":"answer"}`) - tools := m["tools"].([]any) - last := tools[len(tools)-1].(map[string]any) - if last["name"] != "answer" { - t.Fatalf("anthropic json tool = %v", last) + jsonEq(t, "anthropic output_config", m["output_config"], `{"format":{"type":"json_schema","schema":`+string(testSchema)+`}}`) + if m["tools"] != nil || m["tool_choice"] != nil { + t.Errorf("anthropic native JSON must not add tools/tool_choice: %v", m) } - jsonEq(t, "anthropic json tool schema", last["input_schema"], string(testSchema)) m, _ = buildBody(t, FormatAnthropic, "claude", objReq) - tools = m["tools"].([]any) - if tools[0].(map[string]any)["name"] != anthropicJSONToolDefault { - t.Errorf("anthropic default json tool name = %v", tools[0]) - } + jsonEq(t, "anthropic json_object", m["output_config"], `{"format":{"type":"json_schema","schema":{"type":"object"}}}`) // Text is the default: nothing is sent. txt := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, ResponseFormat: &ResponseFormat{Type: ResponseText}} @@ -230,66 +226,21 @@ func TestResponseFormat_PerFormat(t *testing.T) { } } -func TestResponseFormat_AnthropicConflicts(t *testing.T) { +// Native JSON mode combines with tools, ToolChoice, thinking and non-object +// schemas (M4 SL02: the provider decides; the emulation-only ConfigErrors +// return in SL03 behind the AnthropicToolJSON quirk). +func TestResponseFormat_AnthropicNativeCombinations(t *testing.T) { rf := &ResponseFormat{Type: ResponseJSONSchema, Name: "f", Schema: testSchema} cases := []*ChatRequest{ - {Tools: []ToolDef{{Name: "f"}}, ResponseFormat: rf}, // name collision - {Tools: []ToolDef{{Name: "g"}}, ToolChoice: &ToolChoice{Mode: ToolChoiceAuto}, ResponseFormat: rf}, // forced tool vs ToolChoice - {Thinking: "low", ResponseFormat: &ResponseFormat{Type: ResponseJSONObject}}, // forced tool vs thinking + {Tools: []ToolDef{{Name: "f"}}, ResponseFormat: rf}, + {Tools: []ToolDef{{Name: "g"}}, ToolChoice: &ToolChoice{Mode: ToolChoiceAuto}, ResponseFormat: rf}, + {Thinking: "low", ThinkingBudget: 2000, ResponseFormat: &ResponseFormat{Type: ResponseJSONObject}}, + {ResponseFormat: &ResponseFormat{Type: ResponseJSONSchema, Schema: json.RawMessage(`{"type":"array"}`)}}, } for i, req := range cases { req.Messages = []Message{{Role: RoleUser, Content: "x"}} - _, err := buildAnthropicRequest(req, "claude", false) - var ce *ConfigError - if !errors.As(err, &ce) { - t.Errorf("case %d: err = %v, want ConfigError", i, err) - } - } -} - -// The JSON tool call is folded back into Content on both paths. -func TestResponseFormat_AnthropicResultFolding(t *testing.T) { - srvBody := `{"content":[{"type":"tool_use","id":"t","name":"answer","input":{"a":1}}],"stop_reason":"tool_use","usage":{"input_tokens":1,"output_tokens":2}}` - sse := strings.Join([]string{ - `event: message_start` + "\n" + `data: {"type":"message_start","message":{"usage":{"input_tokens":1}}}`, - `data: {"type":"content_block_start","index":0,"content_block":{"type":"tool_use","id":"t","name":"answer"}}`, - `data: {"type":"content_block_delta","index":0,"delta":{"type":"input_json_delta","partial_json":"{\"a\":"}}`, - `data: {"type":"content_block_delta","index":0,"delta":{"type":"input_json_delta","partial_json":"1}"}}`, - `data: {"type":"message_delta","delta":{"stop_reason":"tool_use"},"usage":{"output_tokens":2}}`, - `data: {"type":"message_stop"}`, - }, "\n\n") + "\n\n" - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if strings.Contains(r.Header.Get("Accept"), "event-stream") { - w.Header().Set("Content-Type", "text/event-stream") - _, _ = fmt.Fprint(w, sse) - return - } - _, _ = fmt.Fprint(w, srvBody) - })) - t.Cleanup(srv.Close) - c := newTestClient(t, ProviderConfig{ID: "anthropic", Format: FormatAnthropic, BaseURL: srv.URL}, srv) - req := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, - ResponseFormat: &ResponseFormat{Type: ResponseJSONSchema, Name: "answer", Schema: testSchema}} - - res, err := c.Call(context.Background(), req) - if err != nil { - t.Fatal(err) - } - if res.Content != `{"a":1}` || len(res.ToolCalls) != 0 || res.FinishReason != FinishStop { - t.Errorf("buffered = %+v", res) - } - - var deltas []Delta - res, err = c.CallStream(context.Background(), req, func(d Delta) error { deltas = append(deltas, d); return nil }) - if err != nil { - t.Fatal(err) - } - if res.Content != `{"a":1}` || len(res.ToolCalls) != 0 || res.FinishReason != FinishStop { - t.Errorf("stream = %+v", res) - } - for _, d := range deltas { - if d.Kind != DeltaContent { - t.Errorf("JSON mode must stream content deltas only, got %+v", d) + if _, err := buildAnthropicRequest(req, "claude", false); err != nil { + t.Errorf("case %d: err = %v, want native build", i, err) } } } diff --git a/responses.go b/responses.go index 7ce81e7..8326a0e 100644 --- a/responses.go +++ b/responses.go @@ -1,6 +1,7 @@ package llm import ( + "bytes" "encoding/base64" "encoding/json" "fmt" @@ -36,6 +37,8 @@ type rsInputContentPart struct { Type string `json:"type"` Text string `json:"text,omitempty"` ImageURL string `json:"image_url,omitempty"` + Filename string `json:"filename,omitempty"` + FileData string `json:"file_data,omitempty"` } func responsesContent(m Message) any { @@ -44,9 +47,14 @@ func responsesContent(m Message) any { } parts := make([]rsInputContentPart, 0, len(m.Parts)) for _, p := range m.Parts { - if p.Type == ContentPartImage { + switch p.Type { + case ContentPartImage: parts = append(parts, rsInputContentPart{Type: "input_image", ImageURL: "data:" + wireMIME(p.MIMEType) + ";base64," + base64.StdEncoding.EncodeToString(p.Image)}) - } else { + case ContentPartImageURL: + parts = append(parts, rsInputContentPart{Type: "input_image", ImageURL: p.URL}) + case ContentPartDocument: + parts = append(parts, rsInputContentPart{Type: "input_file", Filename: documentName(p), FileData: "data:application/pdf;base64," + base64.StdEncoding.EncodeToString(p.Data)}) + default: parts = append(parts, rsInputContentPart{Type: "input_text", Text: p.Text}) } } @@ -278,8 +286,9 @@ func buildResponsesInput(req *ChatRequest) (instructions string, input []any) { // ── response ───────────────────────────────────────────────────────────── type rsContentPart struct { - Type string `json:"type"` - Text string `json:"text"` + Type string `json:"type"` + Text string `json:"text"` + Refusal string `json:"refusal"` } type rsOutputItem struct { @@ -310,7 +319,19 @@ type rsErrorBody struct { Code string `json:"code"` } +// UnmarshalJSON tolerates a non-object "error" value ("", false, 0): it is +// not a provider error and must not fail an otherwise-good response. +func (e *rsErrorBody) UnmarshalJSON(b []byte) error { + if b = bytes.TrimSpace(b); len(b) == 0 || b[0] != '{' { + return nil + } + type plain rsErrorBody + return json.Unmarshal(b, (*plain)(e)) +} + type rsResponse struct { + ID string `json:"id"` + Model string `json:"model"` Status string `json:"status"` IncompleteDetails *struct { Reason string `json:"reason"` @@ -343,7 +364,7 @@ func usageFromResponses(u *rsUsage) Usage { } func chatResultFromResponses(r *rsResponse) *ChatResult { - res := &ChatResult{} + res := &ChatResult{Meta: ResponseMeta{ID: r.ID, Model: r.Model}} var summaries []string for _, item := range r.Output { switch item.Type { @@ -364,17 +385,24 @@ func chatResultFromResponses(r *rsResponse) *ChatResult { if p.Type == "output_text" && p.Text != "" { res.Content += p.Text } + if p.Type == "refusal" { + res.Refusal += p.Refusal + } } case "function_call": + args := item.Arguments + if args == "" { + args = "{}" + } res.ToolCalls = append(res.ToolCalls, ToolCall{ ID: item.CallID, Name: item.Name, - Arguments: item.Arguments, + Arguments: args, }) } } res.ReasoningContent = strings.Join(summaries, "\n") - res.FinishReason = responsesFinishReason(r, len(res.ToolCalls) > 0) + res.FinishReason = refusalFinish(responsesFinishReason(r, len(res.ToolCalls) > 0), res.Refusal) if r.Usage != nil { res.Usage = usageFromResponses(r.Usage) } @@ -405,6 +433,9 @@ func responsesFinishReason(r *rsResponse, hasTools bool) string { case strings.Contains(reason, "max_output"): return FinishLength } + if reason != "" { + return "" // unrecognised incomplete reason stays unmapped + } if hasTools { return FinishToolCalls } @@ -462,6 +493,9 @@ func mapResponsesStreamEvent(data []byte, acc *streamAccum) (deltas []Delta, don if err := json.Unmarshal(data, &ev); err != nil { return nil, false, fmt.Errorf("llm: parse responses stream: %w", err) } + if ev.Response != nil { + acc.noteMeta(ev.Response.ID, ev.Response.Model) + } switch ev.Type { case "response.reasoning_summary_text.delta": if ev.Delta != "" { @@ -473,6 +507,11 @@ func mapResponsesStreamEvent(data []byte, acc *streamAccum) (deltas []Delta, don acc.content.WriteString(ev.Delta) deltas = append(deltas, Delta{Kind: DeltaContent, Text: ev.Delta}) } + case "response.refusal.delta": + if ev.Delta != "" { + acc.refusal.WriteString(ev.Delta) + deltas = append(deltas, Delta{Kind: DeltaRefusal, Text: ev.Delta}) + } case "response.function_call_arguments.delta": call := acc.call(ev.OutputIndex) call.args.WriteString(ev.Delta) diff --git a/responses_edges_test.go b/responses_edges_test.go index d144b66..a37f126 100644 --- a/responses_edges_test.go +++ b/responses_edges_test.go @@ -27,7 +27,8 @@ func TestResponsesFinishReason_Table(t *testing.T) { {"max output", inc("max_output_tokens"), true, FinishLength}, {"content filter", inc("content_filter"), false, FinishContentFilter}, {"content filter with tools", inc("content_filter"), true, FinishContentFilter}, - {"incomplete unknown, tools", inc("other"), true, FinishToolCalls}, + // Unrecognised incomplete reasons are unmapped: "" (AGENTS.md invariant 1). + {"incomplete unknown, tools", inc("other"), true, ""}, {"incomplete no details", inc(""), false, FinishLength}, {"failed", &rsResponse{Status: "failed"}, true, ""}, {"cancelled", &rsResponse{Status: "cancelled"}, false, ""}, diff --git a/retry.go b/retry.go index a90c46b..f8416fd 100644 --- a/retry.go +++ b/retry.go @@ -50,6 +50,9 @@ func parseRetryAfter(v string, now time.Time) time.Duration { if secs < 0 { return 0 } + if secs > int(maxRetryAfter/time.Second) { + return maxRetryAfter + } d = time.Duration(secs) * time.Second } else if t, err := http.ParseTime(v); err == nil { if delta := t.Sub(now); delta > 0 { @@ -116,6 +119,30 @@ func retrySleep(ctx context.Context, d time.Duration) bool { } } +// waitRetry sleeps d before the next attempt unless d cannot fit in the +// context's remaining budget, in which case it returns false immediately +// instead of sleeping to the deadline. It also returns false when the +// context ends during the wait. +func waitRetry(ctx context.Context, d time.Duration) bool { + if dl, ok := ctx.Deadline(); ok && time.Until(dl) <= d { + return false + } + return retrySleep(ctx, d) +} + +// withBudgetCause returns last (the final provider or transport error) +// annotated with why the ladder stopped, so the cause stays retrievable via +// errors.As while errors.Is(err, context.DeadlineExceeded/Canceled) still +// holds. When the wait was skipped because it exceeds the remaining +// budget, the deadline is reported. +func withBudgetCause(ctx context.Context, last error) error { + cause := ctx.Err() + if cause == nil { + cause = context.DeadlineExceeded + } + return fmt.Errorf("%w (%w)", last, cause) +} + // RetryPolicy configures the retry ladder shared by chat (buffered and // streaming), speech, transcription and embeddings. Zero fields keep the // defaults. @@ -170,7 +197,11 @@ func terminal(err error) error { // retryable statuses and transport errors back off; anything else (and any // terminal error) is returned as is. func (pc *providerClient) withRetry(ctx context.Context, op func() (time.Duration, error), learn func(*APIError) bool) error { - n := pc.opts.retry.attempts() + return pc.withRetryN(ctx, pc.opts.retry.attempts(), op, learn) +} + +// withRetryN is withRetry with an explicit attempt budget (n >= 1). +func (pc *providerClient) withRetryN(ctx context.Context, n int, op func() (time.Duration, error), learn func(*APIError) bool) error { var ( rateErr *APIError // last 429 rateRA time.Duration @@ -181,10 +212,12 @@ func (pc *providerClient) withRetry(ctx context.Context, op func() (time.Duratio } ra, err := op() if err == nil { + pc.emitAttempt(ctx, nil, 0) return nil } var te *terminalError if errors.As(err, &te) { + pc.emitAttempt(ctx, te.err, 0) return te.err } var ce *ConfigError @@ -197,13 +230,17 @@ func (pc *providerClient) withRetry(ctx context.Context, op func() (time.Duratio // Transport error — retryable. if cerr := ctx.Err(); cerr != nil { // Interrupted by deadline/cancel, not exhaustion. + pc.emitAttempt(ctx, err, 0) return cerr } if last { + pc.emitAttempt(ctx, err, 0) return fmt.Errorf("llm: retry exhausted (%d attempts): %w", n, err) } - if !retrySleep(ctx, pc.retryDelay(0, attempt)) { - return ctx.Err() + d := pc.retryDelay(0, attempt) + pc.emitAttempt(ctx, err, d) + if !waitRetry(ctx, d) { + return withBudgetCause(ctx, err) } continue } @@ -211,12 +248,15 @@ func (pc *providerClient) withRetry(ctx context.Context, op func() (time.Duratio case apiErr.Status == http.StatusTooManyRequests: if billingExhausted(apiErr) { // Permanent: only a recharge fixes this. + pc.emitAttempt(ctx, apiErr, 0) return apiErr } rateErr, rateRA = apiErr, ra if !last { - if !retrySleep(ctx, pc.retryDelay(ra, attempt)) { - // The sleep died with the context (deadline or cancel); + d := pc.retryDelay(ra, attempt) + pc.emitAttempt(ctx, apiErr, d) + if !waitRetry(ctx, d) { + // The wait would outlive (or died with) the context; // still surface the 429 — the caller needs // Status/RetryAfter to plan the retry. return &RateLimitError{APIError: *rateErr, Attempts: attempt + 1, RetryAfter: rateRA} @@ -224,17 +264,21 @@ func (pc *providerClient) withRetry(ctx context.Context, op func() (time.Duratio continue } case apiErr.Retryable && !last: - if !retrySleep(ctx, pc.retryDelay(ra, attempt)) { - return ctx.Err() + d := pc.retryDelay(ra, attempt) + pc.emitAttempt(ctx, apiErr, d) + if !waitRetry(ctx, d) { + return withBudgetCause(ctx, apiErr) } continue case learn != nil && learn(apiErr): // Learn-once fallback engaged: retry immediately. + pc.emitAttempt(ctx, apiErr, 0) if !last { continue } return apiErr } + pc.emitAttempt(ctx, apiErr, 0) if rateErr != nil && !apiErr.Retryable && apiErr.Status != http.StatusTooManyRequests { // A definitive failure after earlier 429s. return apiErr diff --git a/review_fixes_test.go b/review_fixes_test.go index 96db57d..2837627 100644 --- a/review_fixes_test.go +++ b/review_fixes_test.go @@ -75,18 +75,16 @@ func TestAnthropic_ControlEdges(t *testing.T) { m, _ := buildBody(t, FormatAnthropic, "claude", req) jsonEq(t, "none choice", m["tool_choice"], `{"type":"none"}`) - var ce *ConfigError - // JSON mode forces its own tool, so user tools could never be called. + // Native JSON mode (M4 SL02) builds alongside tools and non-object schemas. withTools := toolReq(nil) withTools.ResponseFormat = &ResponseFormat{Type: ResponseJSONObject} - if _, err := buildAnthropicRequest(withTools, "claude", false); !errors.As(err, &ce) { - t.Errorf("JSON mode + tools: err = %v", err) + if _, err := buildAnthropicRequest(withTools, "claude", false); err != nil { + t.Errorf("JSON mode + tools must build natively: %v", err) } - // Anthropic input_schema must be an object schema. arr := &ChatRequest{Messages: []Message{{Role: RoleUser, Content: "x"}}, ResponseFormat: &ResponseFormat{Type: ResponseJSONSchema, Schema: json.RawMessage(`{"type":"array"}`)}} - if _, err := buildAnthropicRequest(arr, "claude", false); !errors.As(err, &ce) { - t.Errorf("non-object schema: err = %v", err) + if _, err := buildAnthropicRequest(arr, "claude", false); err != nil { + t.Errorf("non-object schema must build natively: %v", err) } } diff --git a/sse.go b/sse.go index a7add8e..ea6d2d2 100644 --- a/sse.go +++ b/sse.go @@ -2,6 +2,7 @@ package llm import ( "bufio" + "bytes" "context" "io" "time" @@ -41,11 +42,17 @@ type sseItem struct { // goroutine on a channel send. func parseSSEStream(r io.Reader, ch chan<- sseItem, done <-chan struct{}) { defer close(ch) - br := bufio.NewReaderSize(r, 64*1024) + lr := &sseLineReader{br: bufio.NewReaderSize(r, 64*1024)} var data []byte sawActivity := false + first := true for { - line, err := readSSELine(br) + line, err := lr.readSSELine() + if first { + // A leading UTF-8 BOM is not part of the first field name. + line = bytes.TrimPrefix(line, []byte("\xef\xbb\xbf")) + first = false + } if len(line) > 0 { sawActivity = true switch { @@ -107,32 +114,41 @@ func sendSSE(ch chan<- sseItem, done <-chan struct{}, it sseItem) bool { } } -// readSSELine reads one line including the trailing newline; returns the -// line without it (also strips a trailing \r\n for CRLF frames). Lines -// longer than the bufio buffer are accumulated across ReadSlice calls, -// bounded by sseMaxLineSize. -func readSSELine(br *bufio.Reader) ([]byte, error) { +// sseLineReader splits SSE input on \n, \r\n or a bare \r (all three are +// line endings per the SSE spec). A \r at the end of the buffered data +// never blocks waiting for a possible \n: the pending-LF flag skips it when +// it arrives. +type sseLineReader struct { + br *bufio.Reader + skipLF bool +} + +// readSSELine reads one line and returns it without its terminator. Lines +// longer than sseMaxLineSize fail with errSSEOversized. +func (lr *sseLineReader) readSSELine() ([]byte, error) { var buf []byte for { - frag, err := br.ReadSlice('\n') - if len(buf)+len(frag) > sseMaxLineSize { - return nil, errSSEOversized - } - buf = append(buf, frag...) - if err == bufio.ErrBufferFull { - continue // line spans the buffer boundary; keep accumulating - } + c, err := lr.br.ReadByte() if err != nil { - return buf, err // EOF or read error; trailing line lacks '\n' + return buf, err // EOF or read error; trailing line lacks a terminator } - l := len(buf) - if l > 0 && buf[l-1] == '\n' { - l-- - if l > 0 && buf[l-1] == '\r' { - l-- + if lr.skipLF { + lr.skipLF = false + if c == '\n' { + continue } } - return buf[:l], nil + switch c { + case '\n': + return buf, nil + case '\r': + lr.skipLF = true + return buf, nil + } + if len(buf) >= sseMaxLineSize { + return nil, errSSEOversized + } + buf = append(buf, c) } } @@ -186,12 +202,14 @@ func pumpSSE(ctx context.Context, r io.ReadCloser, idle time.Duration, handle fu } switch it.kind { case sseData: - if timer != nil { - timer.Reset(idle) - } + // Only provider silence counts: the consumer's handle time + // is excluded, so the timer restarts after handle returns. if err := handle(it.data); err != nil { return err } + if timer != nil { + timer.Reset(idle) + } case sseKeepalive: if timer != nil { timer.Reset(idle) diff --git a/stt.go b/stt.go index 1330861..a09a2fc 100644 --- a/stt.go +++ b/stt.go @@ -30,12 +30,13 @@ const maxTranscribeAudioBytes = 25 << 20 // TranscribeRequest describes one speech-to-text conversion. type TranscribeRequest struct { - Audio []byte // required, non-empty: raw audio bytes (caller-owned) - Filename string // file part name for the audio (default "audio.wav") - MIMEType string // optional content type for the audio part - Language string // optional ISO-639-1 hint, OpenAI-compat only - Prompt string // optional conditioning text, OpenAI-compat only - Format string // optional response_format (provider default when empty) + Headers map[string]string // per-request headers (SL03) + Audio []byte // required, non-empty: raw audio bytes (caller-owned) + Filename string // file part name for the audio (default "audio.wav") + MIMEType string // optional content type for the audio part + Language string // optional ISO-639-1 hint, OpenAI-compat only + Prompt string // optional conditioning text, OpenAI-compat only + Format string // optional response_format (provider default when empty) } // TranscribeResult carries the recognized text. Fields the provider @@ -45,6 +46,7 @@ type TranscribeResult struct { Model string Language string DurationSec float64 + Meta ResponseMeta } // Transcribe converts speech to text with the named provider and @@ -130,21 +132,26 @@ func (pc *providerClient) buildTranscribeRequest(model string, req TranscribeReq // transcribe runs the STT request against one provider with the shared // retry ladder and whole-call budget of the buffered chat path. func (pc *providerClient) transcribe(ctx context.Context, model string, req TranscribeRequest) (*TranscribeResult, error) { + ctx, herr := withReqHeaders(ctx, req.Headers) + if herr != nil { + return nil, herr + } body, url, err := pc.buildTranscribeRequest(model, req) if err != nil { return nil, err } ctype := "multipart/form-data; boundary=" + multipartBodyBoundary(body) + ctx = pc.withAttempts(ctx, "transcribe", model) ctx, cancel := context.WithTimeout(ctx, pc.requestTimeout()) defer cancel() var res *TranscribeResult err = pc.withRetry(ctx, func() (time.Duration, error) { - data, ra, err := pc.postMultipart(ctx, url, body, ctype) + data, hdr, ra, err := pc.postMultipart(ctx, url, body, ctype) if err != nil { return ra, err } - r, perr := parseTranscribeResponse(data) + r, perr := parseTranscribeBody(data, req.Format) if perr != nil { // 2xx with an undecodable body is a provider protocol // failure — surface it through the typed error taxonomy @@ -156,6 +163,7 @@ func (pc *providerClient) transcribe(ctx context.Context, model string, req Tran }) } r.Model = model + r.Meta = parseResponseMeta(hdr) res = r return 0, nil }, nil) @@ -185,6 +193,16 @@ type transcribeResponse struct { Duration float64 `json:"duration"` } +// parseTranscribeBody maps a 2xx body to a result. The text, srt and vtt +// response formats are not JSON: the raw body is the transcription. +func parseTranscribeBody(data []byte, format string) (*TranscribeResult, error) { + switch strings.ToLower(format) { + case "text", "srt", "vtt": + return &TranscribeResult{Text: string(data)}, nil + } + return parseTranscribeResponse(data) +} + // parseTranscribeResponse decodes a buffered transcription response. // Both plain json ({text}) and verbose_json ({text,language,duration}) // decode through the same struct — fields the provider omits stay zero. @@ -202,29 +220,29 @@ func parseTranscribeResponse(data []byte) (*TranscribeResult, error) { // postMultipart sends one transcription request and reads the full JSON // body (capped). Mirrors postAudio but with the multipart content type. -func (pc *providerClient) postMultipart(ctx context.Context, url string, body []byte, ctype string) (data []byte, ra time.Duration, err error) { +func (pc *providerClient) postMultipart(ctx context.Context, url string, body []byte, ctype string) (data []byte, hdr http.Header, ra time.Duration, err error) { req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(body)) if err != nil { - return nil, 0, &ConfigError{Msg: "build request: " + err.Error()} + return nil, nil, 0, &ConfigError{Msg: "build request: " + err.Error()} } req.Header.Set("Content-Type", ctype) - pc.setHeaders(req.Header) + pc.setHeaders(req.Header, reqHeadersFrom(ctx)) - resp, err := pc.buffered().Do(req) + resp, err := doObserved(ctx, pc.buffered(), req) if err != nil { - return nil, 0, err + return nil, nil, 0, err } defer func() { _ = resp.Body.Close() }() data, err = io.ReadAll(io.LimitReader(resp.Body, maxResponseSize+1)) if err != nil { - return nil, 0, err + return nil, nil, 0, err } if len(data) > maxResponseSize { - return nil, 0, fmt.Errorf("llm: response exceeds %d bytes", maxResponseSize) + return nil, nil, 0, terminal(fmt.Errorf("llm: response exceeds %d bytes", maxResponseSize)) } ra = parseRetryAfter(resp.Header.Get("Retry-After"), time.Now()) if resp.StatusCode < 200 || resp.StatusCode >= 300 { - return nil, ra, pc.httpError(resp.StatusCode, data) + return nil, nil, ra, pc.httpError(resp.StatusCode, data) } - return data, ra, nil + return data, resp.Header, ra, nil } diff --git a/tools.go b/tools.go new file mode 100644 index 0000000..cf0c44c --- /dev/null +++ b/tools.go @@ -0,0 +1,122 @@ +package llm + +import ( + "context" + "errors" + "fmt" + "sync" +) + +// ToolHandler executes one model-requested tool call and returns its result. +type ToolHandler func(ctx context.Context, call ToolCall) (string, error) + +// ToolLoopOptions tunes RunTools. +type ToolLoopOptions struct { + // MaxTurns is the number of chat calls allowed; <=0 means 8. + MaxTurns int + // Parallel runs one turn's handlers concurrently. Results are still + // appended in call order. + Parallel bool + // OnTurn is called after each chat call (1-based), before handlers run. + // Panics in it are not recovered. + OnTurn func(turn int, res *ChatResult) +} + +// ErrMaxToolTurns reports that the model still wanted tools after MaxTurns. +var ErrMaxToolTurns = errors.New("llm: max tool turns exceeded") + +const defaultMaxToolTurns = 8 + +// RunTools drives a buffered tool loop: it calls the model, runs the +// requested tools through handlers, appends the results and repeats until +// the model stops asking for tools. It returns the final result and the +// complete transcript (req.Messages plus everything appended). req.Messages +// is copied; the caller's slice is never mutated. +// +// A handler error (or panic, or a missing handler) becomes an IsError tool +// result and the loop continues. If MaxTurns chat calls all end in tool +// calls, the last result is returned with ErrMaxToolTurns and a transcript +// without the final (unanswered) assistant turn. +func RunTools(ctx context.Context, cc *ChatClient, req ChatRequest, handlers map[string]ToolHandler, opts ToolLoopOptions) (*ChatResult, []Message, error) { + if cc == nil { + return nil, nil, &ConfigError{Msg: "RunTools: nil ChatClient"} + } + maxTurns := opts.MaxTurns + if maxTurns <= 0 { + maxTurns = defaultMaxToolTurns + } + req.Messages = append([]Message(nil), req.Messages...) + var last *ChatResult + for turn := 1; turn <= maxTurns; turn++ { + if err := ctx.Err(); err != nil { + return last, req.Messages, err + } + res, err := cc.Call(ctx, &req) + if err != nil { + return last, req.Messages, err + } + last = res + if opts.OnTurn != nil { + opts.OnTurn(turn, res) + } + if res.FinishReason != FinishToolCalls || len(res.ToolCalls) == 0 { + req.Messages = append(req.Messages, res.AssistantMessage()) + return res, req.Messages, nil + } + if turn == maxTurns { + return res, req.Messages, fmt.Errorf("%w (%d turns)", ErrMaxToolTurns, maxTurns) + } + results := runToolCalls(ctx, res.ToolCalls, handlers, opts.Parallel) + req.Messages = append(req.Messages, res.AssistantMessage()) + req.Messages = append(req.Messages, results...) + if err := ctx.Err(); err != nil { + return last, req.Messages, err + } + } + return last, req.Messages, nil // unreachable: the loop returns on its last turn +} + +func runToolCalls(ctx context.Context, calls []ToolCall, handlers map[string]ToolHandler, parallel bool) []Message { + out := make([]Message, len(calls)) + run := func(i int) { out[i] = runToolCall(ctx, calls[i], handlers[calls[i].Name]) } + if !parallel { + for i := range calls { + run(i) + } + return out + } + var wg sync.WaitGroup + for i := range calls { + wg.Add(1) + go func() { + defer wg.Done() + run(i) + }() + } + wg.Wait() + return out +} + +func runToolCall(ctx context.Context, call ToolCall, h ToolHandler) (msg Message) { + msg = Message{Role: RoleTool, ToolCallID: call.ID, ToolName: call.Name} + if h == nil { + msg.Content, msg.IsError = "unknown tool", true + return msg + } + if err := ctx.Err(); err != nil { // cancelled: do not start another handler + msg.Content, msg.IsError = err.Error(), true + return msg + } + defer func() { + if r := recover(); r != nil { + msg.Content, msg.IsError = fmt.Sprintf("tool %s panicked: %v", call.Name, r), true + } + }() + content, err := h(ctx, call) + if err != nil { + msg.Content, msg.IsError = err.Error(), true + return msg + } + msg.Content = content + return msg +} diff --git a/transport.go b/transport.go index 2e9a9e0..c62a7f1 100644 --- a/transport.go +++ b/transport.go @@ -60,5 +60,10 @@ func sameHostRedirect(req *http.Request, via []*http.Request) error { if len(via) >= 10 || req.URL.Host != via[0].URL.Host { return http.ErrUseLastResponse } + // Never downgrade https -> http: that would resend credentials in + // plaintext. + if via[0].URL.Scheme == "https" && req.URL.Scheme != "https" { + return http.ErrUseLastResponse + } return nil } diff --git a/tts.go b/tts.go index bfffe41..d4d770a 100644 --- a/tts.go +++ b/tts.go @@ -25,10 +25,11 @@ import ( // SpeakRequest describes one text-to-speech conversion. type SpeakRequest struct { - Text string // required, non-empty - Voice string // provider voice id (no fallback tables — invariant #5) - Format string // container for OpenAI-compat providers ("mp3" default) - Speed float64 // optional playback speed, OpenAI-compat only (0 = omit) + Headers map[string]string // per-request headers (SL03) + Text string // required, non-empty + Voice string // provider voice id (no fallback tables — invariant #5) + Format string // container for OpenAI-compat providers ("mp3" default) + Speed float64 // optional playback speed, OpenAI-compat only (0 = omit) } // SpeakResult carries the synthesized audio. @@ -36,6 +37,7 @@ type SpeakResult struct { Audio []byte // raw audio bytes as the provider returned them Model string // model that produced the audio MIMEType string // provider-declared (or well-known default) MIME type + Meta ResponseMeta } // Speak synthesizes speech with the named provider and model. Buffered @@ -90,17 +92,22 @@ func (pc *providerClient) buildSpeakRequest(model string, req SpeakRequest) ([]b // ladder and whole-call budget of the buffered chat path (binary body, no // SSE). func (pc *providerClient) speak(ctx context.Context, model string, req SpeakRequest) (*SpeakResult, error) { + ctx, herr := withReqHeaders(ctx, req.Headers) + if herr != nil { + return nil, herr + } body, url, mimeHint, err := pc.buildSpeakRequest(model, req) if err != nil { return nil, err } gemini := pc.cfg.Format == FormatGemini + ctx = pc.withAttempts(ctx, "speak", model) ctx, cancel := context.WithTimeout(ctx, pc.requestTimeout()) defer cancel() var res *SpeakResult err = pc.withRetry(ctx, func() (time.Duration, error) { - data, ctype, ra, err := pc.postAudio(ctx, url, body) + data, ctype, hdr, ra, err := pc.postAudio(ctx, url, body) if err != nil { return ra, err } @@ -128,6 +135,7 @@ func (pc *providerClient) speak(ctx context.Context, model string, req SpeakRequ Audio: data, Model: model, MIMEType: resolveAudioMIME(ctype, mimeHint), + Meta: parseResponseMeta(hdr), } return 0, nil }, nil) @@ -150,32 +158,32 @@ func resolveAudioMIME(ctype, fallback string) string { // postAudio sends one TTS request and reads the full binary body (capped). // Unlike post it captures the response Content-Type and never assumes JSON. -func (pc *providerClient) postAudio(ctx context.Context, url string, body []byte) (data []byte, ctype string, ra time.Duration, err error) { +func (pc *providerClient) postAudio(ctx context.Context, url string, body []byte) (data []byte, ctype string, hdr http.Header, ra time.Duration, err error) { req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(body)) if err != nil { - return nil, "", 0, &ConfigError{Msg: "build request: " + err.Error()} + return nil, "", nil, 0, &ConfigError{Msg: "build request: " + err.Error()} } req.Header.Set("Content-Type", "application/json") - pc.setHeaders(req.Header) + pc.setHeaders(req.Header, reqHeadersFrom(ctx)) - resp, err := pc.buffered().Do(req) + resp, err := doObserved(ctx, pc.buffered(), req) if err != nil { - return nil, "", 0, err + return nil, "", nil, 0, err } defer func() { _ = resp.Body.Close() }() ctype = resp.Header.Get("Content-Type") data, err = io.ReadAll(io.LimitReader(resp.Body, maxResponseSize+1)) if err != nil { - return nil, "", 0, err + return nil, "", nil, 0, err } if len(data) > maxResponseSize { - return nil, "", 0, fmt.Errorf("llm: audio response exceeds %d bytes", maxResponseSize) + return nil, "", nil, 0, terminal(fmt.Errorf("llm: audio response exceeds %d bytes", maxResponseSize)) } ra = parseRetryAfter(resp.Header.Get("Retry-After"), time.Now()) if resp.StatusCode < 200 || resp.StatusCode >= 300 { - return nil, "", ra, pc.httpError(resp.StatusCode, data) + return nil, "", nil, ra, pc.httpError(resp.StatusCode, data) } - return data, ctype, ra, nil + return data, ctype, resp.Header, ra, nil } // ── Gemini TTS wire types ──────────────────────────────────────────────── @@ -228,7 +236,7 @@ func buildGeminiSpeakRequest(base, model string, req SpeakRequest) ([]byte, stri }, } b, err := json.Marshal(greq) - return b, fmt.Sprintf("%s/v1beta/models/%s:generateContent", base, model), "", err + return b, fmt.Sprintf("%s/v1beta/models/%s:generateContent", base, geminiModelPath(model)), "", err } // parseGeminiSpeakResponse decodes a buffered Gemini TTS response.