Skip to content

fallbackStructuredOutputStream drops usage from structured-output.complete and RUN_FINISHED chunks #758

@songlairui

Description

@songlairui

Summary

In @tanstack/ai@0.28.0, fallbackStructuredOutputStream (used when an adapter doesn't implement structuredOutputStream — e.g. AnthropicTextAdapter) calls adapter.structuredOutput(), gets back { data, rawText, usage }, but discards result.usage entirely. Downstream consumers reading token usage from either RUN_FINISHED.usage or structured-output.complete.value.usage see undefined, even though the adapter correctly populated the field from the provider response.

Reproduction

Any call shaped like:

```ts
import { chat, EventType } from "@tanstack/ai";
import { AnthropicTextAdapter } from "@tanstack/ai-anthropic";

const stream = chat({
adapter: new AnthropicTextAdapter({ apiKey, baseURL }, "claude-3-5-sonnet"),
systemPrompts: ["..."],
messages: [{ role: "user", content: "..." }],
outputSchema: mySchema,
stream: true,
});

for await (const chunk of stream) {
if (chunk.type === EventType.RUN_FINISHED) {
console.log(chunk.usage); // undefined
}
if (chunk.type === "CUSTOM" && chunk.name === "structured-output.complete") {
console.log(chunk.value.usage); // undefined
}
}
```

The actual provider response includes usage (verified via `console.log(response.usage)` inside `AnthropicTextAdapter.structuredOutput()` — `input_tokens`, `output_tokens`, `cache_read_input_tokens` all present), and `buildAnthropicUsage()` produces a complete `TokenUsage` object, but it never reaches consumers.

Root cause

In `packages/ai/src/activities/chat/index.ts` (built artifact: `dist/esm/activities/chat/index.js` around line 1607 of `@tanstack/ai@0.28.0`):

```js
const result = await adapter.structuredOutput(options); // result.usage is populated here

yield {
type: EventType.CUSTOM,
name: "structured-output.complete",
value: { object: result.data, raw: result.rawText }, // result.usage dropped
model,
timestamp,
};
yield {
type: EventType.RUN_FINISHED,
runId,
threadId,
model,
timestamp,
finishReason: "stop", // no usage field
};
```

Both yields drop `result.usage` despite the adapter's `structuredOutput` return contract including it.

Suggested fix

```diff
yield {
type: EventType.CUSTOM,
name: "structured-output.complete",

  • value: { object: result.data, raw: result.rawText },
  • value: { object: result.data, raw: result.rawText, usage: result.usage },
    model,
    timestamp,
    };
    yield {
    type: EventType.RUN_FINISHED,
    runId,
    threadId,
    model,
    timestamp,
  • finishReason: "stop",
  • finishReason: "stop",
  • usage: result.usage,
    };
    ```

Both touch points needed for backward compat with consumers that read either chunk shape.

Note: `StructuredOutputCompleteEvent.value` type in `types.d.ts:1009` would need `usage?: TokenUsage` added; `RunFinishedEvent.usage` already exists at `types.d.ts:781` (just unused on this path).

Impact

Any cost-tracking / observability layer that relies on `usage` in fallback path streams reports zero/null token counts. Pricing snapshots write but token counts are NULL → cost computation = 0. We hit this on DeepSeek (Anthropic endpoint) where every structured-output call goes through fallback because `AnthropicTextAdapter` doesn't implement `structuredOutputStream`.

Local patch via `pnpm patch` confirmed fix: token counts now flow end-to-end (verified with `promptTokens=125`, `completionTokens=1346`, `promptTokensDetails.cachedTokens=5760` reaching consumer).

Environment

  • `@tanstack/ai@0.28.0`
  • `@tanstack/ai-anthropic@0.15.1`
  • Adapter: `AnthropicTextAdapter` against DeepSeek's `/anthropic` compatible endpoint
  • Node 20.x

Related: #390 (middleware bypass in same fallback path), #526 (request for native streaming structured output in openrouter adapter — would also bypass this fallback when implemented).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions