fix: Calculate stats for streaming responses

[grahamking]

Fixed the /v1/routing/stats streaming zero-usage bug by adding a stream-wrapping helper in switchyard-components-v2 and wiring it into all four profiles. Mostly a port from v1 switchyard-components.

Includes unit test that reproduces the bug against random routing profile.

Closes: #55
Closes: #18

Assisted-by: Claude:Opus 4.7 High

Summary by CodeRabbit

• New Features

  • Added reliable token-usage tracking for both standard and streaming responses.
  • Streaming requests now record usage when usage data becomes available, including OpenAI and Anthropic responses.
  • Added coverage for usage attribution on streamed, routed requests.

• Bug Fixes

  • Improved latency and usage attribution across routing profiles.
  • Recording issues no longer interrupt an active response stream.

[coderabbitai]

Walkthrough

The v2 components add shared buffered and streaming token-usage recording. LLM routing, passthrough, random routing, and stage routing use the helper, which wraps provider streams and records usage when available. Random routing adds coverage for delayed streaming usage events.

Changes

Streaming usage recording

Layer / File(s)	Summary
Shared usage recording helper crates/switchyard-components-v2/Cargo.toml, crates/switchyard-components-v2/src/lib.rs, crates/switchyard-components-v2/src/stats_recording.rs	Adds async stream dependencies and implements buffered, OpenAI, and Anthropic usage recording with latency attribution and warning logs.
Profile success-path integration crates/switchyard-components-v2/src/profiles/llm_routing.rs, crates/switchyard-components-v2/src/profiles/passthrough.rs, crates/switchyard-components-v2/src/profiles/random_routing.rs, crates/switchyard-components-v2/src/profiles/stage_router.rs	Routes successful responses through the shared helper and returns wrapped streaming responses where applicable.
Streaming attribution validation crates/switchyard-components-v2/src/profiles/random_routing.rs	Adds a streaming backend and verifies delayed usage recording, token totals, model attribution, and weak-tier routing.

Estimated code review effort: 4 (Complex) | ~45 minutes

Poem

A rabbit streams through fields of code,
And usage hops along the road.
OpenAI, Anthropic join the play,
Tokens bloom when streams convey.
Stats now catch each watery byte—
Hop, hop, recorded right!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: calculating stats for streaming responses.
Linked Issues check	✅ Passed	The PR adds stream-aware usage recording, updates the affected v2 profiles, and includes a test for streaming usage attribution.
Out of Scope Changes check	✅ Passed	The changes stay focused on streaming stats recording and related profile updates, with no obvious unrelated additions.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

crates/switchyard-components-v2/src/stats_recording.rs (1)

109-170: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Missing test coverage for wrap_openai_responses and wrap_anthropic.

Only the OpenAiStream/wrap_openai_chat path is exercised (via the new test in random_routing.rs). This file has no #[cfg(test)] module, and the other two streaming variants — wrap_openai_responses and wrap_anthropic (with its distinct accumulator-based commit logic) — are untested in the provided context, despite being core to the fix described by this PR.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/switchyard-components-v2/src/stats_recording.rs` around lines 109 -
170, Add a #[cfg(test)] module in stats_recording.rs covering both
wrap_openai_responses and wrap_anthropic. Exercise representative usage-bearing
stream events, consume the wrapped streams, and assert each path records usage
once with the expected accumulated values, including Anthropic’s
accumulator-based commit behavior.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/switchyard-components-v2/src/stats_recording.rs`:
- Around line 76-170: Add concise doc comments to wrap_openai_chat,
wrap_openai_responses, and wrap_anthropic describing when usage is recorded,
that stream events are forwarded unchanged, and the differing single-shot versus
accumulator-driven commit behavior, including why only the OpenAI wrappers use a
committed guard.
- Around line 172-189: The record_usage function currently measures elapsed time
too late for streaming responses. Capture the backend elapsed latency
immediately after the backend call returns, pass that fixed value through the
stream wrappers, and update record_usage to use it when calculating
total_latency_ms and routing_overhead_ms instead of calling
profile_started_at.elapsed().

---

Nitpick comments:
In `@crates/switchyard-components-v2/src/stats_recording.rs`:
- Around line 109-170: Add a #[cfg(test)] module in stats_recording.rs covering
both wrap_openai_responses and wrap_anthropic. Exercise representative
usage-bearing stream events, consume the wrapped streams, and assert each path
records usage once with the expected accumulated values, including Anthropic’s
accumulator-based commit behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f764206e-306c-4894-b56c-c737bbd5f106

📥 Commits

Reviewing files that changed from the base of the PR and between bfabc3b and 6ac4e48.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock, !Cargo.lock

📒 Files selected for processing (7)

• crates/switchyard-components-v2/Cargo.toml
• crates/switchyard-components-v2/src/lib.rs
• crates/switchyard-components-v2/src/profiles/llm_routing.rs
• crates/switchyard-components-v2/src/profiles/passthrough.rs
• crates/switchyard-components-v2/src/profiles/random_routing.rs
• crates/switchyard-components-v2/src/profiles/stage_router.rs
• crates/switchyard-components-v2/src/stats_recording.rs

[ryan-lempka]

LGTM. Can we make sure to comment on / close #21 that this fix is going in so that PR can be closed?