feat(traces): Show cached prompt tokens in token breakdowns#5352
feat(traces): Show cached prompt tokens in token breakdowns#5352ashrafchowdury wants to merge 3 commits into
Conversation
dosubot
Bot
added
the
size:L
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (15)
📝 WalkthroughSummary by CodeRabbit
WalkthroughCache read/write token counts are added to agent usage contracts, aggregation, OpenTelemetry attributes, cumulative tracing, SDK wire serialization, and web token displays. Prompt totals now include cached tokens while cache read/write values remain separately available. ChangesCache-aware token accounting
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Agent
participant RunnerUsage
participant OpenTelemetry
participant TraceSelectors
participant TraceDetails
Agent->>RunnerUsage: Emit input/output/cacheRead/cacheWrite usage
RunnerUsage->>OpenTelemetry: Aggregate usage and stamp gen_ai.usage attributes
OpenTelemetry->>TraceSelectors: Expose cumulative token metrics
TraceSelectors->>TraceDetails: Provide formatted cache token values
TraceDetails->>TraceDetails: Render cache read/write token rows
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
I had faced this issue several times, where I wanted to see how much that amount of token costs, in which part, where and how. The token usage detail popover was misleading |
Railway Preview Environment
Updated at 2026-07-17T07:44:24.763Z |
|
Hey @ashrafchowdury This is a great idea, and the user-facing problem is real. Showing cached-token usage makes the numbers much easier to understand. However, this implementation does not fit the current plan for usage and cost telemetry. We already have a broader design in #5253 that covers the runner contract, cache semantics, cost provenance, trace attribution, and rollups. As a general rule, changes to interfaces inside the system should start with a discussion and design sync before implementation. The boundaries do not all have the same stability or freedom to change. We currently have more freedom at the service-to-runner boundary. That part of the system is new, still somewhat ad hoc, and has not yet been fully refactored or organized. Quick changes there can be acceptable to some degree, especially when they clarify ownership and move the design toward a cleaner structure. Changes in the other direction, especially from the runner back through the service, API, tracing semantics, storage rollups, and UI, need much more thought. The main platform has already been designed, refactored, and organized around stable semantic conventions. A quick interface change there can create inconsistent meanings, duplicate sources of truth, or hidden compatibility problems. Even when the local change looks small, it may make the wider system harder to reason about. For example, #5253 treats input tokens as an inclusive total and cache-read/cache-creation tokens as subcategories. This PR keeps input as non-cached and makes the cache fields additive siblings. Both can make the tooltip add up, but they define different contracts. We should agree on the contract before wiring one shape through the runner, SDK, tracing pipeline, API rollups, and frontend. The goal should be to reduce entropy with each change: make responsibilities clearer, preserve standard meanings, remove ambiguity, and leave fewer special cases behind. We should avoid changes that solve one visible symptom by adding more uncertainty to the underlying interfaces. I also recommend using the coding agent as a design partner before implementation. Ask it questions such as: “What are the best practices here?”, “What would a staff engineer do?”, “Who should own this field?”, “Is there an existing standard?”, and “How can this change reduce complexity across the whole system?” These conversations are useful for learning what a good design looks like, not just for producing code faster. The idea is valuable. I suggest aligning it with #5253 first, agreeing on the canonical usage shape and attribution rules, and then adapting this implementation to that design. |
2 participants
Context
Sending "hi" to an agent in the playground showed "Total 15.0K / Prompt 3 / Completion 14". The total is real (the agent harness sends its system prompt and tool schemas on every model call), but the breakdown was misleading: providers report cached prompt tokens separately from the regular input count, and we dropped that split everywhere. So Prompt + Completion never added up to Total. The trace drawer had the same problem.
Changes
The cache split (cache read / cache write) now flows end to end, and both token tooltips fold it into the displayed prompt count so the numbers add up.
cacheRead/cacheWritefrom the harness, stamps them on the agent span asgen_ai.usage.cache_read.input_tokens/gen_ai.usage.cache_creation.input_tokens, and carries them on the wire.AgentUsage.inputkeeps its harness semantics (non-cached tokens only); the cached portion travels in the new optional fields.AgentUsage(TS) andWireAgentUsage(Python) gained optionalcacheRead/cacheWrite. Golden fixtures and both contract tests updated.ag.metrics.tokens.*.cache_read/cache_creation, which the logfire OTLP adapter already ingests.cumulate_tokensintrees.pynow propagates the cache keys up the span tree alongside prompt/completion/total. Cache keys are written only when non-zero, so stored attributes for non-cache traces are unchanged.With real numbers from a local run:
Before:
Total 32.8K / Prompt 6 / Completion 115After:
Total 32.8K / Prompt 32.7K / Cached (read) 14.9K / Cached (write) 17.7K / Completion 115No DB migration. Span attributes are a JSONB blob; this only adds keys inside it. Old traces render as before.
Tests
cumulate_tokens(11 pass intest_trees.py).What to QA
invoke_agentspan, hover Tokens & Cost. Same breakdown as the playground.Preview
Before:
No way to understand where the 15l tokens costs
After:
Showing the correct token costs + the cached token
Trace drawer