fix(obs): report token and spend metrics on every endpoint, not just chat and messages - #892
Conversation
…chat and messages
The follow-up half of AISIX-Cloud#1234. That fixed the request counters;
the token families had the same gap, in three different shapes:
aisix_llm_{input,output,total}_tokens_total chat + messages
aisix_llm_spend_micro_usd_total chat + messages
aisix_llm_tokens_by_client_total chat + messages + responses
aisix_tokens_consumed_total (legacy) chat ALONE
So a gateway that billed a customer for /v1/embeddings, /v1/rerank,
/v1/audio/* or /v1/images/generations reported none of those tokens, and
`/v1/responses` — the endpoint Codex talks to — reported them only on the
by-client series.
All of them now emit through `request_metrics::record_usage`, the
companion to the request emit added by #888. It takes the same
`Caller` / `Upstream` pair, so the token and request families agree by
construction and a query joining them lines up; it also means this adds
no series dimension the request families do not already carry.
`/v1/realtime` moves into `LLM_ENDPOINTS`. #888 held it out precisely
because it fed none of the token families, which would have inflated the
denominator of every tokens-per-request query. It now reports its session
tokens and, uniquely among the non-chat endpoints, a real cost — so that
reason is gone.
Two endpoints stay tokenless by nature: /v1/audio/speech (billed per
input character) and /v1/videos (per video). They count as requests and
contribute nothing here, which is why aggregate tokens-per-request is only
meaningful per endpoint.
`Caller::from_api_key_id` recovers the team / user / name triple from the
snapshot for the streaming emits, which run from a detached task or a Drop
guard holding only an `api_key_id: &str`. It resolves the same row the
auth extractor matched, so the labels are identical to `Caller::new`.
On the prior "intentional" scoping: the comment in responses.rs cited a
cross-API audit (#646-652) as the judgment behind keeping the per-key
family chat/messages-scoped. Those PRs are about UsageEvent attribution,
request overrides, failed-request events and guardrails — none of them
discusses the token metrics. The citation was inherited rather than
decided, the same shape as the docs that described #888's gap as if it
were the design.
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 43 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (11)
Comment |
1 participant
The follow-up half of api7/AISIX-Cloud#1234, flagged in that issue's closing comment. #888 fixed the request counters; the token families had the same gap — in three different shapes:
aisix_llm_{input,output,total}_tokens_totalaisix_llm_spend_micro_usd_totalaisix_llm_tokens_by_client_totalaisix_tokens_consumed_total(legacy)So a gateway that billed a customer for
/v1/embeddings,/v1/rerank,/v1/audio/*or/v1/images/generationsreported none of those tokens, and/v1/responses— the endpoint Codex talks to — reported them only on the by-client series.Implementation
All of them now emit through
request_metrics::record_usage, the companion to the request emit #888 added. It takes the sameCaller/Upstreampair, which matters for two reasons: the token and request families agree by construction (a query joining them lines up — pinned by a test), and it adds no series dimension the request families do not already carry, so the cardinality cost of this change is zero beyond the counters themselves.Caller::from_api_key_idrecovers the team / user / name triple from the snapshot for the streaming emits, which run from a detached task or a Drop guard holding only anapi_key_id: &str. Threading three labels down every dispatch signature to reach them would have been a lot of plumbing; the id resolves back to the row the auth extractor matched, so the labels come out identical toCaller::new./v1/realtimemoves into the LLM tier#888 held it out of
LLM_ENDPOINTSfor one stated reason: it fed none of the token families, so counting it as an LLM request would inflate the denominator of every tokens-per-request query. It now reports its session tokens and — uniquely among the non-chat endpoints — a real cost, since realtime is the only one that resolves pricing at emit time. That reason is gone, so it belongs with the rest.Endpoints that stay tokenless
/v1/audio/speech(billed per input character) and/v1/videos(per video) report no tokens by nature. They count as requests and contribute nothing here — which is why aggregate tokens-per-request is only meaningful perendpoint, never summed across all of them. Noted in the module docs.On the prior "intentional" scoping
responses.rscarried a comment saying the per-key family "intentionally stays chat/messages-scoped (cross-API audit #646-652 judgment)". I checked those PRs before overriding it: #646/#647 are per-PK attribution tags, #648/#649 per-PK request overrides, #650 failed-request UsageEvents, #651 routing retries, #652 applied_guardrails. None of them discusses the token metrics. The citation was inherited rather than decided — the same shape as the docs that described #888's gap as if it were the design. Flagging it explicitly in case someone knows a reason the audit does not record.Tests
tests/e2e/src/cases/token-metrics-endpoint-coverage-e2e.test.ts, verified failing before and passing after:/v1/responsesreports on every token family including the legacy series, with the real counts (delta-measured, so the propagation probe's own tokens don't mask a wrong value)./v1/embeddingsreports input tokens and the by-client series — and no output-token series, since embeddings produce none.Caller/Upstreambuys.Full DP E2E suite passes (184 files, 495 tests); workspace unit tests pass.