Skip to content

feat(agent): per-day numbers on /debug, a service pause refusal - #12518

Merged
MarkusNeusinger merged 7 commits into
mainfrom
feat/debug-daily-usage
Nov 2, 2026
Merged

MarkusNeusinger merged 7 commits into
mainfrom
feat/debug-daily-usage

Conversation

@MarkusNeusinger

@MarkusNeusinger MarkusNeusinger commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • /debug gets a table of the last 14 UTC days (new chats, plots, cost, fan-out cost, limit hits by code, p95 queue wait), read read-only from the BigQuery telemetry.
  • /debug gets a live line for today from the agents service: the service's spend against its day budget and how many users are at their own limits, counts only, for the instance's lifetime.
  • A spent service budget now has its own refusal, service_paused, apart from a user's own limit. It names the reset at 00:00 UTC, the page adds the user's own time, and GET /v1/status tells the chat before the user types.

Plan

N/A: owner request of 2026-11-02 ("What happens when the $10 daily limit is reached? We definitely need a few more numbers in debug: dollars per day, how many new chats."). This takes over the work of a killed session.

Decisions

  • Window. The brief asked for a window floored to 10 minutes. The table's window is floored to the day instead, so its oldest row is a whole day. The query text and parameter stay the same all day, and the API keeps each answer for 10 minutes as the card does.
  • budget counts too. Limit hits count budget as well as the codes the brief named, because a user's own spent token budget is a limit hit too. locked_out is a safety lockout, so it stays in the card's refusal list.
  • Source handling. live lines fill every column, fanout lines fill only a separate fan-out cost column (they spend the same day budget), and harness lines are never read.
  • A user with both budgets spent keeps budget. The pause text says their own limit is untouched, which would be false for them.
  • The dollars carry eu's 10 %. The live line prices weighted tokens at the main model's input list price in the service's location, so the default of 100 M tokens reads $11.00, not $10.00.
  • The refusal text stays in UTC on the server. The server does not know the user's time zone, so the page adds the local time next to it. /debug shows both.
  • A pipeline halted mid-run keeps plot.reason = budget. That is the plot event's closed failure reason, not a refusal, and it reads "the usage limit is reached", which is true of either budget. The root's next call then sends service_paused through day_refusal.

Flash pre-check

  • First pass: stopped at its token limit without a verdict, after confirming that daily_sql prices and deduplicates as 05_prices_cost.sql does.
  • Second pass, narrowed to the remaining questions: "sound with changes", 3 findings, none applied. The pipeline finding claims the refusal stays budget, which the code does not do (see above). The double-count finding refutes itself in its own text. The client-side reset time is always the next 00:00 UTC, so passing the server's value would change nothing for a correct clock. It found no gap in the consumers of the new code, no exposure beyond counts and times, and no bug in the pause note.

Copilot review

Three findings, one applied:

  • Applied: the page's pause note claimed "your own limit is not affected", which /status cannot know. The note now says only that the chat is paused and until when. The server's refusal keeps the sentence, because day_refusal sends it only to users with their own budget left.
  • Skipped: the fragment's PR references. /pull_request step 7 appends them and the changelog check passes.
  • Done after feat(agent): admin lane, final run limits and anonymous limit telemetry #12507 merged: the admin tier in snapshot and day_refusal; see the next section.

Merged with #12507

  • day_refusal takes the caller's tier. An admin has no own day budget, so the route, the scope guard, the Budget plugin and the paste routes all answer an admin service_paused during a pause, never budget.
  • The live line counts an admin only at the caps that bind every tier (pastes, conversions, strikes). The usage book holds no tier, so it learns who came in the admin tier from _stamp, which every spending route passes (UsageBook.mark_admin).
  • The queue-wait column reads pipeline_run.queue_wait_ms from the line itself, not from the pipeline_runs view, so it fills from feat(agent): admin lane, final run limits and anonymous limit telemetry #12507's deploy on, whether or not the view is applied again, and reads — on every earlier day. A test pins the field name to the telemetry model.

Owner steps

None. No view changes, no setting changes, no writes to BigQuery.

Verification

  • BigQuery, read-only: per-day cost matches the cost view for every day of the window (10 day and source rows, to the rounding). read_daily over the real REST path decodes 14 rows.
  • Scan: one load processes 62,676 bytes in 9 partitions in 0.6 s (a real uncached job; BigQuery bills its 10 MiB minimum). Dry runs of the same query: 0 bytes for a window in the future, 50,674 for 3 days, 62,676 for the 14 days, 70,906 for the whole bucket.
  • Browser, own headless playwright-core, 1440x900 and 390x844, light and dark, normal and paused, with the real rows: no console errors, no page overflow. The mobile check found the limit-hits cell wrapping and a mixed UTC and local clock; both are fixed.
  • Gates after the feat(agent): admin lane, final run limits and anonymous limit telemetry #12507 merge: ruff, ruff format, mypy, 26,034 unit tests, and the app's lint, fm:check, type-check, 1,379 tests and build.

Test plan

  • Unit tests: the query builder, the aggregates, the live status payload, the pause refusal against the personal one, and the paste's 429.
  • Per-day cost matches the cost view in BigQuery.
  • Dry run shows that one load scans only the window's partitions.
  • /debug and the chat's pause note in the browser, both viewports and both themes (the note's shorter final wording is covered by unit tests).
  • After the deploy: open /debug and check that the table and the live line load.

https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8

MarkusNeusinger and others added 4 commits November 2, 2026 00:49
…ice pause refusal

Work in progress taken over from a killed session: the per-day table, the
agents service's live budget view and the `service_paused` refusal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
…refusal

Takes the pause note's clock to UTC plus the reader's own time on /debug, keeps
the limit hits on one line on a phone, counts no chat for a line without a
session, drops an unused `reserve`, and documents all of it: the /debug section,
the API reference, the Plausible reasons and the concept doc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
Copilot AI lite review requested due to automatic review settings November 2, 2026 05:57
@codecov

codecov Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.62687% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
app/src/pages/AgentChatPage.tsx 92.30% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Active moderate and critical findings remain in the token-limit bookkeeping, service-paused messaging, and changelog fragment, so the changes require correction before approval.

3 open issues
What changed in this PR

This pull request adds per-day agent usage visibility to /debug and distinguishes service-wide budget exhaustion from individual user limits across the agents service, API, telemetry, documentation, and frontend.

Changes:

  • Adds a 14-day UTC usage table with chats, plots, costs, limit hits, and queue-wait percentiles.
  • Adds live service-budget and per-user-limit counts through the debug and budget endpoints.
  • Introduces the service_paused refusal, status reporting, localized reset-time messaging, telemetry support, documentation, and tests.
File Description
tests/​unit/​api/​test_agent_usage.py Tests daily usage aggregation, caching, SQL construction, and authorization.
tests/​unit/​api/​test_agent_router.py Tests the live budget proxy and service_paused mapping.
tests/​unit/​agents/​runtime/​test_policy.py Tests localized service-pause refusal text.
tests/​unit/​agents/​runtime/​test_day_budget.py Tests budgets, pricing, refusals, telemetry, status, and budget responses.
docs/​reference/​plausible.md Documents the new analytics refusal reason.
docs/​reference/​api.md Documents status, budget, daily usage, and pause-related API behavior.
docs/​reference/​agent-telemetry.md Documents daily telemetry columns, sources, costs, and live budget semantics.
docs/​concepts/​agent-network.md Documents the service-wide budget and related routes.
changelog.d/​debug-daily-usage.md Adds the changelog fragment for the feature.
app/​src/​sections/​debug/​format.ts Adds currency formatting for debug usage values.
app/​src/​sections/​debug/​AgentUsageSection.tsx Integrates live and per-day usage views into the debug section.
app/​src/​sections/​debug/​AgentDailyUsage.tsx Implements live budget and daily usage table rendering.
app/​src/​sections/​agent-chat/​messages.ts Adds reset-time and service-pause message helpers.
app/​src/​sections/​agent-chat/​messages.test.ts Tests pause and reset-time message formatting.
app/​src/​sections/​agent-chat/​ChatThread.tsx Displays local reset-time information for paused refusals.
app/​src/​sections/​agent-chat/​ChatThread.test.tsx Tests service-pause rendering in chat threads.
app/​src/​pages/​DebugPage.test.tsx Tests daily rows, live budget output, and loading states.
app/​src/​pages/​AgentChatPage.tsx Displays the service-pause notice before message submission.
app/​src/​pages/​AgentChatPage.test.tsx Tests status-driven pause notices and reset handling.
app/​src/​lib/​agent.ts Extends the agent status response type.
app/​src/​hooks/​useAgentSession.ts Tracks service-pause refusals and reset metadata.
app/​src/​hooks/​useAgentSession.test.ts Tests service-pause refusal state and telemetry behavior.
api/​routers/​debug.py Adds the daily usage debug endpoint.
api/​routers/​agent.py Adds the admin-gated budget proxy and pause error mapping.
api/​agent_usage.py Builds and reads the BigQuery daily usage query and response models.
agents/​stream.py Documents the expanded SSE refusal contract.
agents/​main.py Adds status and budget routes and applies service-pause handling.
agents/​anyplot/​telemetry.py Adds service_paused to telemetry codes and mappings.
agents/​anyplot/​prompts/​refusals.yaml Adds localized service-pause refusal messages.
agents/​anyplot/​plugins/​scope_guard.py Applies service-aware daily refusal classification.
agents/​anyplot/​plugins/​ledger.py Adds global budget tracking, snapshots, pricing, and refusal classification.
agents/​anyplot/​plugins/​budget.py Propagates service-pause decisions during model-budget checks.

Review effort: Lite

Comment thread changelog.d/debug-daily-usage.md Resolved
Comment thread agents/anyplot/plugins/ledger.py Resolved
Comment thread app/src/pages/AgentChatPage.tsx Resolved
MarkusNeusinger and others added 2 commits November 2, 2026 07:08
`/status` reports the pause for everyone, so the page cannot know that a user
still has their own budget; one who does not reads a false claim and then gets
`budget`. The server's refusal keeps the sentence, because `day_refusal` sends
it only to users with their own budget left.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
Carries the caller's tier into the pause: `day_refusal` takes it, so an admin,
who has no own day budget, hears `service_paused` from the route, the scope
guard, the Budget plugin and the paste routes. The usage book learns who came in
the admin tier from `_stamp`, which every spending route passes, and the live
view counts an admin only at the caps that bind every tier. The per-day table's
queue wait reads `pipeline_run.queue_wait_ms` from the line itself, so it fills
from #12507's deploy on, before the `pipeline_runs` view is applied again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
@MarkusNeusinger
MarkusNeusinger merged commit 2dee0a6 into main Nov 2, 2026
16 checks passed
@MarkusNeusinger
MarkusNeusinger deleted the feat/debug-daily-usage branch November 2, 2026 07:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants