[MarkusNeusinger]

Summary

• The owner's run limits (decisions of 2026-11-01). The run queue runs 5 runs at once for the whole instance (AGENT_RUN_CONCURRENCY, was 1) and starts at most one every 10 s, in every lane (AGENT_RUNS_PER_MINUTE 6, with the starts spaced 60/6 s apart, RunQueue.start_gap_s), so it holds 60 waiting entries. Lanes differ only in their per-user cap and order: public 2 (AGENT_USER_MAX_RUNS, was 3), admin 5 (AGENT_ADMIN_USER_MAX_RUNS), and premium stays reserved and goes first. A public user's day is 50 runs and 10,000,000 weighted tokens (about $1). The global budget is 100,000,000 (about $10) and binds admins too. Admins are exempt from the per-user day caps and the message minute; the attack strikes bind everyone. The values are the settings.py defaults, the pinned production values, and Cloud Run --concurrency rises to 80 for the 60 waiting streams.
• Who is admin: X-Anyplot-Tier: admin, set in one place of the BFF (agent_context) from the identity its admin gate verified, never from the client's source: fanout. The agents service trusts it only behind check_caller. A missing or other value is public and never a 422, so the API and the agents service deploy in either order. /v1/status reports the caller's own user_max_runs and tier, and the panel reads "at most 5 at a time".
• The fan-out survives the cap. The admin's cap counts every window, so a run_limit in the panel is no failure (nextFanOut in fanoutRules.ts): the library goes back to waiting and the panel shrinks to the runs the server took. A freed place or a 15 s retry tries one library again, and a library gives up only after 30 minutes, which its card says. A single chat's run_limit message is unchanged.
• Anonymous limit telemetry on pipeline_run: lane, queue_wait_ms, queue_position, user_runs_today (1, 2_5, 6_10, 11_20, 21_30, 31_40, 41_plus) and user_tokens_today (lt25, 25_50, 50_75, 75_90, 90_100, over, against the public budget for every tier). They are closed sets and bounded integers, with no user id, no hash and no key. Limit refusals stay in chat_turn.refusal and chat_turn.error; the admin's land there too (tested). The views SQL is updated, and a test pins every model field to its view column. The views are not applied in BigQuery: that write needs the owner's OK after the merge.
• Also:
  • The gates run in a worker thread.
  • Claude root, adapter and reviewer retry 4 times instead of 2; the judge stays at 0.
  • A 429 or 529 that outlasts the retries ends a run as busy, and pipeline_result names the innermost error class.
  • pipeline_render logs the agents-side render slot wait.

Live measurement (local stack, real renderer, Claude Haiku 5.5 in eu, final values)

One 15-library fan-out of scatter-basic as admin, by scripted HTTP through the local BFF, 5 at a time from /status:

	First design (355 s run, superseded)	Final values
Wall time	355 s	402 s (6.7 min); before this PR about 20 min
Result	15 of 15 ok	15 of 15 ok
capacity / run_limit errors	0 / 0	0 / 0
429s	none	none (Vertex shows only 200 for Haiku eu in the window)
Per library	median 89 s, max 136 s	median 82 s, max 191 s (muix, the last to start)
Queue wait (queue_wait_ms)	0 s	p50 1.9 s, max 39 s: the 10 s gap between starts
Render slot wait, agents side (slot_wait_s)	p50 6.9 s, p95 24.0 s	p50 0.0 s, p95 7.7 s, max 17.4 s
Render slot wait, renderer telemetry	p95 0.001 s	(the agents' slot serializes first)
Cost	3.05 M weighted (about $0.30)	3.12 M weighted (about $0.31)

The 10 s gap costs about 40 s at the start of a fan-out, and it spreads the renders: the p95 render slot wait drops from 24 s to 7.7 s. All 15 pipeline_run lines carried lane=admin, and none contained an id.

Decisions taken in the brief's scope (decide and document)

• "1 start every 10 s" is a minimum gap between starts (start_gap_s = 60 / AGENT_RUNS_PER_MINUTE), not just a sliding window of 6. The window alone would let a fan-out start five runs together, and their renders would meet in the one render slot. One setting stays the knob, and the capacity formula gives 60.
• The telemetry fields sit on pipeline_run, because user_runs_today is a pipeline-run ordinal and the queue entry is the run's. A not_ready run carries lane only.
• Fan-out retry policy: shrink to the runs the server took, then one retry per freed place or per 15 s, and give up after 30 minutes. This keeps three panels from burning their libraries without hammering the root model. A simulation of three 15-library panels sharing a cap of 5 finishes all 45 plots in 13.5 to 18 minutes with no library lost.
• Strikes still bind admins. Pastes and conversions keep their per-user limits (memory and sandbox time, not model spend). The fan-out's copies bind no admin: three windows copy 45 datasets within a minute, past the 30 a minute and 150 a day, and the store's byte cap still refuses a copy it has no room for.
• A root call refused for quota or overload stays capacity, and the page's capacity text now names that cause.

Open points

• anyplot-api concurrency (40, one instance). Every chat stream passes through the BFF. At the queue's new capacity, a full queue could hold about 65 BFF streams and crowd out the website's API. Today the admin panel opens at most 5 per window. Before the public launch, raise the API's concurrency or route the chat through its own service. This touches the site's infrastructure, so I left it out of this PR.
• A public user's render can wait behind a fan-out's renders. That is now lower (p95 7.7 s), but a public-first render slot would remove it.

Plan

N/A (owner-approved plan of 2026-11-01 and the owner's follow-ups of the same day).

Test plan

• ruff check, ruff format --check, mypy api core agents
• pytest tests/unit tests/integration (25,900+ passed):
  • the start gap and the shared lanes;
  • TestFanOut: 15 sessions of one admin at the new values all start, none refused, also beside three public users;
  • tier trust, including the production caller check, the exemptions, and an admin's strike lockout;
  • the status payload, the limit telemetry's buckets and validation, and that no new field can carry an id;
  • that an admin's run_limit lands in chat_turn, and the view-column pin;
  • busy versus error, the gates in a worker thread, and the retries.
• App: yarn lint, fm:check, type-check, test (1,362 passed; nextFanOut, the three-panel simulation, and a component test where a refused library waits and runs at the next freed place), build
• Live fan-out above. The panel was checked earlier in the browser ("at most 5 at a time", desktop and mobile, light and dark, no console errors).
• Gemini 3.8 Flash pre-check:
  • First design: 4 findings fixed.
  • Follow-ups (diff against c5ccfa1): the first run stopped at its output cap with no defect found so far; the rerun said "sound", and its one low finding (.catch on plotTurn) is fixed.
• Copilot round on the first design: 2 medium findings fixed and resolved. No second round, by the rule for medium findings.

https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8

[Copilot]

🟡 Changes recommended

Two moderate active findings remain: Cloud Run concurrency can reject admitted streams before the queue’s documented capacity response, and model-service exhaustion is misreported as queue saturation.

2 open issues

• Admin stream capacity can exceed Cloud Run concurrency · New
• Model busy errors are misreported as queue capacity errors · New

What changed in this PR

This pull request adds a trusted administrator execution lane for agent fan-outs while preserving public-user limits. It also improves model-busy handling, render telemetry, frontend status messaging, deployment capacity, documentation, and test coverage.

Changes:

• Adds tier-aware queue lanes, quotas, daily-budget exemptions, and BFF-to-agents tier propagation.
• Classifies exhausted model retries as busy, increases Claude retries, and moves render gates to worker threads.
• Updates fan-out UI behavior, status reporting, Cloud Run concurrency, documentation, changelog, and tests.

Review findings:

• Moderate (2 votes): agents/cloudbuild.yaml may admit more streams than Cloud Run concurrency can handle; aggregate admin waiting streams or increase capacity accordingly.
• Moderate (2 votes): Some pre-pipeline model 429/529 failures still surface as capacity rather than busy; preserve the distinction in the stream error and UI.
• Nit (1 vote): Clarify in docs/concepts/agent-network.md that the global token budget is only the admin’s remaining daily model-related cap.
• Nit (1 vote): Rename the grammatically incorrect test_an_admins... test.

File	Description
tests/​unit/​api/​test_agent_router.py	Verifies trusted admin-tier headers and protection from client-supplied values.
tests/​unit/​agents/​test_settings.py	Tests admin lane settings and the increased global budget.
tests/​unit/​agents/​test_image.py	Updates the expected Cloud Run concurrency setting.
tests/​unit/​agents/​runtime/​test_telemetry.py	Updates telemetry tests for tier-aware message handling.
tests/​unit/​agents/​runtime/​test_run_queue.py	Tests lane limits, capacity, ordering, isolation, and per-user behavior.
tests/​unit/​agents/​runtime/​test_admin_lane.py	Covers admin tiers, exemptions, busy failures, lane isolation, and worker-thread gates.
tests/​unit/​agents/​runtime/​fakes.py	Allows model fakes to raise configured exceptions.
docs/​reference/​api.md	Documents tier headers, status fields, limits, and busy errors.
docs/​concepts/​agent-network.md	Describes admin lanes, quotas, rendering contention, and deployment capacity.
changelog.d/​agent-admin-lane.md	Records the admin lane and related model retry changes.
app/​src/​sections/​agent-chat/​messages.ts	Adds busy and dynamic run-limit messaging.
app/​src/​sections/​agent-chat/​fanoutRules.ts	Displays detailed plot failure messages in fan-out cards.
app/​src/​sections/​agent-chat/​fanoutRules.test.ts	Tests busy failure presentation.
app/​src/​sections/​agent-chat/​FanOut.tsx	Documents caller-specific fan-out concurrency.
app/​src/​sections/​agent-chat/​ChatThread.test.tsx	Tests dynamic run-limit and busy messaging.
app/​src/​pages/​AgentChatPage.tsx	Uses the caller’s reported run limit for fan-out.
app/​src/​lib/​agent.ts	Extends agent status and stream types with tier-aware fields.
api/​routers/​agent.py	Asserts and forwards the verified admin tier.
agents/​README.md	Documents admin settings, limits, and deployment concurrency.
agents/​main.py	Applies tiers to status, ledgers, queue lanes, and busy errors.
agents/​cloudbuild.yaml	Raises agents service Cloud Run concurrency.
agents/​anyplot/​tools/​session.py	Applies tier-aware data-run limits.
agents/​anyplot/​theme_render.py	Moves theme-render gates off the event loop.
agents/​anyplot/​telemetry.py	Adds busy as a pipeline failure reason.
agents/​anyplot/​settings.py	Adds admin lane settings and raises the global token budget.
agents/​anyplot/​schemas.py	Adds busy to the public failure-reason schema.
agents/​anyplot/​run_queue.py	Implements independent admin, premium, and normal queue pools.
agents/​anyplot/​render/​serial.py	Records render-slot wait time.
agents/​anyplot/​render/​contract.py	Carries render-slot wait telemetry.
agents/​anyplot/​plugins/​scope_guard.py	Applies tier-aware daily budget checks.
agents/​anyplot/​plugins/​ledger.py	Implements admin exemptions and tier-aware usage accounting.
agents/​anyplot/​plugins/​budget.py	Applies tier-aware pipeline-run limits.
agents/​anyplot/​pipeline.py	Runs gates in worker threads and classifies model overloads as busy.
agents/​anyplot/​models.py	Adds Claude retry configuration and model-busy detection.

_{Review effort: Lite}

[codecov]

Codecov Report

❌ Patch coverage is 98.42932% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
app/src/sections/agent-chat/FanOut.tsx	89.28%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!