Skip to content

feat(agent): the plot follows the site theme - #12505

Merged
MarkusNeusinger merged 5 commits into
mainfrom
feat/agent-plot-follows-site-theme
Nov 1, 2026
Merged

MarkusNeusinger merged 5 commits into
mainfrom
feat/agent-plot-follows-site-theme

Conversation

@MarkusNeusinger

@MarkusNeusinger MarkusNeusinger commented Nov 1, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Dark site, dark plot. This is the owner's decision of 2026-11-01, after user test E2 showed a bright plot card on the dark site. The first render of each plot version now uses the theme the site shows. The light and dark switch still renders the other theme in one click, and each version still takes one render.
  • How the theme travels. The page sends its effective site theme with every turn as POST .../messages?theme=light|dark. system mode resolves before it is sent. The BFF validates the value and forwards it. The agents service keeps it on the request ledger, and pipeline.run_theme picks the run's theme in this order:
    1. The theme the root passed, which means the user asked for it in the chat.
    2. On a change, the asked theme of the version being changed (CodeVersion.theme_asked, which the change inherits).
    3. The site theme of the turn.
    4. Without a page (adk web, the evals, a BFF deployed earlier), the previous version's theme, else light. This is the old behaviour.
  • Fan-out, prompt and mock. The fan-out's sessions use the same hook, so .plot_all() plots every library in the site theme. The root prompt and the tool description say what an omitted theme means now. The dev BFF mock follows the same rule.
  • --theme dark for the eval matrix (agents/evals/matrix.py, 13 lines; the lead's ask). Every turn of the run sends the site theme, as the page does. The stamp records it as theme. The model-free preflight stays light, because it only checks the renderer.

Decisions (decide-and-document)

  • Query parameter, not a body field. The BFF and the agents service both reject unknown body fields (extra="forbid"), and they deploy on separate triggers. A body field would make every turn a 422 whenever the page or the BFF deploys first. An unknown query parameter is ignored. Gemini Flash went through all six deploy orders: none fails, and the theme only takes effect once all three tiers are live. There is a precedent: GET .../artifacts?theme=.
  • Switching the site theme mid-session changes no version already shown. The next version follows the site theme at the time of its request, as the lead suggested.
  • A theme asked for in the chat holds for later changes of that plot, in both directions: dark on a light site, and light on a dark site. This keeps the old guarantee that "a dark plot the user asked for must not silently come back light". A new plot (.start_over()) follows the site theme again.
  • The theme toggle is a view, not a choice. Looking at v1 in light does not make v2 light.
  • Telemetry: no field carries the theme, so none is added.
  • Exported code: no change. The export's run line already names the rendered theme, and the code's own default stays light.

Open points

  • Dark first renders are not measured yet. The reviewer and the host gates are not calibrated on light renders only. reviewer.md takes one labelled render in either theme, and VQ-07 names both backgrounds. R1 to R3 and the G-probes are geometric or colour-share checks, and grounding reads the drawn texts. But every eval measurement so far renders light: ladder_lab, stress, score, and every committed matrix baseline. So the agent quality numbers say nothing yet about the dark first renders that dark-site users now get. This PR adds --theme dark to the matrix and runs no dark matrix. One dark smoke or full run, with --no-baseline, should happen before the public release.
  • A kept first-plot request (keep_plot_request) carries no theme. "Make it dark" typed before .confirm() is lost on a light site, as before this change.

Plan

N/A (lead brief of 2026-11-01)

Test plan

  • uv run ruff check ., uv run ruff format --check ., mypy api core agents: clean
  • pytest tests/unit/agents tests/unit/api -n auto: 20,683 passed. One render-cancellation test flaked once under xdist and passed alone; it is unrelated.
  • New tests:
    • run_theme precedence table.
    • Service flow:
      • a dark site gets a dark first plot, reviewed in dark;
      • a light site gets a light one;
      • a change follows the site switch;
      • an asked theme holds in both directions;
      • start over follows the site again;
      • the toggle does not decide the next version;
      • a turn without a site theme behaves as before;
      • an invalid theme is a 422.
    • BFF: the theme is forwarded as a query parameter, a turn without one forwards none, and an invalid value is a 422 with no upstream call.
    • Hook and fan-out: the turn's ?theme= follows the site theme at the start of the turn.
  • App: yarn lint, fm:check, type-check, test (1,353 passed) and build.
  • /verify-frontend with the BFF mock (Playwright, own tab), at both viewports:
    • Desktop, 1440 px: the dark site gets a dark v1 from one plot-dark.png, with no render call. The toggle to light renders on demand. After switching the site to light, the next change gets a light v2 (?theme=light; pixel #FAF8F1).
    • Mobile, 390 px: the dark site gives a dark first plot, and the light site gives a light one.
    • Console: only the known font CORS errors on a non-3000 port.
  • One live check of the local agents service: Gemini 3.8 Flash on eu, fake renderer, about 322k tokens, roughly $0.5 at list price. create_plot on ?theme=dark gave plot-dark.png. "make the markers bigger" on ?theme=light gave plot-light.png, so the real root left theme out of the change.
  • Gemini 3.8 Flash pre-check: "sound with changes", $0.15.
    • Applied: the mock now reads "light" as a request too, and three test gaps are closed (asked light on a dark site, start over, the toggle).
    • Not applied: a root eval asserting that theme stays out of refinements. The live check covered one case, and an eval case is out of scope.
  • Copilot (one round): approval recommended, no inline threads. All three notes in its overview are applied: the mock picks the current library's latest version, the tool text names theme='light', and the route tables name the 422.
  • Matrix flag: test_theme_dark_sends_the_site_theme_and_renders_dark, and the eval suite passes (429 tests). No live dark matrix was run.

https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8

MarkusNeusinger and others added 2 commits November 1, 2026 21:40
The owner's decision of 2026-11-01, after user test E2: on the dark site
the first plot rendered light, a bright card on a dark page. Each turn now
carries the theme the site shows, and a plot run renders in it unless the
user asked for a theme in the chat. Still one render per version; the
light and dark switch renders the other theme.

- The page sends `POST .../messages?theme=<light|dark>` (the effective
  theme, so `system` resolves first). A query parameter, not a body field:
  the BFF's and the agents service's bodies are strict, and an unknown
  parameter lets the page, the API and the service deploy in any order.
- `pipeline.run_theme` picks the run's theme: the one the root passed
  (the user asked), the asked theme of the version a change builds on
  (`CodeVersion.theme_asked`), the site theme of the turn, and without a
  page the previous version's theme, else light.
- The fan-out's sessions use the same hook, so they follow the same theme.
- The root prompt and the tool description say what an omitted theme now
  means; the BFF mock follows the same rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
From the Gemini Flash pre-check: the asked theme in both directions, a new
plot after an asked theme (`.start_over()` follows the site again), and the
theme toggle as a view that never decides the next version's theme. The BFF
mock now reads the word "light" as a request too, as the root does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
Copilot AI lite review requested due to automatic review settings November 1, 2026 20:48
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
@codecov

codecov Bot commented Nov 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The reviewed changes have no active blocking findings, with remaining feedback confined to documentation nits and a development-mock behavior concern.

No open issues 🎉

What changed in this PR

This pull request makes generated plots follow the site’s effective light or dark theme while preserving explicitly requested themes across refinements. Theme information is propagated from the frontend through the BFF to the agents service, with backward-compatible fallbacks for page-less callers.

Changes:

  • Sends ?theme=light|dark with agent turns and fan-out requests.
  • Adds theme precedence and persistence for new plots, refinements, toggles, and .start_over().
  • Updates the mock, tests, documentation, prompts, schemas, ledger, and changelog.
File Description Review findings
tests/​unit/​api/​test_agent_router.py Tests BFF theme forwarding, omission, and validation. —
tests/​unit/​agents/​runtime/​test_service_flow.py Covers site-theme rendering, requested-theme persistence, changes, toggles, and fallbacks. —
tests/​unit/​agents/​runtime/​test_pipeline.py Tests theme precedence. —
docs/​reference/​api.md Documents the messages endpoint’s optional theme query parameter. —
docs/​concepts/​agent-network.md Describes site-theme propagation and pipeline behavior. —
changelog.d/​agent-plot-follows-site-theme.md Adds the user-facing changelog entry. —
app/​src/​sections/​agent-chat/​FanOut.test.tsx Verifies fan-out turns use the page theme. —
app/​src/​pages/​AgentChatPage.test.tsx Updates request matching to tolerate query parameters. —
app/​src/​lib/​agent.ts Adds optional theme query construction for chat requests. —
app/​src/​hooks/​useAgentSession.ts Captures and sends the effective theme for each turn. —
app/​src/​hooks/​useAgentSession.test.ts Tests theme propagation and site-theme changes between turns. —
app/​scripts/​agent-bff-mock.mjs Mirrors site-theme handling and requested-theme precedence in the development mock. Moderate: Refinements can inherit the previous library’s theme instead of the current library’s theme; select the newest version belonging to the active library.
api/​routers/​agent.py Validates and forwards the theme query parameter to the agents service. —
agents/​main.py Accepts the theme query parameter and stores it on the request ledger. Nit: Document the 422 response for invalid theme values to match the validation behavior and BFF documentation.
agents/​anyplot/​tools/​session.py Updates the plot_pipeline tool description for omitted themes. Nit: Document both theme='light' and theme='dark' as supported explicit values.
agents/​anyplot/​services.py Records whether a version’s theme was explicitly requested. —
agents/​anyplot/​schemas.py Updates PipelineArgs theme documentation. —
agents/​anyplot/​prompts/​root.md Instructs the root agent on site themes and explicit light/dark requests. —
agents/​anyplot/​plugins/​ledger.py Adds the current site theme to request state. —
agents/​anyplot/​pipeline.py Implements theme precedence, persistence, rendering, and version storage. Nit: Update the related agents/anyplot/agent.py module documentation to distinguish theme_asked from the current turn’s site theme.

Review effort: Lite

MarkusNeusinger and others added 2 commits November 1, 2026 21:54
- The BFF mock's refinement changes the current library's latest version,
  as the service's `latest_rendered(library=)` does, not the session's
  latest of any library.
- The `plot_pipeline` description names `theme='light'` as well.
- The route tables name the `422` for a theme other than light or dark.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
The lead's ask of 2026-11-01: every committed matrix baseline renders
light, so nothing measures dark first renders yet. `--theme dark` sends
the site theme with every turn, as the chat page does, and the stamp
records it. The model-free preflight stays light: it checks only the
renderer. No dark matrix runs in this PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
@MarkusNeusinger
MarkusNeusinger merged commit a755ae4 into main Nov 1, 2026
16 checks passed
@MarkusNeusinger
MarkusNeusinger deleted the feat/agent-plot-follows-site-theme branch November 1, 2026 21:48
MarkusNeusinger added a commit that referenced this pull request Nov 1, 2026
Brings in #12505 and #12503. The one conflict, the messages row of
docs/reference/api.md, takes main's `?theme=` wording and keeps this
branch's summary clause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8
MarkusNeusinger added a commit that referenced this pull request Nov 1, 2026
## Summary
- Owner decision of 2026-11-01 after the user test: "Replies should be
short and to the point, no waffle, and follow-up questions only when
they really matter." `root.md` gets a "Replies" section. A reply leads
with the result in two or three sentences, asks a question only when the
turn cannot go on without the answer or the answer changes the data or
the plot materially, closes with no invitation, and acts on the user's
answer in the same turn. Brevity never drops a notice the data rules
require: the loader's "reads", "keeps time" and "converts" lines, data
changes with their counts, `requested_not_found`, binding warnings, tool
errors and limits. The step-1 summary no longer asks a question by
default and lists nothing the data does not have.
- The "what changed" notes are now written in the language the user
chats in and in plain words, without the code's own names such as
`load_user_data()`, without what stayed as it was, and a fix lists only
itself (`adapter.md`, and the self-review line in `pipeline.py`).
- The prep eval measures every reply: words, questions, and whether it
closes with a question. A closing question in a turn that was not
blocked, or a reply over 100 words, is a reply miss, gated at 10 %
(`--max-reply-misses`). A turn is blocked when the case is tagged
`ask_back`, a data call failed, or a required role is left without a
column. A case can send the page's `summarize_data` action, and four
summary cases join the set.

Decided and documented here:
- **"Offer to ..." in `root.md`:** the information stays, said as one
short statement, through one rule in "Replies" rather than an edit of
every passage. This keeps the diff away from the lines #12503 and #12504
change. Only `needs_attention` drops its "offer to fix one" outright:
the user asks when they want a fix.
- **Notes language (lead, after #12504 merged):** the reply language,
since the notes explain the plot to the user.
`AdaptRequest.reply_language` carries the session language only when it
differs from the plot language, fenced as `<reply_language>` after the
plot language. The adapter then writes `changes` in it and keeps the
plot text in the plot language. Without the field the two languages are
the same.
- **No server-side deduplication of notes:** the repeats in the test
were paraphrases from the self-review call, which no string comparison
catches. The prompt and the self-review line now address the cause, and
the final chat run showed no repeated note.
- **Length bound:** 100 words, about the five sentences `root.md` allows
at most.

## Measurements
Claude Haiku 5.5 on Vertex AI `eu`, $0.985 in all.

Seven scripted chats rebuilt from the user test (A1, A3, B1, B4 and
three summaries, two of them run twice), through `/v1` in process with
the deployed renderer, 20 replies per arm. The baseline arm ran on the
`origin/main` prompts.

| | main | this branch |
|---|---|---|
| Words per reply, mean (median, max) | 58.7 (55, 80) | 50.4 (52, 71) |
| Questions per reply | 0.75 | 0 |
| Replies that close with a question | 13 of 20 | 0 of 20 |
| Summaries: words, with a question | 60.6, 11 of 11 | 55.1, 0 of 11 |
| Plot replies: words | 50.2 | 37.0 |

- **A3, "The scores are out of 100.":** on main the root called no tool
and answered "If you want the value axis to run from 0 to 100, tell me".
The next plot kept the data's maximum. On this branch the root calls
`plot_pipeline` with "Set the value axis to run from 0 to 100 ...", and
the axis runs from 0 to 100.
- **B1 notes:** on main they were English under a German reply and named
`loadUserData()`. On this branch they are German, with no code names.

The prep eval (55 cases) ran with the same eval code on both arms:

| | main | this branch |
|---|---|---|
| Exact operation match | 53 of 55 | 55 of 55 |
| Reply words, median (max) | 69 (123) | 57 (121) |
| Questions per reply | 0.50 | 0.04 |
| Replies that close with a question | 20 of 52 | 2 of 52 |
| Reply misses (gate 10 %) | 21 of 52, FAIL | 3 of 52, PASS |

The main arm ran before the open-role rule joined the blocked
definition, so its reply-miss count is not comparable one to one. The
closing-question and question counts are.

Pre-PR Gemini 3.8 Flash check ($0.21, three findings):
- Applied: a statement of what is possible may close a reply, and only
an invitation may not. Applied: the policy test covers all seven loader
prefixes.
- Skipped: counting a question anywhere in the reply as a reply miss.
The brief defines the check as a closing question, and the report still
shows questions per reply.

Copilot review (one round, three findings):
- Applied: the true median for an even number of replies, with a test.
Applied: the golden adapter-prompt test is renamed for what it now pins.
- Skipped, as a false positive: the "high" `NameError` on `PrepCase`'s
return annotation. The module has `from __future__ import annotations`
and runs on Python 3.14, and its tests pass.

## Plan
N/A (brief from the lead session; owner decision of 2026-11-01)

## Test plan
- [x] `ruff check .`, `ruff format --check .`, `mypy api core agents`
- [x] `pytest tests/unit tests/integration -n 6`: 25,857 passed. New
tests cover the "Replies" rules and every protected notice, the
action-or-message case model, the reply measurement (closing question,
long reply, and blocked by `ask_back`, by a failed call or by an open
role), the reply gate, and a scripted summary run.
- [x] Live chat scenarios and the prep eval, as above.
- [x] After merging `origin/main` (#12505, #12503, #12504) and the
reply-language commit: `ruff`, `mypy api core agents`, and `pytest
tests/unit tests/integration -n 6` with 25,934 passed. Two flow tests
through `/v1` pin that a German plot under English replies gets its
notes in English and the reverse, and that no `<reply_language>` block
appears while the two languages agree.
- [ ] Not measured live: the last notes rule ("never what stayed as it
was") and the reply-language notes, since the $1 measurement budget was
spent ($0.985).

## Open points
- The server's own loader lines ("R reads ...") stay English in the
notes list, because the root reads them by their English prefixes.
- The regression matrix does not read replies, so plot-turn replies were
measured with a scratch driver only. Adding a reply measurement to
`matrix.py` would collide with #12503.
- Haiku still adds "confirm the data in the data panel" to about half of
the summaries: one short sentence, left as it is.

https://claude.ai/code/session_01H7HSkGyea1k2v293XiAMo8

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants