diff --git a/agents/README.md b/agents/README.md index 703f580622d..849ac8c81ae 100644 --- a/agents/README.md +++ b/agents/README.md @@ -451,6 +451,7 @@ Before the first case the harness renders the catalogue file of the first case o | `--max-attempts N` | Adapter attempts per run, 1 to 3 (`AGENT_MAX_ATTEMPTS`, 2 by default), so 1 measures the first attempt alone and 3 a second repair round; the report's stamp records the value as `max_attempts` | | `--thinking-budget ROOT,ADAPTER` | Claude only: adaptive thinking for the root and the adapter, each value (0 to 8,192, 0 off) added to that agent's output cap (`AGENT_ROOT_THINKING_BUDGET`, `AGENT_ADAPTER_THINKING_BUDGET`); the stamp records it as `thinking_budget`, the summary names it, and the thinking tokens are priced as output | | `--budget-usd X` | Stop before a case that could take the estimated list-price cost past X (the spend so far plus the costliest case so far), or once the cost has passed X (exit code 3) | +| `--theme light\|dark` | The site theme every turn sends (`?theme=`), as the chat page does, so `dark` measures dark first renders; without it, no theme is sent and the plots render light, as in every committed baseline. The stamp records it as `theme`; compare a dark run with `--no-baseline` or a dark baseline. The preflight stays light | | `--out DIR` | Output directory, `agents/evals/reports/` by default (git-ignored) | | `--baseline FILE`, `--no-baseline` | The report to compare with; by default `evals/baselines/claude-haiku-5-5.json` when it exists | | `--tolerance F` | The pass-rate drop that counts as a regression, 0.05 (5 percentage points) by default | diff --git a/agents/anyplot/pipeline.py b/agents/anyplot/pipeline.py index 2a816d61dda..b6908cbef0c 100644 --- a/agents/anyplot/pipeline.py +++ b/agents/anyplot/pipeline.py @@ -3,10 +3,10 @@ `run_pipeline` is the single node of the `plot_pipeline` workflow (`tools/session.py`). It reads spec, library, dataset and bindings from server-set session state, never from its input: `PipelineArgs` carries only `change_request`, `base` and `theme` (the one -theme to render; omitted, a change keeps the previous version's theme and a new plot -is light). The session language is read once per run, so the adapter, which writes the -plot's own text in it, and the reviewer, which judges that text against it, share one -value; each prompt gets it fenced as ``. It runs at most +theme to render; omitted, the theme `run_theme` picks, the site's unless the user +asked for one). The session language is read once per run, so the adapter, which +writes the plot's own text in it, and the reviewer, which judges that text against it, +share one value; each prompt gets it fenced as ``. It runs at most `AGENT_MAX_ATTEMPTS` attempts, two by default (the first and one repair round; the eval harness may set one or three): @@ -29,7 +29,7 @@ literal text the plan replaces, up to `MAX_NEW_LITERAL_CHARS`; 3. **render**: `normalise`, the loader substitution (the runtime's `loader_block`, typed from the parser's dtypes and the column kinds), then the one - theme the call asks for (`PipelineArgs.theme`, light by default) through the + theme of the run (`run_theme`: the one the user asked for, else the site's) through the render backend, and the host gates (`render/gates.py`) on exactly that theme; a failed render's summary is the runtime's error grammar over the code that ran (`RuntimeAdapter.error_summary` with the run form, so Julia's names the working form's @@ -415,6 +415,8 @@ class Run: view: SessionView dataset: StoredDataset theme: Theme = "light" + theme_asked: bool = False + """The user asked for `theme` (`run_theme`); the shipped version keeps it as `CodeVersion.theme_asked`.""" language: str = DEFAULT_LANGUAGE """The session language, read once per run: the adapter writes the plot text in it, the reviewer judges it.""" attempts: int = 0 @@ -1249,6 +1251,7 @@ def _store_version(ctx: Context, services: Services, run: Run, result: PlotResul title=shipped.plan.title, library=run.view.library, theme=run.theme, + theme_asked=run.theme_asked, language=run.language, requests=standing_requests([*run.user_requests, run.change_request]), themes={run.theme: ThemeRender("needs_attention", PADDED_REASON) if shipped.padded else ThemeRender("ok")}, @@ -1387,17 +1390,23 @@ def _reraise_unless_schema(exc: Exception) -> None: logger.warning("sub-agent answer failed its schema: %s", type(exc).__name__) -def _default_theme(services: Services, session_id: str, library: str, base: str) -> Theme: - """An omitted theme keeps the previous version's theme on a change; a new plot is light. +def run_theme(asked: Theme | None, site: Theme | None, previous: CodeVersion | None) -> tuple[Theme, bool]: + """The one theme a run renders, and whether the user asked for it (`CodeVersion.theme_asked`). - The root may leave `theme` out of a refinement call, so a dark plot must not - silently come back light. + In order: the theme the root passed (`PipelineArgs.theme`), which the user asked for in the chat; on a + change, the theme of the version it changes (`previous`) when the user asked for that one, because the root + leaves `theme` out of a refinement and a dark plot asked for must not silently come back light; the site + theme the page sent with the turn (`RequestLedger.site_theme`; owner decision of 2026-11-01: a user on the + dark site gets a dark plot, the other theme one click away on the toggle), so a change after a switch of the + site theme follows the switch; without a page (`adk web`, the evals) the version it changes, else light. """ - if base == "previous": - previous = services.versions.latest_rendered(session_id, library=library) - if previous is not None: - return previous.theme - return "light" + if asked is not None: + return asked, True + if previous is not None and previous.theme_asked: + return previous.theme, True + if site is not None: + return site, False + return (previous.theme if previous is not None else "light"), False @node(name="run_pipeline", rerun_on_resume=True) @@ -1434,11 +1443,17 @@ async def run_pipeline(ctx: Context, node_input: PipelineArgs) -> AsyncGenerator return # Taken only once the run has its slot: a run that never starts leaves the user's note for the next one. node_input = take_pending_request(ctx.state, node_input) - theme = node_input.theme or _default_theme(services, ctx.session.id, view.library, node_input.base) + previous = ( + services.versions.latest_rendered(ctx.session.id, library=view.library) + if node_input.base == "previous" + else None + ) + theme, theme_asked = run_theme(node_input.theme, ledger.site_theme, previous) run = Run( view=view, dataset=dataset, theme=theme, + theme_asked=theme_asked, language=view.language, change_request=node_input.change_request, expected_texts=list(node_input.expected_texts), diff --git a/agents/anyplot/plugins/ledger.py b/agents/anyplot/plugins/ledger.py index e439f6136de..ac5113a8fc9 100644 --- a/agents/anyplot/plugins/ledger.py +++ b/agents/anyplot/plugins/ledger.py @@ -44,6 +44,7 @@ from datetime import UTC, datetime from typing import Any, Literal +from ..schemas import Theme from ..settings import AgentSettings @@ -158,6 +159,9 @@ class RequestLedger: """The operation names of every `prepare_data` call that changed the data (closed: `telemetry.PREP_OPS`).""" reply_question: bool = False """The root's last reply sent to the page ends with a question mark (`stream.Translator`); the text is never kept.""" + site_theme: Theme | None = None + """The theme the page showed the site in when the user sent the turn (`/messages?theme=`), which a plot run takes + when neither the user nor the version it changes asked for one (`pipeline.run_theme`); None without a page.""" def book_step(self, agent: str, *, model: str | None, usage: dict[str, int], cached: int) -> None: """Add one model call to the agent's step usage (`usage` as `usage_breakdown` counts it).""" diff --git a/agents/anyplot/prompts/root.md b/agents/anyplot/prompts/root.md index 62492de8fda..37123b956c0 100644 --- a/agents/anyplot/prompts/root.md +++ b/agents/anyplot/prompts/root.md @@ -44,7 +44,7 @@ The session has one language for your replies and for the text in the plot. The - When the data is not confirmed yet (or roles are still unbound) and this library has no plot yet, a typed message that asks for something about the plot ("Setz den Titel auf ...", "log scale on y") is the first plot's request: call `keep_plot_request` with it as `change_request` (and `expected_texts`), which keeps it without running anything. Its `request_kept` `true` means the next `.create_plot()` draws it: say that, and what the user still has to do (`.confirm()`, or the open roles). - Only a `request_kept` `true` from `keep_plot_request` or `plot_pipeline` lets you say that a request is noted, kept or will be used. The "Create plot" action runs `plot_pipeline` with no arguments, so a request only you remember never reaches the plot; `request_kept` `false` or no answer means it is not kept: say so plainly. - `expected_texts`: when the user wrote out a text or a number format the plot must show (a title, a legend entry, an axis title, a unit, a format example such as "6,5 Mio." or "12 %"), list each one, verbatim as the user wrote it (never translated), at most five of at most 120 characters. The server checks the render's drawn texts for each. Leave it out when the user named none; never list a text the user wants removed or replaced, only the new one. - - `theme` is `"light"` or `"dark"`. Leave it out unless the user asks for a theme: a new plot is then light, and a change keeps the theme of the latest version. Set `"dark"` only when the user asks for a dark plot or a dark background, and `"light"` when they ask to go back to light. + - `theme` is `"light"` or `"dark"`. Leave it out unless the user asks for a theme: the plot is then rendered in the theme the user's page shows, and a change keeps a theme the user asked for earlier. Set `"dark"` only when the user asks for a dark plot or a dark background, and `"light"` when they ask for a light one or to go back to light. - When the user only wants to see a finished plot in the other theme, do not call `plot_pipeline`: tell them to use the light and dark switch on the result, which shows the other theme without changing the plot. - When a tool answers `not_ready`, tell the user what is missing: a dataset, bindings for the roles it names, or their confirmation of the data (`data_unconfirmed`: the user checks the preview and the bindings in the data panel and presses `.confirm()` there; you cannot confirm for them). diff --git a/agents/anyplot/schemas.py b/agents/anyplot/schemas.py index 0214b764ef1..a57742eee47 100644 --- a/agents/anyplot/schemas.py +++ b/agents/anyplot/schemas.py @@ -297,11 +297,11 @@ class Binding(_ServerContract): class PipelineArgs(_ServerContract): """The `plot_pipeline` tool input. Spec, library and dataset come from server state. - `theme` is the one theme the run renders and the reviewer sees. Omitted (None), a - change on `base="previous"` keeps the previous version's theme and a new plot is - light; the root passes `dark` only when the user asks for a dark plot. The other - theme of a finished version is rendered on demand by the theme toggle route, - without any model call. + `theme` is the one theme the run renders and the reviewer sees; the root passes it + only when the user asks for a theme. Omitted (None), the run takes the site theme + the page sent with the turn, unless it changes a version whose theme the user asked + for (`pipeline.run_theme`). The other theme of a finished version is rendered on + demand by the theme toggle route, without any model call. """ change_request: str = Field(default="", max_length=MAX_CHANGE_REQUEST_CHARS) diff --git a/agents/anyplot/services.py b/agents/anyplot/services.py index 6c7d122a46b..b86f45721e3 100644 --- a/agents/anyplot/services.py +++ b/agents/anyplot/services.py @@ -72,6 +72,9 @@ class CodeVersion: """The library the version was adapted for; a session can switch library.""" theme: Theme = "light" """The theme the run rendered and the reviewer saw.""" + theme_asked: bool = False + """The user asked for `theme` in the chat, for this version or for the one it changed, so a change of it keeps + the theme; otherwise a change follows the site theme of its turn (`pipeline.run_theme`).""" language: str = "en" """The session language the version's plot text was written in.""" requests: list[str] = field(default_factory=list) diff --git a/agents/anyplot/tools/session.py b/agents/anyplot/tools/session.py index 705c9b0ed5d..41e07079c3d 100644 --- a/agents/anyplot/tools/session.py +++ b/agents/anyplot/tools/session.py @@ -716,9 +716,10 @@ def _fenced_lines(lines: list[str]) -> str: description=( "Adapt the plot to the user's dataset and bindings, render it in one theme, review it, and repair it " "once if needed. Call it with no arguments for 'Create plot'; for a change, pass change_request (English, " - "at most 600 characters) and base='previous'. Leave theme out unless the user asks for a theme: a new plot " - "is light and a change keeps the latest version's theme; pass theme='dark' only when the user asks for a " - "dark plot. Pass expected_texts (at most five) with the texts and number formats the user wrote out for " + "at most 600 characters) and base='previous'. Leave theme out unless the user asks for a theme: the plot " + "then follows the theme of the user's page, and a change keeps a theme the user asked for; pass " + "theme='dark' only when the user asks for a dark plot, and theme='light' only when they ask for a light " + "one. Pass expected_texts (at most five) with the texts and number formats the user wrote out for " "the plot, verbatim. Returns the PlotResult, with what the shipped plot shows under shows." ), input_schema=PipelineArgs, diff --git a/agents/evals/matrix.py b/agents/evals/matrix.py index 9e7c7f00d6e..b7c9b9d9537 100644 --- a/agents/evals/matrix.py +++ b/agents/evals/matrix.py @@ -253,6 +253,11 @@ def parse_args(argv: list[str] | None) -> argparse.Namespace: type=float, help="stop before a case that could take the estimated list-price cost past this (spent + costliest case)", ) + parser.add_argument( + "--theme", + choices=("light", "dark"), + help="the site theme every turn sends (`?theme=`), the theme the plots render in (default: none, light)", + ) parser.add_argument("--out", type=Path, default=DEFAULT_OUT, help="output directory (default agents/evals/reports)") parser.add_argument("--baseline", type=Path, help="baseline report (default: baselines/.json)") parser.add_argument("--no-baseline", action="store_true", help="do not compare with any baseline") @@ -359,6 +364,8 @@ class MatrixConfig: preflight: bool = True render_token: "RenderToken | None" = None keep_attempts: bool = False + theme: str | None = None + """`--theme`: the site theme every turn sends, as the page does; None sends none, and the plots render light.""" argv: list[str] = field(default_factory=list) @@ -1011,6 +1018,7 @@ class CaseRunner: settings: "AgentSettings" renders_dir: Path recorder: AttemptRecorder | None = None + theme: str | None = None async def run(self, case: "EvalCase", repeat: int) -> dict[str, Any]: """One case and repeat, from opening the session to deleting it; never raises for a broken case.""" @@ -1155,7 +1163,8 @@ async def _turn( ) -> dict[str, Any] | None: self.timed.chunks.clear() started = time.monotonic() - response = await self.client.post(f"/v1/sessions/{sid}/messages", headers=headers, json=body) + params = {"theme": self.theme} if self.theme else None + response = await self.client.post(f"/v1/sessions/{sid}/messages", headers=headers, json=body, params=params) turn = Turn( response.status_code, None if response.status_code == 200 else _detail(response), @@ -1278,6 +1287,7 @@ def build_stamp( "tolerance": config.tolerance, "gallery_seed": config.seed, "baseline": str(config.baseline_path) if config.baseline_path else None, + "theme": config.theme, "argv": list(config.argv), } @@ -1395,7 +1405,7 @@ async def _run_matrix( try: transport = httpx.ASGITransport(app=timed) async with httpx.AsyncClient(transport=transport, base_url="http://agents", timeout=None) as client: - runner = CaseRunner(client, timed, collector, settings, config.out / "renders", recorder) + runner = CaseRunner(client, timed, collector, settings, config.out / "renders", recorder, config.theme) for index, (case, repeat) in enumerate(plan, start=1): if config.budget_usd is not None and runs and total + costliest > config.budget_usd: stopped = "budget" @@ -1667,6 +1677,7 @@ def _prepare_and_run(args: argparse.Namespace, settings: "AgentSettings", argv: save_baseline=args.save_baseline, preflight=not args.no_preflight, keep_attempts=args.keep_attempts, + theme=args.theme, render_token=RenderToken(os.environ) if args.gcloud_token else None, argv=list(sys.argv[1:] if argv is None else argv), ) diff --git a/agents/main.py b/agents/main.py index 62f210db5fa..7482db9d3f4 100644 --- a/agents/main.py +++ b/agents/main.py @@ -20,7 +20,7 @@ | `POST /v1/sessions/{sid}/dataset/undo` | | `{preview, profile, bindings, warnings, context_lines, roles, suggestion, version}`: the dataset version before the root's last `prepare_data` or `generate_data` is the current one again, its bindings carried over (`dataset_versions.py`); unconfirmed, like any version the user has not confirmed | `409 run_active`, `422 nothing_to_undo` | | `POST /v1/sessions/{sid}/dataset/to_long` | | The undo route's shape plus `changes`: the page's `.to_long()` applies the wide-table suggestion, recomputed on the server, as the next dataset version with the new columns bound to the roles they were made for; no model call | `409 run_active`, `422 no_suggestion` | | `POST /v1/sessions/{sid}/dataset/copy` | `{from_session}` | The paste route's dataset shape (version 1) plus `confirmed: true`, `complete`, `missing_roles` and `copied_from` (the source's version number): the admin fan-out's copy of the confirmed current data version of `from_session`, a session of the same user and spec, into this fan-out session, byte for byte, with its bindings, their confirmation and the session language; nothing parsed or judged, no model call, no paste counted | `404 session_expired` (either session, also another user's), `409 run_active` (either session), `409 has_dataset` (the target has data), `422 no_dataset`, `422 unconfirmed`, `422 invalid` (the same session, a target that is no fan-out session, another spec, refused bindings), `429 rate_limited` / `429 daily_limit` (`COPIES_PER_MINUTE` / `_PER_DAY`), `503 capacity` | -| `POST /v1/sessions/{sid}/messages` | `{text}` or `{action}` (`create_plot`, `summarize_data`) | SSE `anyplot/1`; the `done` event carries `dataset` (with `roles` and `changes`) after a turn whose `prepare_data` or `generate_data` changed the data, version 1 when `generate_data` made the session's first data, and `language` (`{language, source}`) after every run, and `suggestion` (the `.to_long()` offer in the session language, or null) after every run on a session with data | `413 too_long`, `409 run_active` | +| `POST /v1/sessions/{sid}/messages?theme=` | `{text}` or `{action}` (`create_plot`, `summarize_data`); `theme` (`light`, `dark`, optional) is the site theme the page shows, which a plot run of the turn renders unless the user asked for a theme (`pipeline.run_theme`) | SSE `anyplot/1`; the `done` event carries `dataset` (with `roles` and `changes`) after a turn whose `prepare_data` or `generate_data` changed the data, version 1 when `generate_data` made the session's first data, and `language` (`{language, source}`) after every run, and `suggestion` (the `.to_long()` offer in the session language, or null) after every run on a session with data | `413 too_long`, `409 run_active`, `422` for a `theme` other than `light` or `dark` | | `POST /v1/sessions/{sid}/cancel` | | `204` | | | `POST /v1/sessions/{sid}/outcome` | `{action, version?, reaction?}` (closed sets: `telemetry.OutcomeAction`, `telemetry.Reaction`, the latter with `feedback` only) | `204`: the content-free `outcome` telemetry line (`anyplot/telemetry.py`) | `422`, `429 rate_limited` / `429 daily_limit` (`OUTCOMES_PER_MINUTE` / `_PER_DAY`) | | `POST /v1/sessions/{sid}/versions/{version}/render` | `{theme}` | `{status, reason?, artifacts}` | `404 not_found`, `409 run_active`, `503 capacity` (no render slot in time, or the render store is full) | @@ -2000,8 +2000,12 @@ def _error_code(exc: BaseException) -> str: @app.post("/v1/sessions/{sid}/messages", dependencies=v1_dependencies) async def post_message( - sid: SessionId, body: MessageBody, user: User, runtime: RuntimeDep, rid: RequestId + sid: SessionId, body: MessageBody, user: User, runtime: RuntimeDep, rid: RequestId, theme: Theme | None = None ) -> StreamingResponse: + """One chat turn as SSE. `theme` is the site theme the page shows, the theme a plot run of the turn renders + unless the user asked for one (`pipeline.run_theme`). A query parameter, not a body field: the strict bodies of + this service and the BFF would refuse it from a page deployed before them, while an unknown parameter is ignored. + """ if body.text is not None and len(body.text) > MAX_MESSAGE_CHARS: raise AgentsError(413, "too_long") session = await runtime.session(user, sid) @@ -2017,6 +2021,7 @@ async def post_message( # The refusals sent before the judge (the budget, unsupported content) follow the # session language, never the browser's locale; after the judge, the message's. lang=refusal_lang(view.language) if view else DEFAULT_LANGUAGE, + site_theme=theme, ) _stamp(ledger, view, session) ledger.turn += 1 # this turn's number: the user turns before it, plus one diff --git a/api/routers/agent.py b/api/routers/agent.py index d73f29af0c4..6ff88939c5e 100644 --- a/api/routers/agent.py +++ b/api/routers/agent.py @@ -1067,17 +1067,22 @@ class UpstreamStream: async def open_message_stream( - sid: SessionId, body: MessageBody, ctx: Ctx, client: Client + sid: SessionId, body: MessageBody, ctx: Ctx, client: Client, theme: Literal["light", "dark"] | None = None ) -> AsyncIterator[UpstreamStream]: """Open the upstream stream BEFORE the SSE response starts. A dependency rather than the generator itself, so a 413 here or an upstream 409 `run_active` still reaches the browser as an HTTP status: once the generator runs, the 200 is already on the wire. + + `theme` is the site theme the page shows, which a plot of the turn renders in + unless the user asks for one; forwarded as a query parameter, which an agents + service deployed before it ignores, where its strict body would refuse a field. """ if body.text is not None and len(body.text) > MAX_MESSAGE_CHARS: raise AgentHTTPError(413, "too_long", ctx.request_id) payload = {"text": body.text} if body.text is not None else {"action": body.action} + params: dict[str, str | int] | None = {"theme": theme} if theme is not None else None path = f"/sessions/{sid}/messages" # One wall-clock budget for the whole turn: the ID token, the connection # and the upstream's response headers spend from it too. @@ -1085,7 +1090,9 @@ async def open_message_stream( response: httpx.Response | None try: async with asyncio.timeout_at(deadline.at): - response = await _open_stream(client, ctx, "POST", path, json_body=payload, accept="text/event-stream") + response = await _open_stream( + client, ctx, "POST", path, json_body=payload, params=params, accept="text/event-stream" + ) except (*_UNREACHABLE, TimeoutError) as exc: _log_unreachable(ctx, "POST", path, exc) response = None diff --git a/app/scripts/agent-bff-mock.mjs b/app/scripts/agent-bff-mock.mjs index fe86ee667f4..dda30634157 100644 --- a/app/scripts/agent-bff-mock.mjs +++ b/app/scripts/agent-bff-mock.mjs @@ -109,7 +109,9 @@ * a data change, see above, and a reply naming it; * - ends with "?": a reply only, no pipeline run; * - anything else: a refinement with a repair round, shipped as - * `needs_attention`; "dark" in the text renders the dark theme. + * `needs_attention`; the word "dark" or "light" in the text renders that theme. + * A plot renders in the site theme the page sends (`POST .../messages?theme=`), unless + * the text asked for a theme or the refined version was asked for one (`pipeline.run_theme`). * Pasted data containing "ignore previous" answers `403 data_refused`; pasted * text that looks like a formula answers `422 formula`. `POST .../dataset/convert` * (a dropped binary file) answers by the first bytes: a zip is a workbook with two @@ -984,7 +986,7 @@ const sleep = ms => new Promise(resolve => setTimeout(resolve, ms)); // ---------------------------------------------------------------- the chat turn -async function streamTurn(req, res, session, body, ref) { +async function streamTurn(req, res, session, body, ref, siteTheme = null) { const text = typeof body.text === 'string' ? body.text : ''; const lower = text.toLowerCase(); let closed = false; @@ -1185,12 +1187,22 @@ async function streamTurn(req, res, session, body, ref) { if (session.cancelled && !closed) await sleep(STOP_MS); } if (stopped()) return; - const previous = session.versions[session.versions.length - 1]; - const theme = lower.includes('dark') ? 'dark' : refine && previous ? previous.theme : 'light'; + // The service's `pipeline.run_theme`: a theme asked for (the word "dark" or "light" in the text), the + // asked theme of the version a refinement changes, the site theme of the turn, else the version it + // changes or light. + // The version a refinement changes is this library's latest, as `latest_rendered(library=)` picks it. + const previous = refine + ? session.versions.findLast(item => item.library === session.library) + : undefined; + let askedTheme = previous?.themeAsked ? previous.theme : null; + if (/\blight\b/.test(lower)) askedTheme = 'light'; + else if (/\bdark\b/.test(lower)) askedTheme = 'dark'; + const theme = askedTheme ?? siteTheme ?? previous?.theme ?? 'light'; const version = { number: session.versions.length + 1, library: session.library, theme, + themeAsked: askedTheme !== null, dataset: session.dataset, bindings: session.bindings, // The session language the version's text was written in, as the service keeps it. @@ -2142,10 +2154,14 @@ async function route(req, res) { ); } if (method === 'POST' && rest === '/messages') { + // The page's site theme rides in the query, as at the BFF; any other value is its `422`. + const siteTheme = url.searchParams.get('theme'); + if (siteTheme !== null && siteTheme !== 'light' && siteTheme !== 'dark') + return fail(res, 422, 'invalid', ref); if (session.active) return fail(res, 409, 'run_active', ref); if (typeof body?.text === 'string' && body.text.length > 2000) return fail(res, 413, 'too_long', ref); - return streamTurn(req, res, session, body ?? {}, ref); + return streamTurn(req, res, session, body ?? {}, ref, siteTheme); } if (method === 'POST' && rest === '/outcome') { // The content-free usage beacon (`telemetry.OutcomeAction`): checked like the BFF's body, answered 204. diff --git a/app/src/hooks/useAgentSession.test.ts b/app/src/hooks/useAgentSession.test.ts index 74fd368263b..4c9a52d2808 100644 --- a/app/src/hooks/useAgentSession.test.ts +++ b/app/src/hooks/useAgentSession.test.ts @@ -1,3 +1,5 @@ +import { createElement, type ReactNode } from 'react'; + import { act, renderHook, waitFor } from '@testing-library/react'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; @@ -15,11 +17,21 @@ import { useAgentSession, type UseAgentSessionOptions, } from 'src/hooks/useAgentSession'; +import { type EffectiveTheme, ThemeContext } from 'src/hooks/useLayoutContext'; import type { DatasetChanged, DatasetParsed, RoleSpec } from 'src/lib/agent'; import { pendingReply } from 'src/sections/agent-chat/steps'; const lastItem = (items: ChatItem[]) => items[items.length - 1]; +/** The layout's theme context with `effective` as the theme the site shows. */ +const siteTheme = (effective: EffectiveTheme) => ({ + mode: effective, + effective, + isDark: effective === 'dark', + setMode: () => {}, + cycle: () => {}, +}); + const trackEvent = vi.fn(); vi.mock('src/hooks/useAnalytics', () => ({ useAnalytics: () => ({ trackEvent, trackPageview: vi.fn() }), @@ -1531,6 +1543,8 @@ describe('useAgentSession: turns', () => { const turn = calls.find(call => call.path.endsWith('/messages')); expect(turn?.body).toEqual({ action: 'create_plot' }); + // Outside a layout the site theme is the context's default: light. + expect(turn?.search).toBe('?theme=light'); expect(result.current.state.run).toBeNull(); expect(result.current.state.items.map(item => item.kind)).toEqual([ 'action', @@ -1552,6 +1566,41 @@ describe('useAgentSession: turns', () => { }); }); + it('sends the site theme each turn starts with, so a plot follows the page', async () => { + // User test E2 of 2026-11-01: on the dark site the first plot came back light. + const routes = baseRoutes(); + let number = 0; + routes['POST /sessions/S1/messages'] = () => { + number += 1; + return sse([ + ['plot', { ...PLOT_OK, version: number, artifacts: ['plot-dark.png', 'plot.py'] }], + ['done', {}], + ]); + }; + mockFetch(routes); + let site: EffectiveTheme = 'dark'; + const wrapper = ({ children }: { children: ReactNode }) => + createElement(ThemeContext.Provider, { value: siteTheme(site) }, children); + const { result, rerender } = renderHook(() => useAgentSession(options), { wrapper }); + await waitFor(() => expect(result.current.state.phase).toBe('ready')); + + await act(() => result.current.createPlot()); + const turnSearches = () => + calls.filter(call => call.path.endsWith('/messages')).map(call => call.search); + expect(turnSearches()).toEqual(['?theme=dark']); + // The run rendered the dark theme only: the card shows it, and nothing renders twice. + await waitFor(() => expect(result.current.state.versions[1].images.dark?.state).toBe('ready')); + expect(result.current.state.versions[1].theme).toBe('dark'); + expect(calls.some(call => call.path.endsWith('/render'))).toBe(false); + + // The user switched the site to light: the next turn sends the switch, the shown version stays. + site = 'light'; + rerender(); + await act(() => result.current.sendMessage('bigger markers')); + expect(turnSearches()).toEqual(['?theme=dark', '?theme=light']); + expect(result.current.state.versions[1].theme).toBe('dark'); + }); + it("addresses the artifacts by the server's version number, not a count", async () => { const routes = baseRoutes(); // Version 1 was stored but its event never arrived (a cut stream): this is version 2. diff --git a/app/src/hooks/useAgentSession.ts b/app/src/hooks/useAgentSession.ts index 6c93f314d4a..097162a2777 100644 --- a/app/src/hooks/useAgentSession.ts +++ b/app/src/hooks/useAgentSession.ts @@ -16,6 +16,13 @@ * events instead (the server numbers them in the same order), which drifts * only when a stored version's event never arrives. * + * Theme: every turn sends the theme the site shows when it starts + * (`useTheme().effective`, so `system` resolves first), and a plot of that turn + * renders in it unless the user asked for a theme in the chat (owner decision + * of 2026-11-01); the light and dark switch renders the other one. A switch of + * the site theme mid-session changes no shown version, only the next one. The + * fan-out's sessions are this hook too, so they send the same theme. + * * Bindings: the dataset answer lists the spec's roles. A single role binds * under its own name, a variadic family `y` under its members `y1`, `y2`, ...; * the hook keeps the members contiguous (`compactFamilies`), so clearing `y2` @@ -143,6 +150,7 @@ import { useCallback, useEffect, useReducer, useRef } from 'react'; import { useAnalytics } from 'src/hooks/useAnalytics'; +import { useTheme } from 'src/hooks/useLayoutContext'; import { agentApi, AgentApiError, @@ -1104,11 +1112,14 @@ export function useAgentSession({ }: UseAgentSessionOptions) { const [state, dispatch] = useReducer(agentReducer, library, initialAgentState); const { trackEvent } = useAnalytics(); + const { effective: siteTheme } = useTheme(); // Values async callbacks read after an await; refs, so a stale closure never // acts on a previous session. const stateRef = useRef(state); stateRef.current = state; + const siteThemeRef = useRef(siteTheme); + siteThemeRef.current = siteTheme; const abortRef = useRef(null); const stoppedRef = useRef(false); const stopTimer = useRef | null>(null); @@ -1735,7 +1746,13 @@ export function useAgentSession({ ); let finished = false; try { - const response = await agentApi.postMessage(token, sid, body, controller.signal); + const response = await agentApi.postMessage( + token, + sid, + body, + controller.signal, + siteThemeRef.current + ); if (!response.body) throw new AgentApiError(502, 'upstream'); for await (const raw of readSseEvents(response.body, controller.signal)) { const event = parseAgentEvent(raw); diff --git a/app/src/lib/agent.ts b/app/src/lib/agent.ts index b6340f50615..58312fb1606 100644 --- a/app/src/lib/agent.ts +++ b/app/src/lib/agent.ts @@ -1063,9 +1063,20 @@ export const agentApi = { if (!dataset) throw new AgentApiError(502, 'upstream'); return dataset; }, - /** Opens the chat turn's SSE stream; the caller reads it with `readSseEvents`. */ - postMessage: (token: string, sid: string, body: TurnBody, signal: AbortSignal) => - agentFetch(`${sessionPath(sid)}/messages`, token, { + /** + * Opens the chat turn's SSE stream; the caller reads it with `readSseEvents`. + * `siteTheme`, the theme the site shows, is the theme a plot of the turn + * renders in unless the user asks for one in the chat. It travels in the + * query, not the strict body, so a page deployed before the API is no `422`. + */ + postMessage: ( + token: string, + sid: string, + body: TurnBody, + signal: AbortSignal, + siteTheme?: Theme + ) => + agentFetch(`${sessionPath(sid)}/messages${siteTheme ? `?theme=${siteTheme}` : ''}`, token, { ...post(body), headers: { Accept: 'text/event-stream' }, signal, diff --git a/app/src/pages/AgentChatPage.test.tsx b/app/src/pages/AgentChatPage.test.tsx index 892db1a8458..e93e25d8209 100644 --- a/app/src/pages/AgentChatPage.test.tsx +++ b/app/src/pages/AgentChatPage.test.tsx @@ -330,10 +330,12 @@ function renderAt(url: string) { const PAGE = '/debug/agent?spec=scatter-basic&library=matplotlib&language=python'; const agentCalls = () => fetchMock.mock.calls.filter(([input]) => String(input).includes('/debug/agent')); +/** The path of a call, without the query (a turn carries the site theme there). */ +const pathOf = (input: unknown) => new URL(String(input)).pathname; /** The bodies of the chat turns the page posted, in order. */ const turnBodies = () => agentCalls() - .filter(([input]) => String(input).endsWith('/messages')) + .filter(([input]) => pathOf(input).endsWith('/messages')) .map(([, init]) => JSON.parse(String((init as RequestInit).body)) as Record); const lastTurn = () => turnBodies().slice(-1)[0]; @@ -1540,7 +1542,7 @@ describe('AgentChatPage .plot_all()', () => { const fanOutCalls = (rest: string) => agentCalls() .filter(([input]) => /\/sessions\/F-[a-z0-9]+/.test(String(input))) - .filter(([input]) => String(input).endsWith(rest)) + .filter(([input]) => pathOf(input).endsWith(rest)) .map(([input, init]) => ({ session: String(input).match(/F-[a-z0-9]+/)?.[0], body: (init as RequestInit).body ? JSON.parse(String((init as RequestInit).body)) : null, diff --git a/app/src/sections/agent-chat/FanOut.test.tsx b/app/src/sections/agent-chat/FanOut.test.tsx index 3d716277599..9138a9ad42e 100644 --- a/app/src/sections/agent-chat/FanOut.test.tsx +++ b/app/src/sections/agent-chat/FanOut.test.tsx @@ -1,6 +1,7 @@ import { fireEvent, render, screen, waitFor, within } from '@testing-library/react'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { type EffectiveTheme, ThemeContext } from 'src/hooks/useLayoutContext'; import { FanOut } from 'src/sections/agent-chat/FanOut'; vi.mock('src/hooks/useAnalytics', () => ({ @@ -10,6 +11,7 @@ vi.mock('src/hooks/useAnalytics', () => ({ interface Call { method: string; path: string; + search: string; body: unknown; } @@ -92,10 +94,11 @@ function mockBff(ineligible: string[] = [], copy: () => Response = () => json(20 vi.stubGlobal( 'fetch', vi.fn(async (input: string, init: RequestInit = {}) => { - const path = new URL(input).pathname.replace(/^.*\/debug\/agent/, ''); + const url = new URL(input); + const path = url.pathname.replace(/^.*\/debug\/agent/, ''); const method = init.method ?? 'GET'; const body = typeof init.body === 'string' ? JSON.parse(init.body) : undefined; - calls.push({ method, path, body }); + calls.push({ method, path, search: url.search, body }); if (method === 'POST' && path === '/sessions') { const { library } = body as { library: string }; if (ineligible.includes(library)) return json(422, { detail: 'not_eligible', ref: 'r' }); @@ -136,20 +139,30 @@ function renderFanOut( { onClose = vi.fn(), upload = UPLOAD, - }: { onClose?: () => void; upload?: typeof UPLOAD | null } = {} + site = 'light', + }: { onClose?: () => void; upload?: typeof UPLOAD | null; site?: EffectiveTheme } = {} ) { + const theme = { + mode: site, + effective: site, + isDark: site === 'dark', + setMode: () => {}, + cycle: () => {}, + }; return render( - + + + ); } @@ -208,6 +221,14 @@ describe('FanOut', () => { expect(within(card('altair')).getByTestId('fanout-status')).toHaveTextContent('adapting…'); }); + it("plots every library in the page's site theme", async () => { + mockBff(); + renderFanOut(['matplotlib', 'seaborn'], 2, { site: 'dark' }); + + await waitFor(() => expect(plotTurns()).toHaveLength(2)); + expect(plotTurns().map(call => call.search)).toEqual(['?theme=dark', '?theme=dark']); + }); + it('posts the paste again on a server without the copy, with the bindings of the click', async () => { mockBff([], () => json(404, { detail: 'not_found', ref: 'r-404' })); renderFanOut(['matplotlib', 'seaborn'], 2); diff --git a/changelog.d/agent-plot-follows-site-theme.md b/changelog.d/agent-plot-follows-site-theme.md new file mode 100644 index 00000000000..79cc21593a4 --- /dev/null +++ b/changelog.d/agent-plot-follows-site-theme.md @@ -0,0 +1,18 @@ +### Added + +- **`--theme dark` for the eval matrix.** Every turn of the run sends the + site theme, as the chat page does, so dark first renders can be measured + before the public release. Every committed baseline is light. (#12505) + +### Changed + +- **The agent chat's plot follows the site theme.** On the dark site the + first render of each plot version is now dark, and on the light site it is + light. The other theme is still one click away on the light and dark switch, + and each version still takes one render. The page sends the theme the site + shows with every turn (`POST .../messages?theme=`). The query parameter lets + the page, the API and the agents service deploy in any order. A change made + after a site-theme switch follows the switch. A theme the user asked for in + the chat holds for later changes of that plot. The admin fan-out + (`.plot_all()`) plots every library in the same theme. This fixes user test + E2 of 2026-11-01, where the dark site showed a bright plot card. (#12505) diff --git a/docs/concepts/agent-network.md b/docs/concepts/agent-network.md index 9c7987170fb..33731e72f4e 100644 --- a/docs/concepts/agent-network.md +++ b/docs/concepts/agent-network.md @@ -10,7 +10,7 @@ This document describes the agent network that lets a visitor paste their own da |---|---| | Product step 1 | "Use with my data": the user pastes data on a plot page, the network adapts the catalogue implementation for that spec and library to the data, renders it in a sandbox, reviews it with at most one repair round (a render repaired after a rejection is reviewed once more), and returns the image and the code. For measurements only, the eval switch `AGENT_MAX_ATTEMPTS` (`--max-attempts` of the harness) can raise this to three attempts, a second repair round; the default stays one repair round, and a run makes at most two reviewer calls either way | | Product step 2 | "Find the right plot type", built on the catalogue knowledge that also backs the MCP server | -| Result delivery | Users see and copy both the image and the code: the plot is shown inline in the theme it was rendered in (light unless the user asked for a dark plot), a light and dark switch renders the other theme on demand without any model call, and the image can be copied to the clipboard and downloaded as PNG; the code is shown with syntax highlighting, copied with one click, and downloaded together with `data.csv` so it runs unchanged | +| Result delivery | Users see and copy both the image and the code: the plot is shown inline in the theme it was rendered in (since 2026-11-01 the site theme the page shows, unless the user asked for a theme in the chat), a light and dark switch renders the other theme on demand without any model call, and the image can be copied to the clipboard and downloaded as PNG; the code is shown with syntax highlighting, copied with one click, and downloaded together with `data.csv` so it runs unchanged | | Plot attribution | Owner decision of 2026-10-10 (variant `s1c-jb-sq2` of the comparison): every PNG the service serves carries a footer strip below the plot, "made with any.plot()" on the left and "anyplot.ai/" on the right, in JetBrains Mono with the dot drawn as a brand-green square. It is composited at image level after the run and never written into the user's code, so the exported `plot.py` reproduces the plot without it. MonoLisa, the site's own font, is not used for it, because its EULA covers desktop, web, application and ePub use only. See [Footer strip](#footer-strip) | | Render scope | Python first, designed so every further library is additive. Phase 1 serves matplotlib and seaborn. The runtime registry also has runtimes for plotnine and altair since 2026-10-25, for pygal and lets-plot since 2026-10-26 (steps 1b and 1c of [the phase-2 plan](../../agentic/specs/261024-agent-all-libraries.md)), for ggplot2 on the R runtime since 2026-10-29 (step 2c), for Makie on the Julia runtime since 2026-10-30 (step 5), for Chart.js, D3, ECharts and Highcharts on the JavaScript runtime since 2026-10-30 (JS-4a of step 3), for MUI X on it since 2026-10-31 (JS-4c), and for plotly and bokeh on the Python runtime since 2026-10-31 (PB-2a of step 4, through the renderer's figure mode): all 15 catalogue libraries. `AGENT_LIBRARIES` still defaults to `matplotlib,seaborn`; the deploy serves every library with a runtime (the six Python libraries since 2026-10-28, ggplot2 since 2026-10-29, Makie, Chart.js, D3, ECharts and Highcharts since 2026-10-30, MUI X, plotly and bokeh since 2026-10-31), because only the admin reaches the chat yet (owner decision), and each library's eval matrix at 80 % gates the public release. Highcharts is served under its non-commercial licence (owner decision of 2026-10-30): the site charges nothing | | Data input | Paste text only (CSV, TSV, semicolon CSV, a table aligned with spaces, JSON including API answers that wrap their records, a Python literal), hard cap 1 MB (200 KB until 2026-10-14); data the assistant computes (`run_data_code`, `generate_data`, `prepare_data`) up to 2 MB | @@ -86,7 +86,7 @@ Deviations from the first sketch, each evidence-backed: there is no data-intake 2. `POST /debug/agent/sessions {spec_id, library, locale}`. The BFF validates the spec and library against `core/constants.py` and the enabled libraries from anyplot-agents `GET /v1/status` (cached), loads a catalogue snapshot through `core/catalogue/queries.py` (title, description, data roles, notes, noqa-stripped code, library version), derives the user id, and forwards it. anyplot-agents normalises the code, runs the readiness scan and the SECURITY validator, and answers `422 not_eligible` for blocked pairs (the 13 map specs, validator failures, files without a `THEME` block or with a non-literal save target). anyplot-agents has no database access in phase 1. `GET /debug/agent/eligibility?spec=&library=` gives the button the same answer. `locale` is the browser's locale, recorded with a feedback case; it sets no language, and every session starts in English (see [Session language](#session-language)). 3. `POST .../dataset {text}` runs the deterministic parse and returns a 20-row preview, the `DatasetProfile`, default bindings and warnings. A deterministic pre-filter over every header and cell for text addressed to an AI, then the dataset judge with a data rubric, refuse instruction-like datasets. Both look for injection only: names, postal and e-mail addresses, phone numbers, web addresses and amounts are the user's data and plot like any other value. The judge sees every header and the same five sample rows and top values the root and the adapter get (every value of a column with at most 12 distinct ones, else the five most frequent), with cells in full, in parts of about 4,000 characters, one judge call per part (one part for every eval fixture). When the paste carried notes around the table (`context_lines`, see [Parse](#parse)), they go through the same pre-filter and then, after the dataset judge passed the table, one scope-judge call with the scope rubric, the lines fenced as ``. A pre-filter hit, a verdict other than `in_scope` or no verdict drops every note with a warning and stores the table all the same; no verdict on notes is a strike, because a hidden injection in a copied web table is not the user's doing, and the judge's tokens are booked to the user like the dataset judge's (the `notes_judge` attribution line carries the verdict and the counts, never the text). The route answers the kept notes as `context_lines`, the stored dataset keeps them, and every version that `prepare_data`, `generate_data` or `.to_long()` makes from it carries them on; a new paste replaces them. Judging the notes at the paste rather than in the turn that delivers them keeps every place the root can read them (the summary turn, `get_dataset_profile`) behind one verdict, and lets the data panel show exactly what the assistant got (`# notes sent to the assistant`). No other LLM call happens for a table. Text that is no table the parser can read (prose such as "a bar chart with dog 7, cat 3 and bird 4", `label value` lines without a separator, a list of names typed one per line; at most 4,000 characters and 200 non-blank lines, so that a list without a header fits the schema's 200 rows, never a JSON array or object, which the parser reads or refuses itself) takes the transcriber path instead; free text past those limits is `422 unparseable` with the limit as its cause, never stored as it parsed. One column of typed values (integers, numbers, dates, booleans: the input of a histogram, a density or a box plot), of digit codes kept as text (`01067`) or of categories under a header, at least one repeating, is a table and never reaches the transcriber (live round 7 of 2026-10-16: 800 commute times timed out the model every time); a category list whose header has the categories' form (no `_`, brackets, `:`, `/`, `%`, `#` or digit, and no more words than a category) is stored with the short note `read '
' as the column name`, since no cheap rule tells `Tier` above `Hund, Katze` from `Apfel` above `Birne, Kirsche`; a header with a column name's form gets no note. A list of values without a header line keeps its first line as the first value, in a column named `value`, with a warning (`parse._header_as_row`); one of 20,001 values is refused as too many rows. A transcription whose answer stops at its output cap or breaks the schema's bounds (`models.AnswerCutOff`, not retried), or that runs out of time on a text of at least half the limits (`transcribe.near_transcribe_cap`), is `422 unparseable` with a cause that names the text's line and character count and asks for a separator; a short text that times out (a stalled endpoint: the budget is the only clock, no request has a timeout of its own), an error, a quota or an unparseable answer is `503 guard_unavailable` (owner rule of 2026-10-11): the pre-filter runs on the raw text, the judge model reads it under the `TranscribedTable` schema (`agents/anyplot/data/transcribe.py`, rubric `prompts/transcriber.md`: keep every value as written, invent and compute nothing, a note on anything ambiguous), the table's CSV is parsed like a paste, and the route answers `transcribed: true` with that CSV, the preview and the note, storing nothing. The page shows the table for the user to confirm or discard, because a wrong reading makes everything after it wrong; confirming posts the CSV through this same route with `confirmed: true`, which skips the free-text fallback (a one-column table would otherwise be read again) and the narrowing to the table (the user has seen every row, so no line is left out as a note or a total), while the deterministic parse, the pre-filter, the judge and the store run as for any paste. A binary data file reaches it the same way, through one pre-process (owner decision of 2026-10-15, part 2 of 2): Excel (`.xlsx`, `.xlsm`, `.xlsb`, `.xls`), OpenDocument (`.ods`), Parquet, Feather (v1 and v2), Arrow IPC (file and stream) and NumPy (`.npy`, `.npz`) go as base64 in JSON (the BFF's CSRF guard takes JSON only; at most 5 MB raw, the BFF refuses a larger body by its `Content-Length` before reading it) to `POST /v1/sessions/{sid}/dataset/convert`, which converts the file in the renderer's sandbox and answers the text the paste box takes; nothing is stored and no model reads anything, so the text then goes through the ordinary paste route above (its limits, pre-filter and judges) when the user presses `.parse()`, with the file's name and sheet as `source` (`Bericht.xlsx · sheet ‹2024›`; the sheet in single angle quotes since live test round 4 of 2026-10-15, so the data line's `from "…"` never nests quotes), screened as above. The renderer runs the harness in convert mode (`harness.py --convert input.bin`, `POST /convert`): like data mode it reads the bytes, loads the converter (`agents/anyplot/render/convert.py`, copied next to the harness in the image) and imports every reader it needs before it arms its guard (`ConvertGuard`: `DataGuard` with no code to run at all, but the one `compile` NumPy's `.npy` header reader makes to parse its dict literal), then parses from memory with the guard armed and writes `out.csv`, `meta.json` and `status.json`; a refusal is an exit code (`wire.CONVERT_REASONS`) with the measured value and the cap. The format comes from the first bytes, never from the name: every zip is checked from its central directory first (at most 2,000 entries, 100 MB declared uncompressed, 250 times the archive's size, no encrypted member) and every member is inflated, bounded by its declared size, to prove that size, because calamine's reader trusts none (security review of #12194); every pattern over file content is linear (the review's 30 MB run of `