Repository navigation
Offer a model only when the model check has proven it on the Bot's harness - #1779
Conversation
The picker in the real appA full OpenBot stack on a throwaway Linux box (personal edition, a local Intelligence, an Anthropic key, the Claude Agent SDK harness picked in setup), driven by headless Playwright in Chromium and WebKit, light and dark, at this pull request's head before the last lock commit (45e55bf).
Without the Claude Agent SDK harness picked, the same stack lists Opus 5.5 and Fable 5.2, as it should. At a phone's width (390px) the notice wraps to three lines with no sideways scroll. Every screenshot was looked at: the notice sits above the message box with the composer's other notices, in the muted text colour, and reads correctly in dark mode. Nothing broken. |
76b9ffd to
ab04444
Compare
The picker in the real app, on the regenerated lock (07bc282)A full OpenBot stack on a throwaway Linux box (personal edition, a local Intelligence), headless Playwright in Chromium and WebKit, light and dark. 34 of 34 checks pass. Anthropic key, Claude Agent SDK harness picked in setup (Claude Code 2.1.325 in the lock): the picker lists Claude Opus 5.5, Claude Fable 5.2, Claude Sonnet 5.5 and Claude Haiku 5.5. Pass in Chromium light and dark, WebKit light and dark. A Bot on Google, where the lock refuses Every screenshot was looked at: layout, dark mode and the notice read correctly. Nothing broken. |
07bc282 to
c2351cd
Compare
The model picker lists each provider's models live, but each harness image pins its model client. A provider refusing a new model to an older client (Claude Code 2.1.259 for Claude Opus 5.5 and Fable 5.2) went unnoticed because nothing calls today's models through the published images. A scheduled workflow runs scripts/model-compat.ts once per published harness: it lists each provider's models as the picker does, starts the harness image (the published latest and one built from source) on each provider with a key, sends one run per model, and checks the ChatGPT plan's Codex client version against Codex's catalogue. Failures fail the run and open or update one issue; a provider without a key is reported as skipped.
… Unbiased AI out The client boundary test failed on scripts/model-compat.ts: its provider listings, harness health checks and Mastra's agent API are not OpenBot's server, so they are allowed as third-party. Unbiased AI is a gateway serving its own model, so the check no longer lists or runs it, and the Desktop drift test names it as left out on purpose.
…rness Adds the model lock: shared/model-lock.json, written from the model check's source-image results by scripts/model-compat.ts lock, and the rule in shared/model-lock.ts that the model list applies. The daily check opens or updates one pull request when a model starts or stops answering; a pull request that changes a harness's client runs the check for that harness and fails when a verified model breaks. A conversation set to a model that isn't verified says so in the composer (ModelChoice.unverified).
Every published harness on OpenAI, Anthropic and Google, source images, run on a throwaway Linux box; agent-agno from main after its Claude fix. A rate limit is inconclusive, like a timeout, so a busy key never takes a model away.
… away Two Gemini keys disagreed on gemini-2.5-pro and gemini-2.5-flash-lite through the same clients, because Google closes older models to new projects. Such a failure, a bare 404 and a rate limit now keep what the lock knew, and the pull request check also runs the published image with the same keys, so a verified model counts as broken only when the change's image fails it and the published one doesn't.
…nned The lock records each harness's clients; scripts/model-lock.test.ts now fails when a harness pins another version (an exact requirements pin, Mastra's bun.lock, the Codex client version), which is how a lock written before the Claude Code bump kept hiding Opus 5.5 and Fable 5.2. The pull request check uploads its results so the lock can be written from the change's own run.
From the Model lock run on 1103261, with the repository's keys: the Claude Agent SDK harness on claude-agent-sdk 0.2.182 (Claude Code 2.1.325) answers on all 15 Anthropic models, Opus 5.5 and Fable 5.2 included, and AG2 and Langroid answer on all 29 OpenAI models. The lock from before the bump stays as a test fixture for the rule.
…check it on every provider shared/model-lock.json names agent-langgraph-ts with empty pairs for Anthropic, OpenAI and Google, so nothing is offered on it until the model check has sent a model a real request through its own client and it answered, including on a provider CI has no key for. The check now runs every engine the lock names as well as the published harnesses (scripts/model-compat.ts harnesses, which both workflows' matrices use), so the engine is checked from its source image before it is published. The probe also reads the engine's LangChain and AG-UI client versions, under plain Node when an image has no Bun.
…and test the lock rule on its own lock (#1793) The engine ships @langchain/google, which the probe did not read, so a run recorded no Gemini client for it. Two tests read the live model lock and could only pass while it was empty, so the first lock refresh after a real run would have failed them; they now build on an unrun lock of their own.
…leave a default engine empty agent-langgraph-ts's entry comes from its own run: Anthropic 15, OpenAI 29 and Google 18 verified. The two Gemma 4 models are refused: they answer with their reasoning in the reply text. A test now fails when an engine new Bots run on by default verifies nothing on a provider it is locked for, which the empty pairs it was first locked with would have done to every new Bot.
…d fail a reply that carries its reasoning The staleness guard now reads package-lock.json too, which showed the engine's entry was written before its Gemini client was recorded. A reply with the model's reasoning in it (Gemma 4's <thought> through this engine) now fails the check in its own words, so the lock refuses it on every run instead of by hand.
c2351cd to
d3d64c0
Compare
…cript) included From the Model lock run on d3d64c0 with the repository's keys, every engine's source image on main's pins. agent-langgraph-ts verifies Anthropic 15, OpenAI 29 and Google 18, with its Gemini client recorded. The Gemma 4 models are refused on every engine that reaches Google through its OpenAI-compatible API: they answer with their reasoning in the reply.
The picker in the real app, on this head (046bc43)A full OpenBot stack on a throwaway Linux box, headless Playwright in Chromium and WebKit, light and dark. 38 of 38 checks pass. Anthropic key, Claude Agent SDK harness picked in setup: the picker lists Claude Opus 5.5, Claude Fable 5.2, Claude Sonnet 5.5 and Claude Haiku 5.5. Pass in all four. A Bot on Google: both Every screenshot was looked at: layout, dark mode and the notice read correctly. |
…e before it counts Google now answers gemini-2.5-flash-lite with "no longer available to new users": the lock refuses it, so a person set to it is told, and it never fails a pull request since every image fails it. A one-off timeout (gemini-4-carbon through Antigravity's source image, which answered through the published image in the same run) no longer counts: every failing run is asked once more.
… retired From the Model lock run on d6e4e12 with the repository's keys. Google now answers gemini-2.5-flash-lite "no longer available to new users", so it is refused where the engine passes Google's words on, and unverified where it doesn't. Every other pair is unchanged.
|
Browser check on the exact head 7deb529 (main merged in, #1814's clients included): 46 of 46 pass in Chromium and WebKit, light and dark. Claude Agent SDK harness: the picker lists Claude Opus 5.5, Fable 5.2, Sonnet 5.5 and Haiku 5.5. A Bot on Google: lists Gemini 3.8 Flash and Gemini 2.5 Flash; hides both gemini-omni models, both Gemma 4 models and gemini-2.5-flash-lite, each of which is in Google's live listing; a conversation saved on gemini-omni-1.1-flash says so and falls back to Gemini 3.8 Flash. Every screenshot looked at; nothing off. |
Why
On a Claude plan, every message to Claude Opus 5.5 and Claude Fable 5.2 was refused. The picker lists each provider's models live, but every harness pins its model client, and a provider can refuse a new model to an older client. The daily model check (#1775) found 78 such pairs across 13 harnesses. Fixing them one at a time doesn't hold: every new model can break an old pin. So the picker now offers a model only when the check has proven it on the Bot's harness, through the clients this commit pins.
This builds on #1775 (its commits are here too) and fixes its failing CI: the client boundary test didn't name the check's own requests (provider listings, the harness images it starts). That was the one real failure; "tests" and "verify" failed because that shard did. Unbiased AI is left out of the check: it's a gateway serving its own model.
What
shared/model-lock.json: per harness, the model clients itssourceimage carried, and per provider the models that answered and the ones refused, in the provider's words. Written bybun scripts/model-compat.ts lockfrom the check's results, never by hand.shared/model-lock.tsholds the rule the model list applies.ModelChoice.unverified(additive) names the model a conversation was set to when it's set aside. The composer says, for example: "Claude Opus 5.5 doesn't work with this Bot yet, so this conversation uses Claude Sonnet 5.5. Pick another model to change it."scripts/model-lock.test.tsfails when a harness pins a client at another version than the lock was proven on (an exactrequirements.txtpin, Mastra'sbun.lock, the Codex client version). A lock written before Bundle a Claude Code new enough for current Claude models #1770 kept hiding Opus 5.5 and Fable 5.2 after Claude Code 2.1.325 ran them; this catches that.agent-langgraph-tsis locked from its own run. A test fails when an engine new Bots run on by default verifies nothing on a provider it is locked for, and the staleness guard reads itspackage-lock.json.<thought>…</thought>OK) fails the check, so a model that does it is refused in the lock on every run. The Gemma 4 models do, on every engine that reaches Google through its OpenAI-compatible API.requirements.txt,package.json,bun.lock,Dockerfileor the Codex client version: it builds that harness and the published image, sends every listed model a request through both with the same keys, fails when a model the lock verifies answers through the published image but not the change's, and uploads its results so the lock can be written from them. Fork pull requests get no keys and are skipped.gemini-2.5-proandgemini-2.5-flash-litethrough the same clients, because Google closes older models to new projects. A failure about the key (a model it can't use, a bare 404, a rate limit) keeps what the lock knew. So does one that says nothing about the model (a harness that didn't start, a timeout).Today's lock
Written from this branch's own Model lock run on main (after #1805), with the repository's keys: every engine's
sourceimage on OpenAI, Anthropic and Google. Verified / refused:external_executionin tools)On Google,
gemini-omni-1.1-flashandgemini-omni-1.2-flashare refused everywhere (Interactions API only), and the two Gemma 4 models wherever Google is reached through its OpenAI-compatible API (reasoning in the reply). CrewAI and Mastra hide Google's words, sogemini-2.5-flash-liteandgemini-decision-1read as refusals there until a run with a key that can use them. Agno, Microsoft Agent Framework and Pydantic AI fail every Google model today: harness fixes, which the daily refresh picks up once they merge.Tests
scripts/model-lock.test.ts, the staleness guard: on the rebased branch with the old lock, 11 pass and 1 fails (claude-agent-sdkpinned 0.2.182, locked 0.2.152;ag-ui-langgraphand the Codex client version not recorded). After writing the lock from this branch's run: 12 pass. Readingpackage-lock.jsontoo, it failed again on the LangGraph (TypeScript) entry written before its Gemini client was recorded, and passes on the lock written from this branch's run. The default engine guard fails on that engine's first lock (empty pairs: anthropic, google, openai) and passes on this one. The other tests here cover writing the lock from the check's own rows (Claude Code's refusal, a timeout, a rate limit, a provider with no key, a client bump, two Gemini keys disagreeing), the refresh pull request's words, and the guard.shared/model-lock.test.ts: the rule against real data, the listings OpenAI and Anthropic answered on 2026-11-17 and two locks: the one from before the Claude Code bump (kept as a fixture) hides Opus 5.5 and Fable 5.2 with Anthropic's own words; the shipped one offers them and still holds back the Gemini models Google refuses. 8 pass. Before the lock existed: 3 pass, 4 fail.scripts/model-compat.test.tsandapi/client-boundary.test.ts: 17 and 6 pass. The boundary test failed on Check every listed model daily through the clients each harness ships #1775 before its fix.Keys
ANTHROPIC_API_KEY,OPENAI_API_KEYandGEMINI_API_KEYare set.XAI_API_KEYisn't yet, so xAI is skipped and stays open. No GitHub App is configured for the refresh pull request, so it's opened withGITHUB_TOKENand its CI has to be approved by hand.