Repository navigation
Check every listed model daily through the clients each harness ships - #1775
Closed
davidmckayv wants to merge 5 commits into
Closed
davidmckayv wants to merge 5 commits into
davidmckayv wants to merge 5 commits into
Conversation
The model picker lists each provider's models live, but each harness image pins its model client. A provider refusing a new model to an older client (Claude Code 2.1.259 for Claude Opus 5.5 and Fable 5.2) went unnoticed because nothing calls today's models through the published images. A scheduled workflow runs scripts/model-compat.ts once per published harness: it lists each provider's models as the picker does, starts the harness image (the published latest and one built from source) on each provider with a key, sends one run per model, and checks the ChatGPT plan's Codex client version against Codex's catalogue. Failures fail the run and open or update one issue; a provider without a key is reported as skipped.
davidmckayv
requested review from
MikeRyanDev,
dustywusty,
guidovizoso,
mxmzb,
samjulien and
tylerslaton
as code owners
November 17, 2026 20:41
… Unbiased AI out The client boundary test failed on scripts/model-compat.ts: its provider listings, harness health checks and Mastra's agent API are not OpenBot's server, so they are allowed as third-party. Unbiased AI is a gateway serving its own model, so the check no longer lists or runs it, and the Desktop drift test names it as left out on purpose.
This was referenced Nov 17, 2026
Contributor
Author
|
Included in #1779, which carries this PR's commits and builds on them, so main already has this work. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On a Claude plan, every message to Claude Opus 5.5 and Claude Fable 5.2 was refused. The model picker lists each provider's models live, but the Claude Agent SDK harness pins
claude-agent-sdk==0.2.152, which bundles Claude Code 2.1.259, and Anthropic refuses that version for those models: "Claude Code 2.1.259 does not support this model; version 2.1.290 or newer is required". Nothing caught it, because nothing calls today's models through the clients the harness images ship.What it checks
A new scheduled workflow, Model compatibility (
.github/workflows/model-compat.yml), runs daily and by hand, only onmain, never on a pull request. It runs one job per harness in.github/published-images.json. Each job runsscripts/model-compat.ts, which:latest, the published image people run, andsource, the image built from the commit, which is what the next release ships.It writes one row per harness, image, provider and model, with the provider's own words, to the job summary. On any failure, the run fails and opens or updates a single issue, "Model compatibility check failing". The first passing run closes it. A provider whose key is missing is reported as skipped, not failed, and the summary names the missing secrets. Keys are scrubbed from every detail. Every action is pinned by commit SHA, and zizmor and actionlint pass.
scripts/model-compat.test.tscovers the pure logic against real-world inputs: the picker's filter on real listings, the matrix, reading real harness streams (Claude Code's version refusal, Agno, CrewAI's snapshot replies, Mastra), the Codex version gate, and drift guards that fail if the Desktop gains a provider address or a harness is published that the check doesn't cover. Docs: docs/releasing.md, "Model compatibility".Local run
I ran it on a throwaway Linux box with the Anthropic and OpenAI dev keys, the four jobs at a time. The whole matrix took 9 minutes, and each harness took 1 to 4 minutes, including its source build (about 80 seconds), so one GitHub runner per harness fits easily. Result: 78 failed, 942 passed, 34 skipped (no Gemini, xAI or Unbiased key).
The failures, grouped (each on both
latestandsource):temperatureandtop_pcannot both be specified for this model.Every other harness, provider and model answered. The Claude Agent SDK rows are the refusal people hit today. The rest are harnesses whose request shape those models no longer accept, which the check now shows too.
Secrets needed
Add these repository secrets:
ANTHROPIC_API_KEY,OPENAI_API_KEY,GEMINI_API_KEY(Google, also used for Antigravity),XAI_API_KEYandUNBIASED_API_KEY. Until a secret is added, that provider's rows show as skipped. The workflow uses only the built-in token otherwise, withissues: writefor the one issue.