Skip to content

Check every listed model daily through the clients each harness ships - #1775

Closed
davidmckayv wants to merge 5 commits into
mainfrom
ci/model-compat-check
Closed

davidmckayv wants to merge 5 commits into
mainfrom
ci/model-compat-check

Conversation

@davidmckayv

Copy link
Copy Markdown
Contributor

Why

On a Claude plan, every message to Claude Opus 5.5 and Claude Fable 5.2 was refused. The model picker lists each provider's models live, but the Claude Agent SDK harness pins claude-agent-sdk==0.2.152, which bundles Claude Code 2.1.259, and Anthropic refuses that version for those models: "Claude Code 2.1.259 does not support this model; version 2.1.290 or newer is required". Nothing caught it, because nothing calls today's models through the clients the harness images ship.

What it checks

A new scheduled workflow, Model compatibility (.github/workflows/model-compat.yml), runs daily and by hand, only on main, never on a pull request. It runs one job per harness in .github/published-images.json. Each job runs scripts/model-compat.ts, which:

  1. Lists each provider's models from the same source, with the same filter, as the picker (docs/configuration.md, "The model list"): OpenAI, Anthropic, Google, xAI and Unbiased AI.
  2. Starts the harness image on each provider, with the environment the Desktop writes for it. It sends one AG-UI run per model (Mastra through its own agent API, as OpenBot reaches it), carrying one tool, as a Bot's run does. The harness's own pinned client makes the call. It checks two images: latest, the published image people run, and source, the image built from the commit, which is what the next release ships.
  3. Compares the LangGraph harness's ChatGPT plan Codex client version with the minimum each listed Codex model needs, from openai/codex's published model catalogue. A plan sign-in cannot run in CI. A Claude plan runs through the same bundled Claude Code CLI as an Anthropic key, so the key's rows cover it.

It writes one row per harness, image, provider and model, with the provider's own words, to the job summary. On any failure, the run fails and opens or updates a single issue, "Model compatibility check failing". The first passing run closes it. A provider whose key is missing is reported as skipped, not failed, and the summary names the missing secrets. Keys are scrubbed from every detail. Every action is pinned by commit SHA, and zizmor and actionlint pass.

scripts/model-compat.test.ts covers the pure logic against real-world inputs: the picker's filter on real listings, the matrix, reading real harness streams (Claude Code's version refusal, Agno, CrewAI's snapshot replies, Mastra), the Codex version gate, and drift guards that fail if the Desktop gains a provider address or a harness is published that the check doesn't cover. Docs: docs/releasing.md, "Model compatibility".

Local run

I ran it on a throwaway Linux box with the Anthropic and OpenAI dev keys, the four jobs at a time. The whole matrix took 9 minutes, and each harness took 1 to 4 minutes, including its source build (about 80 seconds), so one GitHub runner per harness fits easily. Result: 78 failed, 942 passed, 34 skipped (no Gemini, xAI or Unbiased key).

Harness Passed Failed Skipped (no key)
agent-adk 88 0 3
agent-ag2 58 30 3
agent-agno 78 10 3
agent-antigravity 0 0 1
agent-claude-sdk 26 4 0
agent-crewai 88 0 3
agent-langgraph-agui (incl. 22 ChatGPT plan rows) 110 0 3
agent-langroid 54 34 3
agent-llamaindex 88 0 3
agent-mastra 88 0 3
agent-microsoft 88 0 3
agent-pydantic-ai 88 0 3
agent-strands 88 0 3

The failures, grouped (each on both latest and source):

Harness Provider Models Provider's answer
agent-claude-sdk Anthropic (key or Claude plan) claude-opus-5-5 API Error: 400 Claude Code 2.1.259 does not support this model; version 2.1.280 or newer is required.
agent-claude-sdk Anthropic (key or Claude plan) claude-fable-5-2 API Error: 400 Claude Code 2.1.259 does not support this model; version 2.1.290 or newer is required.
agent-ag2, agent-langroid OpenAI gpt-5.6-luna, -sol, -terra; gpt-6-astra, -luna, -sol; gpt-6.1-sol, -terra; gpt-6.2-astra, -sol 400: Function tools with reasoning_effort are not supported for this model in /v1/chat/completions. To use function tools, use /v1/responses or set reasoning_effort to 'none'.
agent-ag2, agent-langroid OpenAI gpt-5.2-pro, gpt-5.4-pro, gpt-5.5-pro, o3-pro 404: This is not a chat model and thus not supported in the v1/chat/completions endpoint.
agent-ag2, agent-langroid OpenAI gpt-5-pro 404: This model is only supported in v1/responses and not in v1/chat/completions.
agent-langroid OpenAI gpt-5.5 400: Unsupported value: 'temperature' does not support 0.2 with this model.
agent-langroid OpenAI o3 400: Unsupported parameter: 'parallel_tool_calls' is not supported with this model.
agent-agno Anthropic claude-haiku-4-5, claude-opus-4-5, claude-opus-4-6, claude-sonnet-4-5, claude-sonnet-4-6 400: temperature and top_p cannot both be specified for this model.

Every other harness, provider and model answered. The Claude Agent SDK rows are the refusal people hit today. The rest are harnesses whose request shape those models no longer accept, which the check now shows too.

Secrets needed

Add these repository secrets: ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY (Google, also used for Antigravity), XAI_API_KEY and UNBIASED_API_KEY. Until a secret is added, that provider's rows show as skipped. The workflow uses only the built-in token otherwise, with issues: write for the one issue.

The model picker lists each provider's models live, but each harness image pins its model client. A provider refusing a new model to an older client (Claude Code 2.1.259 for Claude Opus 5.5 and Fable 5.2) went unnoticed because nothing calls today's models through the published images.

A scheduled workflow runs scripts/model-compat.ts once per published harness: it lists each provider's models as the picker does, starts the harness image (the published latest and one built from source) on each provider with a key, sends one run per model, and checks the ChatGPT plan's Codex client version against Codex's catalogue. Failures fail the run and open or update one issue; a provider without a key is reported as skipped.
… Unbiased AI out

The client boundary test failed on scripts/model-compat.ts: its provider listings, harness health checks and Mastra's agent API are not OpenBot's server, so they are allowed as third-party. Unbiased AI is a gateway serving its own model, so the check no longer lists or runs it, and the Desktop drift test names it as left out on purpose.
@davidmckayv

Copy link
Copy Markdown
Contributor Author

Included in #1779, which carries this PR's commits and builds on them, so main already has this work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant