Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
160 changes: 160 additions & 0 deletions .github/workflows/model-compat.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
name: Model compatibility

# Does every model the picker offers still answer through the clients OpenBot ships? The picker
# lists each provider's models live, but every harness image carries its model client at a fixed
# version (the Claude Agent SDK harness bundles a Claude Code CLI, the LangGraph harness sends the
# ChatGPT plan's backend a fixed Codex client version, the others carry the provider SDKs they were
# built with), and a provider can refuse a new model to an older client. scripts/model-compat.ts
# starts each harness image, one job per harness, and sends it one request per listed model on
# every provider that has a key here. See docs/releasing.md, "Model compatibility".
#
# Keys are repository secrets, one per provider. A provider without one is reported as skipped,
# not failed. On any failure the run fails and opens (or updates) one issue titled "Model
# compatibility check failing"; a passing run closes it.
#
# Runs only on a schedule and by hand, on main: never on a pull request, so no fork's code ever
# runs with these keys.
on:
schedule:
- cron: "41 6 * * *"
workflow_dispatch:
inputs:
tags:
description: Harness images to check, comma-separated. `latest` is the published release people run; `source` is built from this commit, which is what the next release ships.
type: string
default: latest,source
required: false

permissions: {}

concurrency:
group: model-compat
cancel-in-progress: false

jobs:
harnesses:
name: List the published harnesses
if: github.repository == 'CopilotKit/OpenBot' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
outputs:
images: ${{ steps.list.outputs.images }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- id: list
run: echo "images=$(jq -c . .github/published-images.json)" >> "$GITHUB_OUTPUT"

check:
name: ${{ matrix.harness }}
needs: harnesses
runs-on: ubuntu-latest
timeout-minutes: 90
permissions:
contents: read
strategy:
fail-fast: false
matrix:
harness: ${{ fromJSON(needs.harnesses.outputs.images) }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version: 1.4.3
# Writes its results and exits 0 whatever they say: the report job below decides.
- name: One request per model through this harness
env:
HARNESS: ${{ matrix.harness }}
TAGS: ${{ inputs.tags || 'latest,source' }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
XAI_API_KEY: ${{ secrets.XAI_API_KEY }}
run: >-
bun scripts/model-compat.ts --harness "$HARNESS" --tags "$TAGS" --exit-zero
--out "$RUNNER_TEMP/model-compat-$HARNESS.json"
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: model-compat-${{ matrix.harness }}
path: ${{ runner.temp }}/model-compat-${{ matrix.harness }}.json
if-no-files-found: error
retention-days: 14

report:
name: Report
needs: [harnesses, check]
if: always() && needs.harnesses.result == 'success'
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write # open, update or close the one "Model compatibility check failing" issue
env:
GH_TOKEN: ${{ github.token }}
ISSUE_TITLE: Model compatibility check failing
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version: 1.4.3
- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
pattern: model-compat-*
path: ${{ runner.temp }}/results
merge-multiple: true
- name: Summarize
id: summary
run: |
set +e
shopt -s nullglob
bun scripts/model-compat.ts report --summary "$RUNNER_TEMP/summary.md" \
--issue "$RUNNER_TEMP/issue-table.md" "$RUNNER_TEMP"/results/*.json
code=$?
echo "code=$code" >> "$GITHUB_OUTPUT"
if [ -f "$RUNNER_TEMP/summary.md" ]; then cat "$RUNNER_TEMP/summary.md" >> "$GITHUB_STEP_SUMMARY"; fi
exit 0
- name: Open or update the issue
if: steps.summary.outputs.code != '0'
run: |
set -euo pipefail
body="$RUNNER_TEMP/issue.md"
{
echo "The model compatibility check failed on $(date -u +%Y-%m-%d): $RUN_URL"
echo
echo "A model refused by a pinned client means that client needs bumping: see docs/releasing.md, \"Model compatibility\". This issue is updated by every failing run and closed by the first passing one."
echo
if [ -f "$RUNNER_TEMP/issue-table.md" ]; then
cat "$RUNNER_TEMP/issue-table.md"
else
echo "No harness job wrote results. See the run."
fi
} > "$body"
number=$(gh issue list --repo "$GITHUB_REPOSITORY" --state open --search "\"$ISSUE_TITLE\" in:title" \
--json number,title --jq "map(select(.title == env.ISSUE_TITLE)) | .[0].number // empty")
if [ -n "$number" ]; then
gh issue edit "$number" --repo "$GITHUB_REPOSITORY" --body-file "$body"
else
gh issue create --repo "$GITHUB_REPOSITORY" --title "$ISSUE_TITLE" --body-file "$body"
fi
- name: Close the issue when everything passes
if: steps.summary.outputs.code == '0'
run: |
set -euo pipefail
number=$(gh issue list --repo "$GITHUB_REPOSITORY" --state open --search "\"$ISSUE_TITLE\" in:title" \
--json number,title --jq "map(select(.title == env.ISSUE_TITLE)) | .[0].number // empty")
if [ -n "$number" ]; then
gh issue close "$number" --repo "$GITHUB_REPOSITORY" --comment "Passing again: $RUN_URL"
fi
- name: Fail when a model failed
if: steps.summary.outputs.code != '0'
run: |
echo "::error::Model compatibility check failing. See the summary above and the issue titled \"$ISSUE_TITLE\"."
exit 1
6 changes: 6 additions & 0 deletions api/client-boundary.allow.ts
Original file line number Diff line number Diff line change
Expand Up @@ -392,6 +392,12 @@ export const ALLOWED: Allowed[] = [
rule: "third-party",
why: "image registries, never OpenBot's server",
},
{
file: "scripts/model-compat.ts",
what: /^(fetch\(\)|\/health|\/api\/agents\/openbot\/generate)$/,
rule: "third-party",
why: "model providers' listings and the harness images the model check starts on this machine, never OpenBot's server",
},
// Scripts that run and stop a server on this machine.
{
file: "scripts/update-local.ts",
Expand Down
55 changes: 55 additions & 0 deletions docs/releasing.md
Original file line number Diff line number Diff line change
Expand Up @@ -993,6 +993,61 @@ it when it is not. Signed desktop builds are the exception: they read the `macos
`windows-signing` environments described above and in [Windows signing](windows-signing.md).
Downloads sync signs in to AWS with GitHub's OIDC token rather than a stored key.

## Model compatibility

The model picker lists each provider's models live, but every harness image carries its model
client at a fixed version: the Claude Agent SDK harness bundles a Claude Code CLI (through
`claude-agent-sdk` in `agent-claude-sdk/requirements.txt`), the LangGraph harness sends the ChatGPT
plan's backend a fixed Codex client version (`DEFAULT_CLIENT_VERSION` in
`agent-langgraph-agui/src/chatgpt_models.py`), and the others carry the provider SDKs they were built
with. A provider can refuse a new model to an older client while the picker keeps offering it:
"Claude Code 2.1.259 does not support this model; version 2.1.290 or newer is required".

**Model compatibility** (`.github/workflows/model-compat.yml`) checks for that daily, and by hand
from the Actions tab. One job per harness in `.github/published-images.json` runs
[`scripts/model-compat.ts`](../scripts/model-compat.ts), which:

- lists each provider's models from the same source, with the same filter, as the picker (see
[The model list](configuration.md#the-model-list));
- starts the harness image on each provider the way the Desktop configures it, and sends one AG-UI
run per model, so the harness's own client makes the call. It checks `latest`, the published
image people run, and `source`, the image built from this commit;
- compares the LangGraph harness's Codex client version with the minimum each ChatGPT plan model
needs, from openai/codex's published model catalogue. A plan sign-in cannot run in CI. A Claude
plan runs through the same Claude Code CLI as an Anthropic key, so the key's rows cover it.

It writes a pass, fail or skipped row per harness, image, provider and model, with the provider's
own words, to the run's summary. On any failure the run fails and opens or updates one issue,
**Model compatibility check failing**, and the first passing run closes it. It runs only on a
schedule or by hand on `main`, never on a pull request.

Each provider needs its key as a repository secret: `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`,
`GEMINI_API_KEY` (Google, also used for Antigravity) and `XAI_API_KEY`. A provider without one is
reported as skipped, not failed, and the summary names the missing secrets. Unbiased AI is not
checked: it is a gateway serving its own model, not a provider's listing.

**When it fails**, read the provider's words in the table:

- A version refusal (a model "not supported" by the client's version, or a ChatGPT plan model that
needs a newer Codex) means bumping the pinned client: `claude-agent-sdk` in
`agent-claude-sdk/requirements.txt` (each release bundles a newer Claude Code CLI), or
`DEFAULT_CLIENT_VERSION` in `agent-langgraph-agui/src/chatgpt_models.py`, or the harness's
provider SDK in its `requirements.txt` or `package.json`. Run the check by hand: the `source` rows
show whether the bump fixed it before a release ships it, and the `latest` rows stay red until
one does.
- A refused parameter or endpoint (a model that takes tools only through `/v1/responses`, or
refuses a `temperature`) means that harness builds its request in a way the model no longer
accepts: fix how the harness sets up its model, or bump the framework if a newer one does.
- Billing, rate limits or a refused key are the key's problem, not a client's: fix the key.

To run it locally (needs Docker), with keys in the environment or an env file:

```sh
bun scripts/model-compat.ts --env-file .env --harness agent-claude-sdk --provider anthropic
```

`--model <regex>` narrows the models, `--tags latest` skips building from source.

## The one thing CI cannot do

The smoke journey in `tests/smoke` is the only check that proves the parts are wired to each other:
Expand Down
Loading
Loading