Skip to content

Lock the LangGraph (TypeScript) engine before its first release, and check it on every provider - #1778

Closed
davidmckayv wants to merge 8 commits into
feat/model-lockfrom
feat/model-lock-langgraph-ts
Closed

davidmckayv wants to merge 8 commits into
feat/model-lockfrom
feat/model-lock-langgraph-ts

Conversation

@davidmckayv

Copy link
Copy Markdown
Contributor

What

New Bots are moving to the LangGraph (TypeScript) engine, agent-langgraph-ts. A model should be offered on that engine only once it has answered a real request sent through the engine's own client. This adds the engine to the model check and the model lock:

  • Locked before its first release. shared/model-lock.json names agent-langgraph-ts with empty pairs for Anthropic, OpenAI and Google. Under the existing rule (lockVerdict), a locked pair offers only verified models, so nothing is offered on the engine until a run proves it. That includes a provider CI has no key for: its pair stays locked and empty rather than open.
  • Checked from before it is published. scripts/model-compat.ts now runs every engine the lock names, as well as the published harnesses, as long as shared/harnesses.json has its row. A new harnesses subcommand prints that list, and both workflows (model-compat.yml, model-lock.yml) build their matrix from it. Until the image is published, its latest rows are inconclusive (the lock ignores them) and source builds from agent-langgraph-ts/Dockerfile. model-lock.yml also triggers on agent-*/package-lock.json.
  • Its clients are recorded. The probe reads @langchain/langgraph, @langchain/core, @langchain/anthropic, @langchain/openai, @langchain/google-genai and the @ag-ui/* packages. It falls back to node -e when an image has no Bun, and reads each manifest from the working directory up, so a package whose exports map hides package.json is still read.
  • Providers. The engine's credential is any-provider, so it runs on Anthropic, OpenAI, Google and xAI (skipped without a key) with the environment the other any-provider harnesses get. Google goes through Gemini's OpenAI-compatible address, as the Python LangGraph harness is checked today.
  • docs/releasing.md, "Model lock": how a new engine is locked before its first release.

This branch doesn't add the engine's shared/harnesses.json row or its .github/published-images.json entry. Those come with the engine itself, on feat/langgraph-ts.

Tests

scripts/model-compat-langgraph-ts.test.ts, against the real listings in shared/fixtures/model-listings/:

  • before any run, no Anthropic, OpenAI or Gemini model is offered on langgraph-ts;
  • after a run, exactly the models that answered on source are offered, refusals keep the provider's words, and a provider with no key stays locked and empty;
  • an inconclusive failure (timeout, harness didn't start, image not published, stream cut) never verifies a model, and never takes one away;
  • an engine the lock names is checked before it is published, and a lock entry with no harness row isn't run;
  • the engine gets one run per listed model on each provider;
  • the client versions are read under plain Node from an image's working directory, including a hoisted package and an exports map without package.json;
  • both workflows take their matrix from harnesses.

Red first: 6 of the 8 failed before the change. The other 2 pin behaviour buildLock already had. All 8 pass now.

shared/model-lock.test.ts: the drift test "every engine the lock names is a harness OpenBot publishes" now lets an engine that hasn't run yet have no clients. It still requires a shared/harnesses.json row, so on this branch it fails until the engine's row lands. With that row added locally, it passes and bun scripts/model-compat.ts harnesses lists agent-langgraph-ts. Four other tests in that file already fail on feat/model-lock, because its lock is still empty there. This change doesn't affect them.

How to run

bun scripts/model-compat.ts harnesses
bun scripts/model-compat.ts --harness agent-langgraph-ts --tags source --env-file .env --out langgraph-ts.json
bun scripts/model-compat.ts lock langgraph-ts.json

Real run

Not run yet. The engine's source (agent-langgraph-ts/) isn't on any branch yet, so there was no image to build. The run is one command (above) on a Linux box with Docker once the engine is on feat/langgraph-ts.

The model picker lists each provider's models live, but each harness image pins its model client. A provider refusing a new model to an older client (Claude Code 2.1.259 for Claude Opus 5.5 and Fable 5.2) went unnoticed because nothing calls today's models through the published images.

A scheduled workflow runs scripts/model-compat.ts once per published harness: it lists each provider's models as the picker does, starts the harness image (the published latest and one built from source) on each provider with a key, sends one run per model, and checks the ChatGPT plan's Codex client version against Codex's catalogue. Failures fail the run and open or update one issue; a provider without a key is reported as skipped.
… Unbiased AI out

The client boundary test failed on scripts/model-compat.ts: its provider listings, harness health checks and Mastra's agent API are not OpenBot's server, so they are allowed as third-party. Unbiased AI is a gateway serving its own model, so the check no longer lists or runs it, and the Desktop drift test names it as left out on purpose.
…rness

Adds the model lock: shared/model-lock.json, written from the model check's source-image results by scripts/model-compat.ts lock, and the rule in shared/model-lock.ts that the model list applies. The daily check opens or updates one pull request when a model starts or stops answering; a pull request that changes a harness's client runs the check for that harness and fails when a verified model breaks. A conversation set to a model that isn't verified says so in the composer (ModelChoice.unverified).
…check it on every provider

shared/model-lock.json names agent-langgraph-ts with empty pairs for Anthropic, OpenAI and Google, so nothing is offered on it until the model check has sent a model a real request through its own client and it answered, including on a provider CI has no key for. The check now runs every engine the lock names as well as the published harnesses (scripts/model-compat.ts harnesses, which both workflows' matrices use), so the engine is checked from its source image before it is published. The probe also reads the engine's LangChain and AG-UI client versions, under plain Node when an image has no Bun.
…and test the lock rule on its own lock (#1793)

The engine ships @langchain/google, which the probe did not read, so a run recorded no Gemini client for it. Two tests read the live model lock and could only pass while it was empty, so the first lock refresh after a real run would have failed them; they now build on an unrun lock of their own.
@davidmckayv

Copy link
Copy Markdown
Contributor Author

Its commits are now part of #1779, which carries the model lock for this engine, so this one closes.

@davidmckayv
davidmckayv deleted the feat/model-lock-langgraph-ts branch November 17, 2026 23:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant