Repository navigation
Lock the LangGraph (TypeScript) engine before its first release, and check it on every provider - #1778
Closed
davidmckayv wants to merge 8 commits into
Closed
Lock the LangGraph (TypeScript) engine before its first release, and check it on every provider#1778davidmckayv wants to merge 8 commits into
davidmckayv wants to merge 8 commits into
Conversation
The model picker lists each provider's models live, but each harness image pins its model client. A provider refusing a new model to an older client (Claude Code 2.1.259 for Claude Opus 5.5 and Fable 5.2) went unnoticed because nothing calls today's models through the published images. A scheduled workflow runs scripts/model-compat.ts once per published harness: it lists each provider's models as the picker does, starts the harness image (the published latest and one built from source) on each provider with a key, sends one run per model, and checks the ChatGPT plan's Codex client version against Codex's catalogue. Failures fail the run and open or update one issue; a provider without a key is reported as skipped.
… Unbiased AI out The client boundary test failed on scripts/model-compat.ts: its provider listings, harness health checks and Mastra's agent API are not OpenBot's server, so they are allowed as third-party. Unbiased AI is a gateway serving its own model, so the check no longer lists or runs it, and the Desktop drift test names it as left out on purpose.
…rness Adds the model lock: shared/model-lock.json, written from the model check's source-image results by scripts/model-compat.ts lock, and the rule in shared/model-lock.ts that the model list applies. The daily check opens or updates one pull request when a model starts or stops answering; a pull request that changes a harness's client runs the check for that harness and fails when a verified model breaks. A conversation set to a model that isn't verified says so in the composer (ModelChoice.unverified).
…check it on every provider shared/model-lock.json names agent-langgraph-ts with empty pairs for Anthropic, OpenAI and Google, so nothing is offered on it until the model check has sent a model a real request through its own client and it answered, including on a provider CI has no key for. The check now runs every engine the lock names as well as the published harnesses (scripts/model-compat.ts harnesses, which both workflows' matrices use), so the engine is checked from its source image before it is published. The probe also reads the engine's LangChain and AG-UI client versions, under plain Node when an image has no Bun.
davidmckayv
requested review from
MikeRyanDev,
dustywusty,
guidovizoso,
mxmzb,
samjulien and
tylerslaton
as code owners
November 17, 2026 21:47
davidmckayv
force-pushed
the
feat/model-lock
branch
2 times, most recently
from
November 17, 2026 22:20
76b9ffd to
ab04444
Compare
…and test the lock rule on its own lock (#1793) The engine ships @langchain/google, which the probe did not read, so a run recorded no Gemini client for it. Two tests read the live model lock and could only pass while it was empty, so the first lock refresh after a real run would have failed them; they now build on an unrun lock of their own.
davidmckayv
force-pushed
the
feat/model-lock
branch
from
November 17, 2026 23:30
07bc282 to
c2351cd
Compare
Contributor
Author
|
Its commits are now part of #1779, which carries the model lock for this engine, so this one closes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
New Bots are moving to the LangGraph (TypeScript) engine,
agent-langgraph-ts. A model should be offered on that engine only once it has answered a real request sent through the engine's own client. This adds the engine to the model check and the model lock:shared/model-lock.jsonnamesagent-langgraph-tswith empty pairs for Anthropic, OpenAI and Google. Under the existing rule (lockVerdict), a locked pair offers only verified models, so nothing is offered on the engine until a run proves it. That includes a provider CI has no key for: its pair stays locked and empty rather than open.scripts/model-compat.tsnow runs every engine the lock names, as well as the published harnesses, as long asshared/harnesses.jsonhas its row. A newharnessessubcommand prints that list, and both workflows (model-compat.yml,model-lock.yml) build their matrix from it. Until the image is published, itslatestrows are inconclusive (the lock ignores them) andsourcebuilds fromagent-langgraph-ts/Dockerfile.model-lock.ymlalso triggers onagent-*/package-lock.json.@langchain/langgraph,@langchain/core,@langchain/anthropic,@langchain/openai,@langchain/google-genaiand the@ag-ui/*packages. It falls back tonode -ewhen an image has no Bun, and reads each manifest from the working directory up, so a package whoseexportsmap hidespackage.jsonis still read.any-provider, so it runs on Anthropic, OpenAI, Google and xAI (skipped without a key) with the environment the other any-provider harnesses get. Google goes through Gemini's OpenAI-compatible address, as the Python LangGraph harness is checked today.docs/releasing.md, "Model lock": how a new engine is locked before its first release.This branch doesn't add the engine's
shared/harnesses.jsonrow or its.github/published-images.jsonentry. Those come with the engine itself, onfeat/langgraph-ts.Tests
scripts/model-compat-langgraph-ts.test.ts, against the real listings inshared/fixtures/model-listings/:sourceare offered, refusals keep the provider's words, and a provider with no key stays locked and empty;exportsmap withoutpackage.json;harnesses.Red first: 6 of the 8 failed before the change. The other 2 pin behaviour
buildLockalready had. All 8 pass now.shared/model-lock.test.ts: the drift test "every engine the lock names is a harness OpenBot publishes" now lets an engine that hasn't run yet have no clients. It still requires ashared/harnesses.jsonrow, so on this branch it fails until the engine's row lands. With that row added locally, it passes andbun scripts/model-compat.ts harnesseslistsagent-langgraph-ts. Four other tests in that file already fail onfeat/model-lock, because its lock is still empty there. This change doesn't affect them.How to run
Real run
Not run yet. The engine's source (
agent-langgraph-ts/) isn't on any branch yet, so there was no image to build. The run is one command (above) on a Linux box with Docker once the engine is onfeat/langgraph-ts.