chore: refresh the models.dev catalog snapshot - #3324
Conversation
4fdd601 to
ba2b9de
Compare
ba2b9de to
31a9326
Compare
Regenerate the checked-in catalog from a single fetch of https://models.dev/api.json on 2026-08-20, sha256 da57b8e4281fd65d725ef7f2f73f681d052848c07737c4aaa1c707d91ad26e33. The digest records which payload produced this snapshot. models.dev serves a rolling document and the payload is not archived here, so this is a provenance record, not a reproducibility guarantee. Models 1690 -> 1779, deprecated 82 -> 98. Pricing gains 86 entries, drops 9, and reprices 37 — openrouter:~google/gemini-flash-latest input falls 1.5 -> 0.375, deepinfra:moonshotai/Kimi-K3 rises 2.7 -> 2.85. Three consumers had to follow, and nothing more. Upstream deprecated north-mini-code-free and laguna-s-2.1-free, which provider-registry validates at module load, so importing the registry threw. Removing the two ids is the response that validation exists to force. The four that remain are unchanged from main and are still checked for active and tool-capable on every import. They come from the set #1720 chose by transcribing "active, tool-capable, cost=0" out of the snapshot, pinned by hand only because the snapshot carries no cost field. Individual ids have had operational attention since — #1720 called big-pickle anonymously, #2166 replaced an unhealthy bootstrap model and probed the fallbacks — but the four have never been verified as a set, and this refresh adds none. It also adds no reachable model: it only drops two the upstream retired. gemini-3-pro-preview left the catalog, so the Google thinking test moves to gemini-3.1-pro-preview, whose effort values carry the same no-off shape the test asserts. The snapshot now sits about 2 KB under Biome's 1 MiB default, and this refresh alone added 57 KB, so the next one crosses it. An over-size file is skipped with a warning while format:check still exits 0, so crossing would quietly stop checking the snapshot rather than fail on it. The ceiling is raised for that one file through an override; a global raise would also disarm the guard for hand-written files, which is what it exists for. Generated-by: Claude Code
Google's fallback list is what the registry offers when protocol model discovery fails, so an id that no longer exists is worse than a shorter list: it presents a model with no metadata that the API will reject. gemini-2.0-flash left the catalog in the snapshot refresh and gemini-1.5-pro was already absent before it, leaving gemini-2.5-flash as the only entry the snapshot still knows. Whether one fallback is enough for Google, and which current models should join it, is a separate judgement this does not make. Three other providers carry fallback ids their own snapshot section does not list (anthropic, tencent-token-plan, zai-coding-plan); all predate this refresh and are left alone here. Generated-by: Claude Code
31a9326 to
8aa2935
Compare
hqhq1025
left a comment
There was a problem hiding this comment.
Requesting changes.
[P1] Migrate persisted targets for the models removed by this refresh.
This updates the fallback lists for new connections, but existing user catalogs are deliberately preserved by bootstrap-runtime-policy.ts:122-127. Their default target remains valid as long as the old id is still enabled (connection-catalog-document.ts:668-674), and Host admission trusts the persisted models[] row (connection-model-admission.ts:14-16). At this head I reproduced north-mini-code-free, laguna-s-2.1-free, and gemini-2.0-flash remaining visible as available/default-capable choices and being admitted for execution after upgrade. OpenCode Free has no model-list contract that can later repair the state; Google also preserves the old catalog when discovery fails. Please add a targeted upgrade migration/tombstone that removes these stale ids from persisted inventories and enabled sets and clears or repairs a matching default target, with base-state upgrade tests.
[P2] Filter newly deprecated ZenMux models out of the fallback authority.
The refreshed snapshot changes 13 ZenMux models from active to deprecated, but provider-registry.ts:234-241 still builds zenmuxModelIds without the lifecycle filter used by OpenRouter, OpenCode, and Ollama Cloud. Discovery then keeps only that fallback set (model-fetcher.ts:624-626), and the catalog marks the deprecated rows available and canUseAsChatDefault: true. I reproduced this for all 13 affected ids, including deepseek/deepseek-chat, stepfun/step-3, and five xAI ids. Filter deprecated metadata before constructing the ZenMux fallback list and add focused regression coverage.
Verified on 8aa2935a130be7b2a3cc0b0a0dbbcd6033797242: clean isolated gcr checkouts, Node 22.22.1/npm 11.19.0 install and postinstall, full build:test, core/runtime typechecks, 558 core tests, 70 targeted runtime tests, format, lint, generated metadata/pricing integrity checks, and the >1 MiB Biome override probe. The required test check is green.
Codex-assisted review performed under the maintainer-approved review workflow.
This refresh flips 15 ZenMux models from active to deprecated. Two carry a model-level protocol override and are excluded from the fallback source, so 13 reach it — `deepseek/deepseek-chat`, `stepfun/step-3`, five xAI ids, and six others. Discovery keeps only the fallback set, and the catalog marks whatever survives `available` and `canUseAsChatDefault`, so all 13 would be offered as usable choices for a provider that had none before this branch. `toolCallingModelIds` filters on tool-calling capability only; the providers that exclude deprecated ids do it at the call site. ZenMux now does the same. This is deliberately the narrow fix: it repairs what this refresh caused, which is the ZenMux count going from 0 to 13. It is not the whole defect. Six providers still ship deprecated ids in their fallback lists — 28 models across `togetherai`, `mistral`, `xiaomi`, `deepinfra`, `nvidia`, and `openai`, with `openai` writing its list by hand rather than through `toolCallingModelIds`. Reverting this refresh does not change any of them, so converging them is separate work, tracked in #3355. The new contract test records that set as an explicit boundary: a provider that regresses into it fails, and a listed provider that gets cleaned up without being removed from the list also fails, so the boundary shrinks as #3355 lands instead of going stale. Both directions were falsified before this commit: dropping the ZenMux filter fails two tests naming the 13 ids, and adding an already-clean provider to the boundary list fails the second. Generated-by: Claude Code
7e64456 to
8dca808
Compare
|
@hqhq1025 Thanks — both findings reproduce, and the ZenMux one is fixed in 8dca808. [P2] Fixed, narrowly. Confirmed: 15 ZenMux models flip active→deprecated in this refresh; 2 carry a model-level protocol override and are excluded from the fallback source, so 13 reach it. One correction to the scope: this is not a ZenMux-specific gap, it is the default. Six providers still ship deprecated ids in their fallback lists — 28 models: [P1] Real, but it predates this branch — filed as #3354 rather than fixed here. The chain reproduces as you describe, and What I could not find is a precedent for a refresh fixing it. #2050 removed The test I applied is whether reverting this PR removes the problem. For ZenMux it does — the 13 ids go back to active. For this one it does not: the three ids return to the fallback lists but remain retired upstream, so a persisted default still fails at call time. That makes it a standing gap this refresh surfaces rather than one it introduces, and closing it means building a model-catalog migration with upgrade-baseline tests — an intent that should not share a revert with a data refresh. Happy to take it as a follow-up if you would rather it block here. Verification on this head: 560 core tests, format, lint, core/runtime typechecks. Both directions of the new test were falsified before commit — dropping the ZenMux filter fails two tests naming the 13 ids, and adding an already-clean provider to the boundary list fails the second. AI-assisted review response: Claude Code reproduced both findings, measured the six-provider scope, and drafted this reply; the scoping decision and the #2050 precedent check were verified by me. |
hqhq1025
left a comment
There was a problem hiding this comment.
Reviewed the complete revision at 8dca808f45575c6c8508690491dba118c638cdee, including the generated catalog and pricing deltas, fallback lifecycle behavior, the ZenMux correction, the author's response, and the prior requested-changes disposition.
The persisted-catalog issue is real but pre-existing: existing catalogs retain the same stale rows and admission behavior on both main and this head, while this revision prevents new catalogs from acquiring the retired fallback ids. The broader migration belongs in #3354 and is not a blocker for this data refresh. No material actionable finding remains under policy v7.
The required check passes, review-thread state is complete with no unresolved threads, and 630 focused core/runtime tests plus build:test, format, and lint passed locally.
Codex-assisted review performed under the maintainer-approved review workflow.
Summary
Refresh the checked-in models.dev catalog snapshot, and let the two consumers that had to follow, follow.
The snapshot comes from a single fetch of
https://models.dev/api.jsonon 2026-08-20, sha256da57b8e4281fd65d725ef7f2f73f681d052848c07737c4aaa1c707d91ad26e33. The digest records which payload produced this snapshot; models.dev serves a rolling document and the payload is not archived here, so this is a provenance record, not a reproducibility guarantee. Archiving it is tracked with the rest of the models.dev provenance work under #3270.What moved: models 1690 → 1779, deprecated 82 → 98. Pricing gains 86 entries, drops 9, and reprices 37 —
openrouter:~google/gemini-flash-latestinput falls 1.5 → 0.375,deepinfra:moonshotai/Kimi-K3rises 2.7 → 2.85.Three consumers had to follow:
north-mini-code-freeandlaguna-s-2.1-free, whichprovider-registry.tsvalidates at module load, so importing the registry threw. Removing the two ids is the response that validation exists to force. The four that remain are unchanged frommainand are still checked for active and tool-capable on every import.gemini-3-pro-previewleft the catalog, so the Google thinking test moves togemini-3.1-pro-preview, whose effort values carry the same no-offshape the test asserts.format:checkstill exits 0 — crossing would quietly stop checking the snapshot rather than fail on it. The ceiling is raised for that one file through an override; a global raise would also disarm the guard for hand-written files, which is what it exists for.Refs #3270 — the models.dev attribution and snapshot provenance for the G3 release legal-files gate land separately and do not depend on this.
Verification
npm run format:check,npm run lint— cleannpm --workspace @maka/core run typecheck,npm --workspace @maka/runtime run typecheck— cleannode --test --test-concurrency=4 packages/core/dist/**/*.test.js— 566 pass, 0 failopencode-free-anonymous,claude-subscription-runtime,responses-wire-contract,model-factory-thinking,cost) — 70 pass, 0 failReview focus
hy3-free,nemotron-3.5-lightning-freeandmuse-spark-1.2-contributor-freeto keyless users. Two adversarial reviews rejected that: a zero catalog price is not an anonymous entitlement, and OpenCode Zen documents both zero-price models and a "log in, add billing, copy an API key" access path.big-pickleanonymously; fix(opencode-free): replace unhealthy bootstrap model #2166 replaced an unhealthy bootstrap model and probed the fallbacks), and the one lifecycle edit, fix: derive thinking-strength controls from models.dev reasoning_options #2050, droppedhy3-freefor an upstream deprecation rather than a failed call — but the four have never been verified as a set, and this refresh adds no verification. Keeping them is the no-behaviour-change option for a data refresh; establishing entitlement, and deciding whether the list should follow the snapshot once it is established, belongs in its own change with that evidence.anthropic,tencent-token-plan,zai-coding-plan). All predate this refresh and are untouched; the Google entry changes only because this refresh is what retiredgemini-2.0-flash.gemini-2.5-flashis the only entry the snapshot still knows. Which current models should join it is a judgement this PR does not make.AI use
Select exactly one:
Tool(s) and scope: Claude Code (Opus 5) regenerated the snapshot, made the consumer changes, and drafted this description. Two further models (Fable, and Codex through a review harness) were run as adversarial reviewers over two rounds; their findings removed the snapshot-derived offering described above and corrected two factual claims in an earlier revision of this description. AI review is not independent human review. The human contributor reviewed the diff and the commit messages and owns the merge decision. Commits carry
Generated-by: Claude Code.Checklist
Does this PR entail a change in behavior?