Skip to content

chore: refresh the models.dev catalog snapshot - #3324

Merged
Astro-Han merged 3 commits into
mainfrom
chore/models-dev-refresh-catalog-snapshot
Aug 20, 2026
Merged

chore: refresh the models.dev catalog snapshot#3324
Astro-Han merged 3 commits into
mainfrom
chore/models-dev-refresh-catalog-snapshot

Conversation

@Astro-Han

@Astro-Han Astro-Han commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Refresh the checked-in models.dev catalog snapshot, and let the two consumers that had to follow, follow.

The snapshot comes from a single fetch of https://models.dev/api.json on 2026-08-20, sha256 da57b8e4281fd65d725ef7f2f73f681d052848c07737c4aaa1c707d91ad26e33. The digest records which payload produced this snapshot; models.dev serves a rolling document and the payload is not archived here, so this is a provenance record, not a reproducibility guarantee. Archiving it is tracked with the rest of the models.dev provenance work under #3270.

What moved: models 1690 → 1779, deprecated 82 → 98. Pricing gains 86 entries, drops 9, and reprices 37 — openrouter:~google/gemini-flash-latest input falls 1.5 → 0.375, deepinfra:moonshotai/Kimi-K3 rises 2.7 → 2.85.

Three consumers had to follow:

  • Upstream deprecated north-mini-code-free and laguna-s-2.1-free, which provider-registry.ts validates at module load, so importing the registry threw. Removing the two ids is the response that validation exists to force. The four that remain are unchanged from main and are still checked for active and tool-capable on every import.
  • gemini-3-pro-preview left the catalog, so the Google thinking test moves to gemini-3.1-pro-preview, whose effort values carry the same no-off shape the test asserts.
  • The snapshot now sits about 2 KB under Biome's 1 MiB default, and this refresh alone added 57 KB, so the next one crosses it. An over-size file is skipped with a warning while format:check still exits 0 — crossing would quietly stop checking the snapshot rather than fail on it. The ceiling is raised for that one file through an override; a global raise would also disarm the guard for hand-written files, which is what it exists for.

Refs #3270 — the models.dev attribution and snapshot provenance for the G3 release legal-files gate land separately and do not depend on this.

Verification

  • npm run format:check, npm run lint — clean
  • npm --workspace @maka/core run typecheck, npm --workspace @maka/runtime run typecheck — clean
  • node --test --test-concurrency=4 packages/core/dist/**/*.test.js — 566 pass, 0 fail
  • The registry- and pricing-dependent runtime suites (opencode-free-anonymous, claude-subscription-runtime, responses-wire-contract, model-factory-thinking, cost) — 70 pass, 0 fail
  • Not run: the full repository suite, desktop suites, packaging.

Review focus

  • 37 repriced models feed cost computation directly. The rows are upstream facts, but this is the part of the diff with user-visible money consequences.
  • This does not change which models a user can reach, except by removing two the upstream retired. The anonymous free offering goes from six ids to four; no model is added. An earlier revision of this PR instead derived the offering from the snapshot — any model published at a zero rate, active and tool-capable — which would have auto-shipped hy3-free, nemotron-3.5-lightning-free and muse-spark-1.2-contributor-free to keyless users. Two adversarial reviews rejected that: a zero catalog price is not an anonymous entitlement, and OpenCode Zen documents both zero-price models and a "log in, add billing, copy an API key" access path.
  • The four surviving pins are not evidence of anonymous entitlement either, and this PR does not claim they are. They come from the set feat: add opencode-free as a zero-credential default provider #1720 chose by transcribing the same rule out of the snapshot — "active, tool-capable, cost=0", pinned by hand only because the snapshot carries no cost field. Individual ids have had operational attention since (feat: add opencode-free as a zero-credential default provider #1720 called big-pickle anonymously; fix(opencode-free): replace unhealthy bootstrap model #2166 replaced an unhealthy bootstrap model and probed the fallbacks), and the one lifecycle edit, fix: derive thinking-strength controls from models.dev reasoning_options #2050, dropped hy3-free for an upstream deprecation rather than a failed call — but the four have never been verified as a set, and this refresh adds no verification. Keeping them is the no-behaviour-change option for a data refresh; establishing entitlement, and deciding whether the list should follow the snapshot once it is established, belongs in its own change with that evidence.
  • Three other providers list fallback ids their own snapshot section does not contain (anthropic, tencent-token-plan, zai-coding-plan). All predate this refresh and are untouched; the Google entry changes only because this refresh is what retired gemini-2.0-flash.
  • Google is left with one fallback. gemini-2.5-flash is the only entry the snapshot still knows. Which current models should join it is a judgement this PR does not make.

AI use

Select exactly one:

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: Claude Code (Opus 5) regenerated the snapshot, made the consumer changes, and drafted this description. Two further models (Fable, and Codex through a review harness) were run as adversarial reviewers over two rounds; their findings removed the snapshot-derived offering described above and corrected two factual claims in an earlier revision of this description. AI review is not independent human review. The human contributor reviewed the diff and the commit messages and owns the merge decision. Commits carry Generated-by: Claude Code.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No

@Astro-Han
Astro-Han force-pushed the chore/models-dev-refresh-catalog-snapshot branch from 4fdd601 to ba2b9de Compare August 20, 2026 13:47
@Astro-Han Astro-Han changed the title feat(core): derive the OpenCode Free model set from the models.dev snapshot chore: refresh the models.dev catalog snapshot Aug 20, 2026
@Astro-Han
Astro-Han marked this pull request as ready for review August 20, 2026 13:47
@Astro-Han
Astro-Han marked this pull request as draft August 20, 2026 14:15
@Astro-Han
Astro-Han force-pushed the chore/models-dev-refresh-catalog-snapshot branch from ba2b9de to 31a9326 Compare August 20, 2026 14:28
Regenerate the checked-in catalog from a single fetch of
https://models.dev/api.json on 2026-08-20, sha256
da57b8e4281fd65d725ef7f2f73f681d052848c07737c4aaa1c707d91ad26e33. The digest
records which payload produced this snapshot. models.dev serves a rolling
document and the payload is not archived here, so this is a provenance record,
not a reproducibility guarantee.

Models 1690 -> 1779, deprecated 82 -> 98. Pricing gains 86 entries, drops 9, and
reprices 37 — openrouter:~google/gemini-flash-latest input falls 1.5 -> 0.375,
deepinfra:moonshotai/Kimi-K3 rises 2.7 -> 2.85.

Three consumers had to follow, and nothing more.

Upstream deprecated north-mini-code-free and laguna-s-2.1-free, which
provider-registry validates at module load, so importing the registry threw.
Removing the two ids is the response that validation exists to force. The four
that remain are unchanged from main and are still checked for active and
tool-capable on every import. They come from the set #1720 chose by transcribing
"active, tool-capable, cost=0" out of the snapshot, pinned by hand only because
the snapshot carries no cost field. Individual ids have had operational
attention since — #1720 called big-pickle anonymously, #2166 replaced an
unhealthy bootstrap model and probed the fallbacks — but the four have never
been verified as a set, and this refresh adds none. It also adds no reachable
model: it only drops two the upstream retired.

gemini-3-pro-preview left the catalog, so the Google thinking test moves to
gemini-3.1-pro-preview, whose effort values carry the same no-off shape the test
asserts.

The snapshot now sits about 2 KB under Biome's 1 MiB default, and this refresh
alone added 57 KB, so the next one crosses it. An over-size file is skipped with
a warning while format:check still exits 0, so crossing would quietly stop
checking the snapshot rather than fail on it. The ceiling is raised for that one
file through an override; a global raise would also disarm the guard for
hand-written files, which is what it exists for.

Generated-by: Claude Code
Google's fallback list is what the registry offers when protocol model
discovery fails, so an id that no longer exists is worse than a shorter list:
it presents a model with no metadata that the API will reject. gemini-2.0-flash
left the catalog in the snapshot refresh and gemini-1.5-pro was already absent
before it, leaving gemini-2.5-flash as the only entry the snapshot still knows.

Whether one fallback is enough for Google, and which current models should join
it, is a separate judgement this does not make. Three other providers carry
fallback ids their own snapshot section does not list (anthropic,
tencent-token-plan, zai-coding-plan); all predate this refresh and are left
alone here.

Generated-by: Claude Code
@Astro-Han
Astro-Han force-pushed the chore/models-dev-refresh-catalog-snapshot branch from 31a9326 to 8aa2935 Compare August 20, 2026 14:44
@Astro-Han
Astro-Han marked this pull request as ready for review August 20, 2026 14:47

@hqhq1025 hqhq1025 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes.

[P1] Migrate persisted targets for the models removed by this refresh.

This updates the fallback lists for new connections, but existing user catalogs are deliberately preserved by bootstrap-runtime-policy.ts:122-127. Their default target remains valid as long as the old id is still enabled (connection-catalog-document.ts:668-674), and Host admission trusts the persisted models[] row (connection-model-admission.ts:14-16). At this head I reproduced north-mini-code-free, laguna-s-2.1-free, and gemini-2.0-flash remaining visible as available/default-capable choices and being admitted for execution after upgrade. OpenCode Free has no model-list contract that can later repair the state; Google also preserves the old catalog when discovery fails. Please add a targeted upgrade migration/tombstone that removes these stale ids from persisted inventories and enabled sets and clears or repairs a matching default target, with base-state upgrade tests.

[P2] Filter newly deprecated ZenMux models out of the fallback authority.

The refreshed snapshot changes 13 ZenMux models from active to deprecated, but provider-registry.ts:234-241 still builds zenmuxModelIds without the lifecycle filter used by OpenRouter, OpenCode, and Ollama Cloud. Discovery then keeps only that fallback set (model-fetcher.ts:624-626), and the catalog marks the deprecated rows available and canUseAsChatDefault: true. I reproduced this for all 13 affected ids, including deepseek/deepseek-chat, stepfun/step-3, and five xAI ids. Filter deprecated metadata before constructing the ZenMux fallback list and add focused regression coverage.

Verified on 8aa2935a130be7b2a3cc0b0a0dbbcd6033797242: clean isolated gcr checkouts, Node 22.22.1/npm 11.19.0 install and postinstall, full build:test, core/runtime typechecks, 558 core tests, 70 targeted runtime tests, format, lint, generated metadata/pricing integrity checks, and the >1 MiB Biome override probe. The required test check is green.

Codex-assisted review performed under the maintainer-approved review workflow.

This refresh flips 15 ZenMux models from active to deprecated. Two carry a
model-level protocol override and are excluded from the fallback source, so 13
reach it — `deepseek/deepseek-chat`, `stepfun/step-3`, five xAI ids, and six
others. Discovery keeps only the fallback set, and the catalog marks whatever
survives `available` and `canUseAsChatDefault`, so all 13 would be offered as
usable choices for a provider that had none before this branch.

`toolCallingModelIds` filters on tool-calling capability only; the providers
that exclude deprecated ids do it at the call site. ZenMux now does the same.
This is deliberately the narrow fix: it repairs what this refresh caused, which
is the ZenMux count going from 0 to 13.

It is not the whole defect. Six providers still ship deprecated ids in their
fallback lists — 28 models across `togetherai`, `mistral`, `xiaomi`,
`deepinfra`, `nvidia`, and `openai`, with `openai` writing its list by hand
rather than through `toolCallingModelIds`. Reverting this refresh does not
change any of them, so converging them is separate work, tracked in #3355. The
new contract test records that set as an explicit boundary: a provider that
regresses into it fails, and a listed provider that gets cleaned up without
being removed from the list also fails, so the boundary shrinks as #3355 lands
instead of going stale.

Both directions were falsified before this commit: dropping the ZenMux filter
fails two tests naming the 13 ids, and adding an already-clean provider to the
boundary list fails the second.

Generated-by: Claude Code
@Astro-Han
Astro-Han force-pushed the chore/models-dev-refresh-catalog-snapshot branch from 7e64456 to 8dca808 Compare August 20, 2026 17:48
@Astro-Han

Copy link
Copy Markdown
Contributor Author

@hqhq1025 Thanks — both findings reproduce, and the ZenMux one is fixed in 8dca808.

[P2] Fixed, narrowly. Confirmed: 15 ZenMux models flip active→deprecated in this refresh; 2 carry a model-level protocol override and are excluded from the fallback source, so 13 reach it. toolCallingModelIds filters on tool-calling capability only, and the providers that exclude deprecated ids do it at the call site. ZenMux now does the same.

One correction to the scope: this is not a ZenMux-specific gap, it is the default. Six providers still ship deprecated ids in their fallback lists — 28 models: togetherai (11), mistral (8), xiaomi (3), deepinfra (3), nvidia (2), openai (1, gpt-4-turbo). openai writes its list by hand, so converging every list onto the helper would not cover it either. Reverting this refresh changes none of them, so I kept this PR to what the refresh caused — ZenMux going 0 → 13 — and filed the rest as #3355. The new contract test records those six as an explicit boundary and fails both when a provider regresses into it and when a listed provider is cleaned up without being removed from the list, so it shrinks as #3355 lands instead of going stale.

[P1] Real, but it predates this branch — filed as #3354 rather than fixed here. The chain reproduces as you describe, and opencode-free is the worst case: modelDiscovery: { kind: 'fallback' }, whose own reason string states the anonymous endpoint has no model-list contract, so there is no later discovery pass to repair the state at all — unlike Google, where the window is only a failed discovery.

What I could not find is a precedent for a refresh fixing it. #2050 removed hy3-free from the same OpenCode Free list for the same reason — an upstream deprecation — and touched no persistence: git show 41a983804 --name-only matches zero files under packages/storage/ or packages/runtime-host/. There is no model-catalog migration or tombstone mechanism; the tombstones in sqlite-session-metadata-store.ts serve session metadata only.

The test I applied is whether reverting this PR removes the problem. For ZenMux it does — the 13 ids go back to active. For this one it does not: the three ids return to the fallback lists but remain retired upstream, so a persisted default still fails at call time. That makes it a standing gap this refresh surfaces rather than one it introduces, and closing it means building a model-catalog migration with upgrade-baseline tests — an intent that should not share a revert with a data refresh.

Happy to take it as a follow-up if you would rather it block here.

Verification on this head: 560 core tests, format, lint, core/runtime typechecks. Both directions of the new test were falsified before commit — dropping the ZenMux filter fails two tests naming the 13 ids, and adding an already-clean provider to the boundary list fails the second.

AI-assisted review response: Claude Code reproduced both findings, measured the six-provider scope, and drafted this reply; the scoping decision and the #2050 precedent check were verified by me.

@hqhq1025 hqhq1025 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the complete revision at 8dca808f45575c6c8508690491dba118c638cdee, including the generated catalog and pricing deltas, fallback lifecycle behavior, the ZenMux correction, the author's response, and the prior requested-changes disposition.

The persisted-catalog issue is real but pre-existing: existing catalogs retain the same stale rows and admission behavior on both main and this head, while this revision prevents new catalogs from acquiring the retired fallback ids. The broader migration belongs in #3354 and is not a blocker for this data refresh. No material actionable finding remains under policy v7.

The required check passes, review-thread state is complete with no unresolved threads, and 630 focused core/runtime tests plus build:test, format, and lint passed locally.

Codex-assisted review performed under the maintainer-approved review workflow.

@Astro-Han
Astro-Han merged commit 410addd into main Aug 20, 2026
1 check passed
@Astro-Han
Astro-Han deleted the chore/models-dev-refresh-catalog-snapshot branch August 20, 2026 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants