feat: drive reasoning variants from models.dev reasoning_options#31570
Open
rekram1-node wants to merge 1 commit into
Open
feat: drive reasoning variants from models.dev reasoning_options#31570rekram1-node wants to merge 1 commit into
rekram1-node wants to merge 1 commit into
Conversation
Parse the curated reasoning_options field from models.dev api.json and use its effort values to generate reasoning variants instead of the hardcoded per-package tables, in both the v1 provider catalog and the v2 catalog plugin. - core: ModelsDev.ReasoningOption discriminated union (toggle | effort | budget_tokens); effort values stay open strings and unknown option types are tolerated since api.json is cast, not decoded - core: ReasoningVariants shared per-package effort encoder used by v1 ProviderTransform.variants and the v2 ModelsDevPlugin - core: v2 catalog generates effort variants from reasoning_options; curated experimental modes win id collisions; anthropic profile gains the effort semantic - llm: anthropic protocol supports adaptive thinking, lowers effort to output_config.effort, and sends the effort-2025-11-24 beta header - opencode: resolved Provider.Model carries capabilities.reasoningOptions (unknown types and null effort values dropped at the mapping boundary); config models accept reasoning_options; models without usable effort data fall back to the hardcoded tables unchanged Catalog-wide audit vs live api.json: 2975 models byte-identical, 38 diffs, all data correcting stale hardcoded effort lists.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
models.dev now curates a
reasoning_optionsfield per provider model (431 models live). This PR parses it and uses its effort values to generate reasoning variants instead of the ~500-line hardcoded per-package tables — in both the v1 provider catalog and the v2 catalog plugin.How
Schema (forward-compatible by design)
ModelsDev.ReasoningOptionis a discriminated union mirroring models.dev:toggle | effort | budget_tokens. Effortvaluesare open strings (tiers likexhighwere added over time) and budget bounds are optional.api.jsonis cast, never decoded, so unknown future option types cannot fail parsing; they are filtered at the single mapping boundary into the resolved catalog, so schema-validated surfaces (HTTP/SDK/openapi) only ever see known variants.ConfigProviderV1.Model) accept the same union; null effort values are stripped when mapping so the resolvedcapabilities.reasoningOptionscontract matches its generated SDK type exactly.v1 (
ProviderTransform.variants)ReasoningVariantsin core). Models without usable effort data (toggle/budget-only, absent) fall back to the hardcoded tables unchanged.v2
ModelsDevPlugingenerates effort variants fromreasoning_optionsalongsideexperimental.modes(curated modes win id collisions), partitioned throughModelRequest.normalizeAiSdkOptions.effortsemantic, and the nativeAnthropicMessagesprotocol now supports adaptive thinking (thinking: {type: "adaptive", display?}), lowersefforttooutput_config.effort(wire shape verified against@ai-sdk/anthropic), and conditionally sends theanthropic-beta: effort-2025-11-24header. Without this, anthropic effort variants would be dead on arrival in v2.effortaisdk-shaped so it re-partitions into the semantic option — more correct than the old rawoutput_config.effortlowering, which never sent the required beta header.Verification
api.json(old vs new variants for all ~3000 reasoning models): 2975 byte-identical, 0 encoding changes, 38 diffs — every diff is curated data correcting stale hardcoding (e.g.vercel/openai/o1,o3losing bogusnone/minimal/xhigh, deepseek-v4 narrowing to[high,max],grok-4.3/mistral/fireworks gaining missing variants,gemini-3-prodropping unsupportedmedium). Anthropic, OpenAI, Azure, Copilot, OpenRouter, and SAP produce byte-identical output from data vs hardcoded.dev, verified by stashing).Not in scope
ModelV2.Capabilitiesstill carries no reasoning fields — variants are the selection surface in v2 (Ref.variant), so nothing needed it yet.budget_tokensdata is parsed and carried but does not yet drive budget-style variants; those still come from the hardcoded fallback (current data matches the hardcoded budgets exactly).