Skip to content

fix(pricing): prefer a native provider over aggregator re-listings - #1279

Open
samirhvbr wants to merge 1 commit into
usestrix:mainfrom
samirhvbr:fix/pricing-prefer-native-provider
Open

fix(pricing): prefer a native provider over aggregator re-listings#1279
samirhvbr wants to merge 1 commit into
usestrix:mainfrom
samirhvbr:fix/pricing-prefer-native-provider

Conversation

@samirhvbr

Copy link
Copy Markdown

Problem

resolve_litellm_model() resolves a bare model name that several LiteLLM providers list by returning the alphabetically first matching key when those listings agree on price. LiteLLM's model_cost has since grown aggregator re-listings that sort ahead of a model's first-party provider. With the pinned litellm==1.90.1, grok-4.5 now has three same-priced listings:

  • openrouter/x-ai/grok-4.5
  • perplexity/xai/grok-4.5
  • xai/grok-4.5

so the resolver returns openrouter/x-ai/grok-4.5 instead of xai/grok-4.5. The cost estimate is unaffected (identical price), but the reported provider is wrong — and tests/test_pricing.py::test_resolves_common_bare_model_names fails on main against the pinned litellm.

Fix

Prefer the first-party listing: the match whose key is exactly <litellm_provider>/<name>. When that match is unique, return it before falling back to price consensus. This restores native-provider attribution and also resolves names the price-consensus rule gave up on when aggregators disagree on price (a unique first-party listing among differently priced mirrors).

Tests

  • test_resolves_common_bare_model_names passes again against litellm 1.90.1.
  • Adds two deterministic, mock-backed regression tests so the behavior no longer rides on the live LiteLLM catalog.
  • ruff format, ruff check, mypy, and pytest tests/test_pricing.py all pass.

🤖 Generated with Claude Code

resolve_litellm_model() tie-breaks a bare model name that several LiteLLM
providers list by returning the alphabetically first key whenever those
listings agree on price. As LiteLLM's model_cost has grown, aggregators
that re-key a model under a namespaced path now sort ahead of the model's
own first-party provider: with litellm 1.90.1, "grok-4.5" resolves to
"openrouter/x-ai/grok-4.5" instead of "xai/grok-4.5". The estimate stays
correct (same price), but the reported provider is wrong, and it breaks
test_resolves_common_bare_model_names on the pinned litellm.

Prefer the first-party listing: the match whose key is exactly
"<litellm_provider>/<name>". When that is unique, return it before falling
back to price consensus. This restores native-provider attribution and
also resolves names the old rule gave up on when aggregators disagreed on
price (e.g. a unique first-party listing among differently priced mirrors).

Add deterministic, mock-backed regression tests for both cases so the
behavior no longer rides on live LiteLLM catalog data.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 6, 2026 14:57

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is narrowly scoped, improves determinism/provider attribution, and is covered by targeted regression tests.

Pull request overview

This PR fixes LiteLLM model-name resolution so that when a bare model name matches both aggregator re-listings and the model’s first-party provider listing, the resolver prefers the native provider key (e.g., xai/grok-4.5) rather than the alphabetically first aggregator key. This keeps provider attribution accurate and stabilizes resolution behavior as LiteLLM’s model_cost catalog grows.

Changes:

  • Update resolve_litellm_model() to prefer a unique first-party (“native”) "<litellm_provider>/<name>" match before falling back to the existing price-consensus tie-break.
  • Add two mock-backed regression tests covering native-vs-aggregator preference and the “unique native even when aggregators disagree on price” case.
File summaries
File Description
strix/report/pricing.py Adjusts model resolution logic to prioritize unique native provider listings over aggregator re-listings.
tests/test_pricing.py Adds deterministic tests that pin the resolver behavior for native-provider preference scenarios.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@greptile-apps

greptile-apps Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR updates LiteLLM model resolution to prefer a unique first-party provider listing over aggregator re-listings before applying price consensus.

  • Detects native listings using the catalog entry’s litellm_provider.
  • Preserves the existing price-consensus fallback when no unique native listing exists.
  • Adds deterministic regression coverage for equal-price and differing-price aggregator listings.

Confidence Score: 5/5

The PR appears safe to merge with no actionable correctness, security, or repository-rule violations identified.

The native-provider preference is applied only when exactly one matching catalog key identifies itself as the provider’s direct listing, while ambiguous cases retain the existing price-consensus behavior.

Important Files Changed

Filename Overview
strix/report/pricing.py Adds a narrowly scoped native-provider preference while retaining existing exact-match and consensus behavior.
tests/test_pricing.py Adds isolated regression tests that restore LiteLLM’s global catalog and clear the resolver cache.

Reviews (1): Last reviewed commit: "fix(pricing): prefer a native provider o..." | Re-trigger Greptile

samirhvbr added a commit to samirhvbr/strix that referenced this pull request Sep 6, 2026
…#1279)

Bump .fork-version to 1.5.3+shvia.14 and record the LiteLLM native-provider
resolution fix (cherry-picked from the upstream PR usestrix#1279 branch) in FORK.md.
Also realign version.md's "Versão atual" line, which was stale at shvia.12.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
samirhvbr added a commit to samirhvbr/strix that referenced this pull request Sep 6, 2026
Sync the fork line onto usestrix/strix v1.6.2 (was 1.5.3). Conflict
resolution and reconciliation of the fork's patches:

- warmup: adopt upstream's wait_for_import_warmup() approach and drop the
  fork's synchronous agents pre-import (_preimport_thread_unsafe_sdk), which
  upstream's own warmup rework supersedes. Remove the now-obsolete
  tests/test_warmup.py (upstream covers this in tests/test_import_warmup.py).
- runner: keep upstream's dedicated `except RateLimitError` block; narrow the
  fork's F3 usage-limit handler to the non-RateLimitError (LiteLLM) case so
  there is no dead code, and record the same "rate_limited" exit reason.
- viewer: rebuild the compiled frontend bundle (static/) from the merged
  source so the fork's local-PDF button rides on upstream's new frontend.
- test_viewer: adapt the historical-run-data test to the fork's session-only
  gating (the fork removed the upstream email-verification gate).

Full suite: 1740 passed (isolated HOME). The pricing native-provider fix
(shvia.14, PR usestrix#1279) is carried on top.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@itzzdev09

Copy link
Copy Markdown
Contributor

Drive-by review — the native-route preference is the right call, but the placement here widens the change well beyond #1288, and the existing guard test only looks like it still covers the old behavior.

The guard test passes by accident

test_resolver_does_not_guess_between_differently_priced_providers asserts that differently-priced routes resolve to None. It stays green under this patch only because its fixtures omit litellm_provider:

"provider-a/example": {"input_cost_per_token": 1.0, "output_cost_per_token": 2.0},

With no litellm_provider, the new comprehension compares key == "None/example", never matches, so native is empty and the price-consensus path runs as before.

In the real registry that never happens. Every entry carries the field:

litellm 1.100.0 — dict entries 3888 | with litellm_provider 3888 | missing: []

So in production the native list is populated for essentially every model, and returning native[0] before the price check means the differently-priced case no longer returns None. The behavior the test exists to protect is gone while the test still passes.

How much this actually moves

Sweeping the registry for bare names where main returns None (routes disagree on price) but this patch returns a native route:

121 bare model names change from None to a guess. For example:

gemma-7b-it -> groq/gemma-7b-it
  routes: anyscale/google/gemma-7b-it, fireworks_ai/accounts/fireworks/models/gemma-7b-it, groq/gemma-7b-it
  prices: (2e-07, 2e-07), (5e-08, 8e-08), (1.5e-07, 1.5e-07)

Those are ~4x apart on input cost, so the resolved route drives a materially different cost estimate. That may well be the behavior you want — a first-party listing is a better guess than nothing — but it is a separate decision from #1288, where the three grok-4.5 routes are priced identically and picking among them is free.

(Measured on litellm 1.100.0 rather than the 1.90.1 in uv.lock; the litellm_provider coverage and the shape of the divergence are not version-sensitive, though the exact count will be.)

Suggestion

Either:

  1. Move the native preference inside the len(matches) == 1 or len(prices) == 1 branch, so it only breaks ties the resolver was already willing to resolve — that fixes BUG: pricing resolver returns the aggregator route for equally priced models (test failing on main) #1288 with no behavior change anywhere else; or
  2. Keep it where it is, but give the guard test fixtures a real litellm_provider so it actually exercises the new path, and state the widened contract in the docstring: a unique first-party listing beats price consensus.

#1289 takes approach (1) on the same issue, so it's worth the two of you settling which contract you want rather than the placement being decided by merge order.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants