diff --git a/CHANGELOG.md b/CHANGELOG.md index 56ba9f5b..3e255434 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,28 @@ User-visible changes to Imp are recorded here. +## Unreleased + +### Fixed + +- Breaking: `Imp.Core.LMResponse.cost`, and the `:cost` on a + `:model_response` event, is what the provider reported charging, as its + docs said, and `nil` when the provider reported no charge. It was ReqLLM's + catalog estimate for every model in ReqLLM's catalog, so every spend total + built on it counted an estimate as money spent, and OpenRouter's own charge, + which OpenRouter responses carry unasked, was overridden by it. OpenRouter + calls now report OpenRouter's charge; one made with the caller's own + provider key reports OpenRouter's fee plus the upstream charge, or `nil` + when the upstream charge is missing. A call to any catalog-priced provider + that reports no charge (Anthropic, OpenAI, Google, Groq, xAI and others) + now has a `nil` cost where it had the estimate. The estimate is the new `estimated_cost` field on + `Imp.Core.LMResponse` and `:estimated_cost` on the `:model_response` event; + `billing` is the breakdown behind that estimate, as it always was, and its + docs now say so. Migration: a host that sums `cost` treats `nil` as an + unknown charge, not a free one; a host that wants the old number for calls + with no reported charge reads `estimated_cost` for them, knowing it is an + estimate. + ## 0.6.0 — 2026-09-28 ### Security diff --git a/decisions.md b/decisions.md index 08d27a7c..601dfce1 100644 --- a/decisions.md +++ b/decisions.md @@ -8,6 +8,7 @@ not necessarily when it was made. | Date | Decision | Source and reason | Status | Retires when | | --- | --- | --- | --- | --- | +| 2026-09-28 | A model call's `cost` (`Imp.Core.LMResponse`, the `:model_response` event) is only a charge the provider reported, and `nil` when it reported none; ReqLLM's catalog price is `estimated_cost`, never `cost`. | deepfates/imp#252. `lib/imp/core.ex` `LMResponse` moduledoc, `test/model_response_cost_test.exs`. Hosts sum `cost` as money spent (Dwell's daily cap); the catalog price differs from the charge when prices change, when routing picks another endpoint, and by ReqLLM rounding each line item to a millionth of a dollar. | In force. | Does not retire. | | 2026-09-28 | Imp follows semantic versioning. Before 1.0, a release that changes what a caller receives or can rely on bumps the minor version (`0.5` to `0.6`), and a release of fixes that change nothing a caller relies on bumps the patch; `{:imp, "~> 0.x"}` then never takes a breaking release unasked. | Owner, 2026-09-28. `CHANGELOG.md` marks each breaking change "Breaking:" with its migration, and `RELEASE_NOTES.md` names them. | In force. | Does not retire. | | 2026-09-26 | `Imp.Optimizer.GEPA` with no `:execution_profile` runs DSPy's GEPA (`:gepa_v0_1_4_merge`, or `:gepa_v0_1_4` with `use_merge: false`), and its reflection records are DSPy's; Imp's own search is `execution_profile: :beam_native`, chosen explicitly. | Owner, 2026-09-26. An upstream name promises upstream semantics; the published GEPA results (`research/RESULTS.md` R3, R5) used the pinned profile, as `:gepa_v0_1_4`; and no reason for the BEAM-native default was ever recorded. `lib/imp/optimizer/gepa.ex` moduledoc, `test/gepa_agent_reflection_test.exs`. | In force. | Does not retire. | | 2026-09-25 | Prompts and parsed values use neutral spellings: types in words (`string`, `integer`, `true or false`, `one of: a, b`, `list of strings`, `object`, declared once in `Imp.Adapter.FieldType`), values in JSON (`null`, `true`, `["x", "y"]`), in every adapter, ReAct, and the MIPROv2, GEPA and SIMBA prompts. A non-string answer for a string field is kept as its JSON text; a null is no value. Parity with DSPy means the same fields, order and constraints in the prompt, the same fields and types accepted, and the same errors raised, not DSPy's text or Python's `str()` spelling. | The owner's intention, 2026-09-25: Python spellings (`Literal[...]`, `True`, `None`, `repr(Example)`) read as foreign in an Elixir library and teach the model nothing the neutral words do not. `test/adapter_type_wording_test.exs`, `test/upstream_exam/adapters_test.exs`; the golden trace and the MIPROv2 proposer differentials compare prompts after putting DSPy's spellings into Imp's words (`Imp.DSPyWording`). | In force. | Does not retire. | diff --git a/lib/imp/core.ex b/lib/imp/core.ex index 75f6e226..cc4e0b0f 100644 --- a/lib/imp/core.ex +++ b/lib/imp/core.ex @@ -75,25 +75,52 @@ defmodule Imp.Core do @moduledoc """ Provider-neutral LM response: normalized outputs, usage, cost, and raw data. - `cost` is the provider's reported total for this call in USD as a - non-negative float, or `nil` when the provider reported nothing Imp can - read as a number. Providers report that total in several shapes — a bare - number, a string, a `Decimal`, or a cost breakdown map carrying a `total` - — and Imp reads the number out of all of them here, so a host reading a - call's money never has to learn a provider library's private shape. - - `billing` is the provider's cost breakdown map, untouched, when the - provider reported one, and `nil` otherwise. It is the detail behind `cost` - (line items, input and output splits); its shape belongs to the provider, - so it is evidence to inspect rather than a contract to depend on. + `cost` is what the provider says it charged for this call, in USD, as a + non-negative float, or `nil` when the provider reported no charge. `nil` + means the charge is unknown, not that the call was free. OpenRouter reports + its charge, unasked, as the `"cost"` field of its usage object; + ReqLLM carries that field through unchanged, and Imp reads it from there. + A provider whose response carries no charge — the Anthropic, OpenAI and + Google APIs called directly among them — gives a `nil` cost. On an + OpenRouter call made with the caller's own provider key (`"is_byok"`), + `cost` is OpenRouter's fee plus the upstream charge it reports in + `"cost_details"`, and `nil` when that upstream charge is missing. An LM + client other than ReqLLM reports its charge as `:cost` in its response + metadata. + + `estimated_cost` is ReqLLM's estimate for the call: the reported token + counts priced from its model catalog, as a non-negative float, or `nil` + when the catalog has no price for the model or the call was streamed. It + is a different number from + the charge whenever prices have changed, the provider routed to an endpoint + with other prices, or the provider prices caching and reasoning differently + from the catalog, so a host that falls back on it when `cost` is `nil` is + choosing to count an estimate as spend. + + Either figure may arrive as a bare number, a string, a `Decimal` or a + breakdown map carrying a `total`, and Imp reads the number out of each, so + a host reading a call's money never has to learn a provider library's + private shape. + + `billing` is the breakdown behind `estimated_cost` — ReqLLM's catalog line + items and input, output and reasoning splits — untouched, when there is + one, and `nil` otherwise. Its shape belongs to ReqLLM, so it is evidence to + inspect rather than a contract to depend on. """ - defstruct outputs: [], usage: %{}, cost: nil, billing: nil, metadata: %{}, raw: nil + defstruct outputs: [], + usage: %{}, + cost: nil, + estimated_cost: nil, + billing: nil, + metadata: %{}, + raw: nil @type t :: %__MODULE__{ outputs: list(), usage: map(), - cost: number() | nil, + cost: float() | nil, + estimated_cost: float() | nil, billing: map() | nil, metadata: map(), raw: term() @@ -118,15 +145,15 @@ defmodule Imp.Core do def response(raw) do with {:ok, outputs, metadata} <- split_outputs(raw) do usage = response_usage(metadata) - - reported = reported_cost(metadata, usage) + estimate = Map.get(usage, :cost) {:ok, %LMResponse{ outputs: outputs, usage: usage, - cost: cost_number(reported), - billing: billing_breakdown(reported), + cost: cost_number(reported_cost(metadata, usage)), + estimated_cost: cost_number(estimate), + billing: billing_breakdown(estimate), metadata: metadata, raw: raw }} @@ -267,12 +294,34 @@ defmodule Imp.Core do end end + # In ReqLLM's usage map the atom keys are ReqLLM's own: its usage step + # prices the token counts from the model catalog and stores that estimate as + # `:cost` and `:total_cost`. The provider's wire fields that ReqLLM does not + # interpret stay under their string keys, so OpenRouter's charge is + # `"cost"`. OpenRouter includes it without being asked (its + # `usage: %{include: true}` request option is not needed for it). A client + # that is not ReqLLM reports its charge as `:cost` in its own metadata. + # + # With the caller's own provider key (`"is_byok"`), OpenRouter's `"cost"` is + # only its fee; the provider billed the key separately, and OpenRouter + # reports that as `"cost_details"."upstream_inference_cost"`. Counting the fee + # alone would understate spend, so without the upstream figure the charge + # is unknown. + defp reported_cost(_metadata, %{"is_byok" => true} = usage) do + upstream = usage |> Map.get("cost_details") |> map_value(:upstream_inference_cost, nil) + + case Map.get(usage, "cost") do + fee when is_number(fee) and is_number(upstream) -> fee + upstream + _incomplete -> nil + end + end + defp reported_cost(metadata, usage) do - map_value(usage, :cost, map_value(metadata, :cost, nil)) + Map.get(usage, "cost", map_value(metadata, :cost, nil)) end - # A cost breakdown is the provider's own map. Anything else a provider - # reports as a cost is a value, not a breakdown, so there is nothing to keep. + # A cost breakdown is ReqLLM's map. Anything else under `:cost` is a value, + # not a breakdown, so there is nothing to keep. defp billing_breakdown(%_struct{}), do: nil defp billing_breakdown(reported) when is_map(reported), do: reported defp billing_breakdown(_reported), do: nil diff --git a/lib/imp/lm.ex b/lib/imp/lm.ex index ffa3df6b..2b763a4c 100644 --- a/lib/imp/lm.ex +++ b/lib/imp/lm.ex @@ -17,16 +17,20 @@ defmodule Imp.LM do is recorded the same way, with the usage the provider reported at the end of the stream. - The response event's metadata carries the - money for that call in `:cost`: the provider's reported total in USD as a - non-negative float, or `nil` when the provider reported nothing Imp can read - as a number. A host summing spend reads that number and nothing else. - - Providers report the total as a bare number, a string, a `Decimal` or a cost - breakdown map, and Imp reads the number out of all four. When the provider - reported a breakdown, that map is also on the event as `:billing`, unchanged; - when it reported none, there is no `:billing` key. A breakdown's shape is the - provider's, so treat it as evidence to inspect, not as a contract. + The response event's metadata carries the money for that call as two + numbers, each a non-negative float in USD or `nil`, as on + `Imp.Core.LMResponse`. `:cost` is what the provider reported charging, and + `nil` when it reported no charge, which means the charge is unknown rather + than zero. `:estimated_cost` is ReqLLM's catalog price for the reported + tokens, and `nil` when the catalog has no price for the model or the call + was streamed. A host summing + money spent sums `:cost`; one that falls back on `:estimated_cost` for calls + with no reported charge is counting an estimate, and should know it. + + When ReqLLM priced the call, the breakdown behind `:estimated_cost` is also + on the event as `:billing`, unchanged; otherwise there is no `:billing` key. + A breakdown's shape is ReqLLM's, so treat it as evidence to inspect, not as + a contract. """ @typedoc """ @@ -219,6 +223,7 @@ defmodule Imp.LM do model: request.config.model, usage: response.usage, cost: response.cost, + estimated_cost: response.estimated_cost, response: response.metadata }, response.billing diff --git a/priv/public_api.json b/priv/public_api.json index fd825129..9dd3b0ac 100644 --- a/priv/public_api.json +++ b/priv/public_api.json @@ -3230,6 +3230,7 @@ "struct_fields": [ "billing", "cost", + "estimated_cost", "metadata", "outputs", "raw", diff --git a/test/model_response_cost_test.exs b/test/model_response_cost_test.exs index 4f9a4bd2..49768513 100644 --- a/test/model_response_cost_test.exs +++ b/test/model_response_cost_test.exs @@ -1,7 +1,7 @@ defmodule Imp.ModelResponseCostTest do - # The money a host reads off a model call. Imp owns this boundary: the cost - # on a `:model_response` event is a number, never a provider library's - # private breakdown shape. + # The money a host reads off a model call. Imp owns this boundary: `cost` is + # what the provider reported charging and `estimated_cost` is ReqLLM's catalog + # price, each a number or nil, never a provider library's private shape. use ExUnit.Case, async: false @breakdown %{ @@ -20,8 +20,9 @@ defmodule Imp.ModelResponseCostTest do } defmodule BilledStub do - # A ReqLLM response whose usage carries a billing breakdown, the shape - # ReqLLM.Usage.Cost.merge/3 leaves on a priced call. + # A ReqLLM response with the usage map a test gives it. ReqLLM's usage step + # leaves its catalog estimate under atom keys (ReqLLM.Usage.Cost.merge/3) + # and a provider's own fields, such as OpenRouter's "cost", under strings. def generate_text(model, messages, opts) do usage = Keyword.fetch!(opts, :stub_usage) @@ -78,8 +79,10 @@ defmodule Imp.ModelResponseCostTest do {events, response} end - test "a billed call reports the total as a number with the breakdown beside it" do + test "a priced call reports the provider's charge as cost and the catalog price apart" do usage = %{ + "cost" => 0.0021, + "cost_details" => %{"upstream_inference_cost" => 0.0021}, input_tokens: 3, output_tokens: 2, total_tokens: 5, @@ -92,24 +95,99 @@ defmodule Imp.ModelResponseCostTest do {_events, response} = model_response(usage) - assert response.metadata.cost == 0.001858 + assert response.metadata.cost == 0.0021 assert is_float(response.metadata.cost) + assert response.metadata.estimated_cost == 0.001858 assert response.metadata.billing == @breakdown assert response.metadata.billing.line_items == @breakdown.line_items end - test "a call the provider did not price reports no cost and no billing" do + test "a call ReqLLM priced but the provider did not report has no cost, only an estimate" do + usage = %{input_tokens: 3, output_tokens: 2, total_tokens: 5, cost: @breakdown} + + {_events, response} = model_response(usage) + + assert response.metadata.cost == nil + assert response.metadata.estimated_cost == 0.001858 + assert response.metadata.billing == @breakdown + end + + test "a call nobody priced reports no cost, no estimate and no billing" do usage = %{input_tokens: 3, output_tokens: 2, total_tokens: 5} {_events, response} = model_response(usage) assert response.metadata.cost == nil + assert response.metadata.estimated_cost == nil refute Map.has_key?(response.metadata, :billing) assert response.metadata.usage == usage end - test "the ATIF export carries the number, not the breakdown map" do - usage = %{input_tokens: 3, output_tokens: 2, total_tokens: 5, cost: @breakdown} + # The whole path a resident's call takes: an OpenRouter-shaped body over + # HTTP, ReqLLM's decoding and its usage step pricing the tokens from the + # model's catalog entry, then Imp. The entry prices 7000 input and 3000 + # output tokens at 0.0019; OpenRouter says it charged 0.0025. + test "an OpenRouter response keeps OpenRouter's charge as cost and ReqLLM's price as the estimate" do + base_url = + Imp.Test.LocalHTTP.start(fn _request -> + {200, + %{ + "id" => "gen-cost", + "object" => "chat.completion", + "model" => "openai/gpt-4.1-nano", + "choices" => [ + %{ + "index" => 0, + "finish_reason" => "stop", + "message" => %{"role" => "assistant", "content" => "pong"} + } + ], + "usage" => %{ + "prompt_tokens" => 7000, + "completion_tokens" => 3000, + "total_tokens" => 10_000, + "cost" => 0.0025, + "is_byok" => false, + "cost_details" => %{"upstream_inference_cost" => 0.0025} + } + }} + end) + + lm = + Imp.req_llm( + %{ + provider: :openrouter, + id: "openai/gpt-4.1-nano", + model: "openai/gpt-4.1-nano", + base_url: base_url <> "/v1", + pricing: %{ + currency: "USD", + components: [ + %{id: "token.input", kind: "token", unit: "token", per: 1_000_000, rate: 0.1}, + %{id: "token.output", kind: "token", unit: "token", per: 1_000_000, rate: 0.4} + ] + } + }, + api_key: "local-test-key", + cache: false + ) + + request = Imp.LM.new_request(lm, [%{role: :user, content: "ping"}], [], "cost test") + assert {:ok, response} = Imp.Clients.ReqLLM.request(lm, request) + + assert response.cost == 0.0025 + assert_in_delta response.estimated_cost, 0.0019, 1.0e-12 + assert response.billing.total == response.estimated_cost + end + + test "the ATIF export carries both numbers, not the breakdown map in their place" do + usage = %{ + "cost" => 0.002, + input_tokens: 3, + output_tokens: 2, + total_tokens: 5, + cost: @breakdown + } {events, _response} = model_response(usage) document = Imp.Trajectory.to_atif(events) @@ -119,19 +197,23 @@ defmodule Imp.ModelResponseCostTest do |> Enum.map(&get_in(&1, ["extra", "model_observation"])) |> Enum.find(&is_map/1) - assert observation["cost"] == 0.001858 + assert observation["cost"] == 0.002 + assert observation["estimated_cost"] == 0.001858 assert observation["billing"]["total"] == 0.001858 end - test "a total reported as a string or a Decimal reads as the same number" do + test "a figure reported as a string or a Decimal reads as the same number" do for total <- ["0.001858", " 0.001858 ", Decimal.new("0.001858")] do raw = %{ __imp_lm_output__: "pong", - __imp_lm_metadata__: %{req_llm: %{usage: %{cost: %{@breakdown | total: total}}}} + __imp_lm_metadata__: %{ + req_llm: %{usage: %{:cost => %{@breakdown | total: total}, "cost" => total}} + } } assert {:ok, response} = Imp.Core.response(raw) assert response.cost == 0.001858 + assert response.estimated_cost == 0.001858 assert response.billing.total == total end @@ -141,19 +223,65 @@ defmodule Imp.ModelResponseCostTest do } assert {:ok, response} = Imp.Core.response(bare) - assert response.cost == 0.5 + assert response.cost == nil + assert response.estimated_cost == 0.5 assert response.billing == nil end - test "a cost that cannot be read as a non-negative number is nothing, not a guess" do + # With the caller's own provider key, OpenRouter's "cost" is its fee only, + # and the provider's own charge is reported beside it. + test "an OpenRouter call on the caller's own key costs the fee plus the upstream charge" do + raw = %{ + __imp_lm_output__: "pong", + __imp_lm_metadata__: %{ + req_llm: %{ + usage: %{ + "is_byok" => true, + "cost" => 0.0001, + "cost_details" => %{"upstream_inference_cost" => 0.002}, + cost: @breakdown + } + } + } + } + + assert {:ok, response} = Imp.Core.response(raw) + assert_in_delta response.cost, 0.0021, 1.0e-12 + assert response.estimated_cost == 0.001858 + end + + test "an OpenRouter call on the caller's own key with no upstream charge has no cost" do + for details <- [nil, %{}, %{"upstream_inference_cost" => nil}] do + usage = + %{"is_byok" => true, "cost" => 0.0001, cost: @breakdown} + |> then(&if(details, do: Map.put(&1, "cost_details", details), else: &1)) + + raw = %{__imp_lm_output__: "pong", __imp_lm_metadata__: %{req_llm: %{usage: usage}}} + + assert {:ok, response} = Imp.Core.response(raw) + assert response.cost == nil + assert response.estimated_cost == 0.001858 + end + end + + test "a client other than ReqLLM reports its charge in its own metadata" do + raw = %{__imp_lm_output__: "pong", __imp_lm_metadata__: %{cost: 0.25}} + + assert {:ok, response} = Imp.Core.response(raw) + assert response.cost == 0.25 + assert response.estimated_cost == nil + end + + test "a figure that cannot be read as a non-negative number is nothing, not a guess" do for reported <- ["free", %{total: "free"}, %{total: nil}, -0.5, %{total: -0.5}] do raw = %{ __imp_lm_output__: "pong", - __imp_lm_metadata__: %{req_llm: %{usage: %{cost: reported}}} + __imp_lm_metadata__: %{req_llm: %{usage: %{:cost => reported, "cost" => reported}}} } assert {:ok, response} = Imp.Core.response(raw) assert response.cost == nil + assert response.estimated_cost == nil end end diff --git a/test/timed_out_connection_test.exs b/test/timed_out_connection_test.exs index 1a8e4590..90b9bc7f 100644 --- a/test/timed_out_connection_test.exs +++ b/test/timed_out_connection_test.exs @@ -72,7 +72,9 @@ defmodule Imp.TimedOutConnectionTest do messages = [%{role: :user, content: "ping"}] assert {:error, %Imp.LMError{retryable: true}} = Imp.LM.generate(lm, messages, opts) - assert {:ok, _reply} = Imp.LM.generate(lm, messages, opts) + # Only the first call has to time out. The second gets room to be answered + # on a loaded machine; landing on the stuck connection still fails below. + assert {:ok, _reply} = Imp.LM.generate(lm, messages, Keyword.put(opts, :timeout, 5_000)) assert [first, second] = Agent.get(seen, & &1) refute first == second