What's true now (0.7.0). Two gaps in how usage is recorded for streamed calls. Both were found in the 0.7.0 release fact-check.
Imp.Usage doesn't see streamed calls. A streamed call runs in a separate process (Imp.Streaming.start_program_stream), so Imp.Usage.track/1 wrapped around Imp.collect/3, or around a streamed Imp.call, returns %{}. That's true even when the stream succeeds. Separately, with track_usage, Imp.Predict.call drops its frame on error (lib/imp/predict.ex:93-97), so a failed call's usage never reaches the caller. The usage is on the run's :model_response events and nowhere else.- A streamed call has no estimate.
estimated_cost is nil because Imp takes ReqLLM's metadata-handle usage only when no chunk reported any (put_handle_value(:usage, …, :when_missing), lib/imp/clients/req_llm.ex ~2226). The handle does carry priced usage (cost, total_cost and the rest). Merging those priced fields into the chunk usage gives a streamed call the same cost/estimated_cost as the non-streamed call. That was checked with a temporary patch, since reverted.
Target.
Imp.Usage.track reports streamed calls, successful or failed, the same way it reports non-streamed ones.- A streamed call gets
estimated_cost when ReqLLM priced it.
Done when.
- A test of
Imp.Usage.track around Imp.collect records a streamed call's usage and cost, and the same for a stream that failed after usage arrived. Both fail on 0.7.0.
- A streamed OpenRouter call through LocalHTTP has
cost equal to the provider's figure and estimated_cost equal to the catalog figure.
- The 0.7.0 Known limit is removed.
What's true now (0.7.0). Two gaps in how usage is recorded for streamed calls. Both were found in the 0.7.0 release fact-check.
Imp.Usagedoesn't see streamed calls. A streamed call runs in a separate process (Imp.Streaming.start_program_stream), soImp.Usage.track/1wrapped aroundImp.collect/3, or around a streamedImp.call, returns%{}. That's true even when the stream succeeds. Separately, withtrack_usage,Imp.Predict.calldrops its frame on error (lib/imp/predict.ex:93-97), so a failed call's usage never reaches the caller. The usage is on the run's:model_responseevents and nowhere else.estimated_costis nil because Imp takes ReqLLM's metadata-handle usage only when no chunk reported any (put_handle_value(:usage, …, :when_missing),lib/imp/clients/req_llm.ex~2226). The handle does carry priced usage (cost,total_costand the rest). Merging those priced fields into the chunk usage gives a streamed call the samecost/estimated_costas the non-streamed call. That was checked with a temporary patch, since reverted.Target.
Imp.Usage.trackreports streamed calls, successful or failed, the same way it reports non-streamed ones.estimated_costwhen ReqLLM priced it.Done when.
Imp.Usage.trackaroundImp.collectrecords a streamed call's usage and cost, and the same for a stream that failed after usage arrived. Both fail on 0.7.0.costequal to the provider's figure andestimated_costequal to the catalog figure.