Skip to content

Extend OpenAITelemetryPlugin: Add tracking & pricing for cached tokens #1582

@waldekmastykarz

Description

@waldekmastykarz

Feature Request

Extend the OpenAITelemetryPlugin to support token tracking and pricing information for cached tokens.


Background
Currently, the OpenAITelemetryPlugin tracks and calculates token usage and associated pricing. However, many OpenAI APIs (especially when caching is present) price cached tokens differently than non-cached tokens.

Requested Enhancement

  • Update the plugin to distinguish between regular and cached tokens.
  • Track cached tokens separately and calculate/report cost accurately based on their distinct rates.
  • Factor in OpenAI pricing models that treat cached tokens differently, and expose the split and totals in pricing information/output.
  • Update relevant reporting and metrics methods to show cached token counts and costs distinctly.

Impacted Code References:

Benefits

  • Improved reporting and transparency for users with cache-aware pricing.
  • More accurate cost/control for workloads with cache/usage mix.
  • Better insights in telemetry and exported reports for complex scenarios.

Metadata

Metadata

Labels

No labels
No labels
No fields configured for Feature.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions