Python: include all usage detail fields in OTEL span attributes (fixes #5511)

[hanhan761]

Summary

_build_response_attributes()\ in \observability.py\ only extracts \input_token_count\ and \output_token_count\ from
esponse.usage_details. Any additional token counts set by providers (e.g. \cached_input_tokens,
easoning_tokens) are silently dropped from OTEL span attributes.

Changes

Replaced the hardcoded extraction of \input_token_count/\output_token_count\ with a loop over all integer-valued keys in \usage_details:

• Standard fields (\input_token_count, \output_token_count) use the existing gen_ai semantic convention attribute names
• Any other integer fields (e.g. \cached_input_tokens,
  easoning_tokens) are set as \gen_ai.usage.\

Related

Fixes #5511

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Extends response usage attribute capture to include all integer usage metrics beyond just input/output tokens, emitting them as gen_ai.usage.* OpenTelemetry attributes.

Changes:

• Iterates over all usage_details items instead of only reading input_token_count and output_token_count.
• Maps known keys to existing OtelAttr.INPUT_TOKENS / OtelAttr.OUTPUT_TOKENS attributes.
• Emits any additional integer usage fields as gen_ai.usage.<key> attributes.

[hanhan761]

All review comments addressed:

1. OTel standard compliance — replaced gen_ai.usage. dynamic prefix with explicit mapping to OTel Gen-AI semantic convention attributes:

   • gen_ai.usage.input_tokens ← input_token_count
   • gen_ai.usage.output_tokens ← output_token_count
   • gen_ai.usage.cache_creation.input_tokens ← cache_creation_input_token_count (new standard field)
   • gen_ai.usage.cache_read.input_tokens ← cache_read_input_token_count (new standard field)
   • gen_ai.usage.reasoning.output_tokens ← reasoning_output_token_count (new standard field)

2. UsageDetails expanded — added cache_creation_input_token_count, cache_read_input_token_count, and reasoning_output_token_count to the TypedDict

3. Providers updated — Anthropic, OpenAI Responses API, and OpenAI Chat Completions now populate the standard field names instead of provider-specific keys

4. Copilot feedback — use of constant for string literal and proper boolean exclusion already addressed in previous commit

@eavanvalkenburg Ready for re-review.

[hanhan761]

Addressed: restored legacy provider field names alongside the new OTel-standard ones for backward compatibility.

• anthropic.cache_creation_input_tokens + cache_creation_input_token_count
• anthropic.cache_read_input_tokens + cache_read_input_token_count
• openai.cached_input_tokens + cache_read_input_token_count
• openai.reasoning_tokens + reasoning_output_token_count
• completion/reasoning_tokens + reasoning_output_token_count
• prompt/cached_tokens + cache_read_input_token_count

Existing integrations using the legacy keys continue to work unchanged.

[github-actions]

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
packages/anthropic/agent_framework_anthropic
_chat_client.py	448	35	92%	461, 464, 545, 638, 640, 783, 819–820, 898, 900, 930–931, 976, 992–993, 1000–1002, 1006–1008, 1012–1015, 1131, 1141, 1193, 1341–1342, 1359, 1372, 1385, 1410–1411
packages/core/agent_framework
_types.py	1190	97	91%	59, 68–69, 123, 128, 147, 149, 153, 157, 159, 161, 163, 181, 185, 211, 233, 238, 243, 247, 277, 696–697, 856–857, 1292, 1364, 1399, 1419, 1429, 1481, 1613–1615, 1797, 1900–1905, 1930, 1985, 1990, 2000, 2008, 2015–2019, 2037, 2110, 2118–2120, 2125, 2228, 2251, 2506, 2530, 2629, 2883, 3093, 3152, 3191, 3202, 3204–3208, 3210, 3213–3221, 3231, 3320, 3457, 3462, 3467, 3472, 3476, 3560–3562, 3591, 3679–3683
observability.py	859	61	92%	402, 404–405, 408, 411, 414–415, 420–421, 427–428, 434–435, 442, 444–446, 449–451, 456–457, 463–464, 470–471, 478, 655–656, 855, 859–861, 863, 867–868, 872, 910, 912, 923–925, 927–929, 933, 941, 1065–1066, 1301, 1561–1562, 1666, 1790, 1831–1832, 1975, 2166, 2363, 2585, 2587
packages/openai/agent_framework_openai
_chat_client.py	1098	150	86%	276, 289, 639–643, 651–654, 660–664, 714–721, 723–725, 732–734, 780, 788, 811, 929, 1028, 1087, 1089, 1091, 1093, 1159, 1173, 1253, 1263, 1268, 1311, 1422–1423, 1438, 1665, 1670, 1674–1676, 1680–1681, 1764, 1774, 1801, 1807, 1817, 1823, 1828, 1834, 1839–1840, 1859, 1862–1865, 1879, 1881, 1889–1890, 1902, 1944, 2002–2003, 2038, 2060–2061, 2076–2077, 2095–2096, 2139, 2305, 2343–2344, 2362, 2442–2450, 2480, 2590, 2625, 2640, 2660–2670, 2683, 2694–2698, 2712, 2726–2737, 2746, 2778–2781, 2791–2792, 2803–2805, 2819–2821, 2831–2832, 2838, 2853
_chat_completion_client.py	365	27	92%	431, 527–528, 532, 764–772, 774–778, 866, 868, 885, 906, 934, 947, 971, 991, 1306
TOTAL	38639	4419	88%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
7755	34 💤	0 ❌	0 🔥	2m 3s ⏱️

[eavanvalkenburg]

@hanhan761 there are checks failing, Bandit mistakes tokens for passwords, so that needs to ignored.

[TaoChenOSU]

If possible, could you also add these usage results to the OpenAI client?

[eavanvalkenburg]

If possible, could you also add these usage results to the OpenAI client?

that is already in this PR @TaoChenOSU

[TaoChenOSU]

If possible, could you also add these usage results to the OpenAI client?

that is already in this PR @TaoChenOSU

Ahh, you're right. And sorry, I forgot to post my comment yesterday.

[eavanvalkenburg]

@hanhan761 there are still checks failing, please have a look and preferable also run all the checks locally!

[eavanvalkenburg]

still failing @hanhan761 please let us know if you can see this through, we want to get this fixed! make sure to use the dev setup guide for the way things are set up by the CI so that it is easier to validate locally

[eavanvalkenburg]

why are all these tests removed?

[eavanvalkenburg]

Thank you @hanhan761 for opening this and for the work/investigation here. I've opened #6493 as a smaller replacement that keeps the existing provider-specific usage keys while adding the OTel standard mappings, so I'm closing this PR in favor of that one.