Python: Fix Python OTel usage detail attributes

[eavanvalkenburg]

Motivation and Context

Closes #5511.

This replaces and supersedes #6194 with a smaller, focused change. Python observability currently emits only input/output token counts on OTel spans and drops cache/read/reasoning usage details that providers already expose.

Description

This change maps known UsageDetails token fields to the exact OTel GenAI semantic-convention attributes:

• gen_ai.usage.cache_creation.input_tokens
• gen_ai.usage.cache_read.input_tokens
• gen_ai.usage.reasoning.output_tokens

It adds standard generic usage keys for OpenAI and Anthropic parsing while preserving the existing provider-specific keys for compatibility. The same mapping is used for direct chat spans and aggregated invoke_agent spans, and zero-valued mapped token counts are preserved.

Thanks to @hanhan761 for the original investigation and implementation work in #6194.

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the Contribution Guidelines
• All unit tests pass, and I have added new tests where possible
• Is this a breaking change? No

[github-actions]

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
packages/anthropic/agent_framework_anthropic
_chat_client.py	448	35	92%	461, 464, 545, 638, 640, 783, 819–820, 898, 900, 930–931, 976, 992–993, 1000–1002, 1006–1008, 1012–1015, 1131, 1141, 1193, 1341–1342, 1359, 1372, 1385, 1410–1411
packages/core/agent_framework
_types.py	1190	96	91%	59, 68–69, 123, 128, 147, 149, 153, 157, 159, 161, 163, 181, 185, 211, 233, 238, 243, 247, 277, 696–697, 856–857, 1292, 1364, 1399, 1419, 1429, 1481, 1613–1615, 1900–1905, 1930, 1985, 1990, 2000, 2008, 2015–2019, 2037, 2110, 2118–2120, 2125, 2228, 2251, 2506, 2530, 2629, 2883, 3093, 3152, 3191, 3202, 3204–3208, 3210, 3213–3221, 3231, 3320, 3457, 3462, 3467, 3472, 3476, 3560–3562, 3591, 3679–3683
observability.py	853	61	92%	408, 410–411, 414, 417, 420–421, 426–427, 433–434, 440–441, 448, 450–452, 455–457, 462–463, 469–470, 476–477, 484, 661–662, 861, 865–867, 869, 873–874, 878, 916, 918, 929–931, 933–935, 939, 947, 1071–1072, 1307, 1567–1568, 1672, 1796, 1837–1838, 1981, 2172, 2369, 2586, 2588
packages/openai/agent_framework_openai
_chat_client.py	1102	150	86%	276, 289, 639–643, 651–654, 660–664, 714–721, 723–725, 732–734, 780, 788, 811, 929, 1028, 1087, 1089, 1091, 1093, 1159, 1173, 1253, 1263, 1268, 1311, 1422–1423, 1438, 1665, 1670, 1674–1676, 1680–1681, 1764, 1774, 1801, 1807, 1817, 1823, 1828, 1834, 1839–1840, 1859, 1862–1865, 1879, 1881, 1889–1890, 1902, 1944, 2002–2003, 2038, 2060–2061, 2076–2077, 2095–2096, 2139, 2305, 2343–2344, 2362, 2442–2450, 2480, 2590, 2625, 2640, 2660–2670, 2683, 2694–2698, 2712, 2726–2737, 2746, 2778–2781, 2791–2792, 2803–2805, 2819–2821, 2831–2832, 2838, 2853
_chat_completion_client.py	365	17	95%	431, 527–528, 532, 765, 767, 772, 775, 866, 868, 885, 906, 934, 947, 971, 991, 1306
TOTAL	39553	4499	88%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
7863	34 💤	0 ❌	0 🔥	2m 8s ⏱️

[Copilot]

Pull request overview

This PR fixes Python OTel GenAI span usage attributes by mapping additional provider-reported token usage details (cache create/read and reasoning tokens) onto the corresponding OTel semantic-convention attributes, while preserving legacy/provider-specific usage keys for compatibility.

Changes:

• Add OTel GenAI usage attributes for cache creation/read input tokens and reasoning output tokens, and map known UsageDetails keys (including legacy OpenAI/Anthropic keys) onto them.
• Extend OpenAI and Anthropic usage parsing to emit standard generic usage keys in addition to existing provider-specific keys, preserving zero values.
• Add/extend unit tests to validate the new mapping behavior for both direct chat spans and aggregated invoke_agent spans.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
python/packages/core/agent_framework/observability.py	Adds new OTel usage attributes and central mapping logic from UsageDetails keys to OTel span attributes (including legacy/provider keys).
python/packages/core/agent_framework/_types.py	Extends UsageDetails with standard fields for cache and reasoning token counts.
python/packages/core/tests/core/test_observability.py	Adds coverage for additional usage mapping, legacy key mapping, capture flags, and invoke_agent aggregation.
python/packages/openai/agent_framework_openai/_chat_completion_client.py	Emits standard mapped keys for cached and reasoning tokens (preserving zeros) for Chat Completions usage parsing.
python/packages/openai/agent_framework_openai/_chat_client.py	Emits standard mapped keys for cached and reasoning tokens for Responses usage parsing.
python/packages/openai/tests/openai/test_openai_chat_completion_client.py	Adds test ensuring OpenAI Chat Completions parsing includes both legacy and standard mapped usage detail keys (including zeros).
python/packages/openai/tests/openai/test_openai_chat_client.py	Extends tests to assert new standard mapped keys and that zero values are preserved.
python/packages/anthropic/agent_framework_anthropic/_chat_client.py	Emits standard mapped cache token keys alongside existing Anthropic-specific keys, preserving zeros.
python/packages/anthropic/tests/test_anthropic_client.py	Adds tests for Anthropic parsing including standard mapped keys and preservation of zero cache tokens.

[github-actions]

Automated Code Review

Reviewers: 5 | Confidence: 85% | Result: All clear

Reviewed: Correctness, Security Reliability, Test Coverage, Failure Modes, Design Approach

---

Automated review by eavanvalkenburg's agents