Python: Add tool call/result content types and update connectors and samples

[Copilot]

Motivation and Context

MEAI introduced new content types for tool calls/results (code interpreter, MCP server, image generation) and enhancements to hosted file metadata; Python core needed parity so connectors can emit/consume these shapes.

Description

• Types
  • Added CodeInterpreterToolCallContent, CodeInterpreterToolResultContent, ImageGenerationToolCallContent, ImageGenerationToolResultContent, MCPServerToolCallContent, MCPServerToolResultContent; extended HostedFileContent with optional media_type and name; updated Contents union and serialization.
  • FunctionApprovalRequestContent now accepts MCPServerToolCallContent in addition to standard function calls.
• Tools
  • Added HostedImageGenerationTool with typed options/additional properties and translate those options to OpenAI Responses parameters.
• Connectors
  • Updated OpenAI Responses parsing to emit paired tool call/result contents for code interpreter, image generation, and MCP server calls (streaming and non-streaming), even when outputs are empty, ensuring consistent pairing and typing; added option translation for image generation tools and removed raw dict image-generation remapping so callers rely on the tool type; MCP outputs now use the shared content parser.
  • Updated OpenAI Assistants to surface code interpreter and MCP server tool calls.
  • Updated Anthropic connector to emit MCP server and code interpreter tool call/result contents while keeping generic tool_use blocks as FunctionCallContent for non-code-execution tools, fixed missing ChatResponse/CitationAnnotation imports and code-block handling to satisfy type checks, and ensured MCP parsing/typing is consistent.
• Samples
  • Refreshed OpenAI image generation and code interpreter samples to use the new tool call/result contents and proper tool lists.
• Tests
  • Expanded core type/tool tests, OpenAI response/assistants tests, Anthropic tests, and added coverage for MCP server tool call acceptance in function approval requests; added guards to skip optional lightning tests when agentlightning is not installed.

Example:

call = CodeInterpreterToolCallContent(call_id="c1", inputs=[TextContent(text="print('hi')")])
result = CodeInterpreterToolResultContent(call_id="c1", outputs=[TextContent(text="log"), UriContent(uri="https://img")])
img_tool = HostedImageGenerationTool(options={"response_format": "uri", "image_size": "1024x1024"})

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the Contribution Guidelines
• All unit tests pass, and I have added new tests where possible
• Is this a breaking change? If yes, add "[BREAKING]" prefix to the title of the PR.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>.NET: Python: add *ToolCallContent and *ToolCallResultContent</issue_title>
<issue_description>MEAI added several new content types: dotnet/extensions#6964 and dotnet/extensions#6664 and dotnet/extensions#6749

Below the .Net versions are linked to make it easy to reference.

CodeInterpreterContent

• Adds new CodeInterpreterToolCallContent and CodeInterpreterToolResultContent types
• Updates the OpenAI Assistants and Responses ChatClient implementations to produce them

Update HostedFileContent

• Updates HostedFileContent with an optional MediaType and Name, matching the corresponding properties on DataContent and UriContent.

HostedMCP

• Adds content types for MCP tool calls and results (HostedMcpServerToolCallContent, HostedMcpServerToolResultContent) to work in conjuction with HostedMCPTool

Image Generation

• Adds HostedImageGenerationTool
• Adds ImageGenerationToolCallContent and ImageGenerationToolResultContent

Computer Use
TBD
</issue_description>

<agent_instructions>Reference the dotnet links to figure out the structure and then implement them in the right way in the _tools.py for tools and _types.py for types. Make sure to also add the new content types to the Contents union typevar. There should be only python changes and they should be tested, so add test cases for all the types in test_types.py and for tools in test_tools. Everything should be inside the core package, except for some handling of these new types in different connectors, like Anthropic.</agent_instructions>

Comments on the Issue (you are @copilot in this section)

• Fixes Python: Add additional content types to match MEAI #2970

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[eavanvalkenburg]

looking good so far, there are also samples with image generation and code, can you update those as well @copilot

[Copilot]

looking good so far, there are also samples with image generation and code, can you update those as well @copilot

Updated the OpenAI image generation and code interpreter samples to use the new tool call/result content shapes and proper tool lists (1079faa).

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• api.openai.com
  • Triggering command: /usr/bin/python python -m pytest packages/core/tests/core/test_types.py packages/core/tests/core/test_tools.py packages/core/tests/openai/test_openai_responses_client.py (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[eavanvalkenburg]

some additional comments, and the McpServer contents also need handling, also those names should be MCP instead of Mcp (python standard vs dotnet standard) @copilot

[eavanvalkenburg]

couple of notes, and OpenAI Assistants also supports Code Interpreter, so that needs handling. Anthropic also support HostedMCPTool and Code interpreter, so those types need handling as well. @copilot

[eavanvalkenburg]

couple of notes, and OpenAI Assistants also supports Code Interpreter, so that needs handling. Anthropic also support HostedMCPTool and Code interpreter, so those types need handling as well. @copilot

@copilot please work on this

[Copilot]

some additional comments, and the McpServer contents also need handling, also those names should be MCP instead of Mcp (python standard vs dotnet standard) @copilot

Updated MCP content naming, parsing, and connector handling; also added option translation for image generation tools as requested (3866935).

couple of notes, and OpenAI Assistants also supports Code Interpreter, so that needs handling. Anthropic also support HostedMCPTool and Code interpreter, so those types need handling as well. @copilot...

Added code interpreter and MCP handling across Responses, Assistants, and Anthropic connectors to emit the new content types (3866935).

[eavanvalkenburg]

@copilot FunctionApprovalRequestContent also needs to allow MCPServerToolCallContent as the function_call inside of it.

[Copilot]

@copilot FunctionApprovalRequestContent also needs to allow MCPServerToolCallContent as the function_call inside of it.

FunctionApprovalRequestContent now accepts MCPServerToolCallContent (mapping or instance) as the function_call and keeps it typed (3904d09).

[eavanvalkenburg]

@copilot python/packages/core/agent_framework/openai/_responses_client.py:348 is not needed, folks should use the Tool type

[Copilot]

@copilot python/packages/core/agent_framework/openai/_responses_client.py:348 is not needed, folks should use the Tool type

Removed the raw dict image_generation remapping; callers should use the tool type directly (f7c856c).

[giles17]

There are also these samples that use the raw dictionary:

• https://github.com/microsoft/agent-framework/blob/main/python/samples/getting_started/agents/openai/openai_responses_client_streaming_image_generation.py
• https://github.com/microsoft/agent-framework/blob/main/python/samples/getting_started/agents/azure_ai/azure_ai_with_image_generation.py

[eavanvalkenburg]

Thanks @giles17 fixed