Realtime Client Proposal

[tarekgh]

Realtime Client Proposal

⚠️ Important Notes

• This is an experimental proposal. All APIs introduced here are subject to change, and breaking changes should be expected as the design evolves.
• MCP (Model Context Protocol) hosted tools support is not fully supported or tested yet. While the plumbing for HostedMcpServerTool is in place, it has not been validated end-to-end and may change significantly.

---

Overview

This PR introduces a Realtime Client abstraction layer for Microsoft.Extensions.AI, enabling bidirectional, streaming communication with realtime AI services (e.g., OpenAI's Realtime API). The design follows the same middleware/pipeline patterns established by IChatClient and extends them to realtime sessions over WebSocket connections.

Key changes include:

• New abstractions (IRealtimeClient, IRealtimeSession, DelegatingRealtimeSession) in Microsoft.Extensions.AI.Abstractions
• Strongly-typed client/server message types for audio streaming, text, transcription, function calls, and error handling
• Session configuration via RealtimeSessionOptions supporting audio formats, voice activity detection, transcription, tools, and tracing
• Middleware pipeline via RealtimeSessionBuilder with built-in support for:
  • Logging (LoggingRealtimeSession)
  • OpenTelemetry (OpenTelemetryRealtimeSession) following GenAI semantic conventions
  • Function invocation (FunctionInvokingRealtimeSession) with automatic tool call resolution
• OpenAI provider implementation (OpenAIRealtimeClient, OpenAIRealtimeSession) using WebSocket connections
• Refactored function invocation — extracted shared logic from FunctionInvokingChatClient into reusable components (FunctionInvocationProcessor, FunctionInvocationHelpers, FunctionInvocationLogger) so both chat and realtime sessions share the same invocation pipeline

Files changed: 63 (11,231 insertions, 319 deletions)

---

Supported Realtime Messages

Client Messages (sent to the server)

Message Type	Description
RealtimeClientConversationItemCreateMessage	Creates a conversation item (text, audio, or image) to add to the session context.
RealtimeClientInputAudioBufferAppendMessage	Appends a chunk of audio data (PCM) to the server's input audio buffer.
RealtimeClientInputAudioBufferCommitMessage	Commits the accumulated audio buffer, signaling the server that the audio input is complete.
RealtimeClientResponseCreateMessage	Requests the server to generate a response based on the current conversation context.

Server Messages (received from the server)

Message Type	Description
RealtimeServerOutputTextAudioMessage	Carries incremental or completed text and audio output from the model.
RealtimeServerInputAudioTranscriptionMessage	Carries transcription results (incremental or completed) for user audio input.
RealtimeServerResponseCreatedMessage	Indicates a response has been created, includes token usage information.
RealtimeServerResponseOutputItemMessage	Represents a new output item (e.g., function call) added during response generation.
RealtimeServerErrorMessage	Carries error details when the server encounters a problem processing a request.

Server Message Types (RealtimeServerMessageType enum)

Type	Description
RawContentOnly	The response contains only raw/unparsed content.
OutputTextDelta	Incremental text output update from the model.
OutputTextDone	Text output streaming is complete.
OutputAudioDelta	Incremental audio output update from the model.
OutputAudioDone	Audio output streaming is complete.
OutputAudioTranscriptionDelta	Incremental model-generated transcription of audio output.
OutputAudioTranscriptionDone	Model-generated transcription of audio output is complete.
InputAudioTranscriptionDelta	Incremental transcription of user audio input.
InputAudioTranscriptionCompleted	Transcription of user audio input is complete.
InputAudioTranscriptionFailed	Transcription of user audio input has failed.
ResponseCreated	A new response has been created by the server.
ResponseDone	The response is fully complete.
Error	An error occurred while processing the request.
McpCallInProgress	An MCP tool call is in progress.
McpCallCompleted	An MCP tool call has completed successfully.
McpCallFailed	An MCP tool call has failed.
McpListToolsInProgress	Listing MCP tools is in progress.
McpListToolsCompleted	Listing MCP tools has completed.
McpListToolsFailed	Listing MCP tools has failed.

---

Usage Examples

1. Creating a Realtime Client

using Microsoft.Extensions.AI;

// Create an OpenAI realtime client with your API key and model
IRealtimeClient realtimeClient = new OpenAIRealtimeClient(apiKey: "your-api-key", model: "gpt-realtime");

2. Creating a Session

// Create a basic session
IRealtimeSession session = await realtimeClient.CreateSessionAsync();

3. Enabling Middlewares (Logging, OpenTelemetry, Function Invocation)

Use RealtimeSessionBuilder to compose a middleware pipeline around the session:

// Define AI functions for tool calling
AIFunction getWeatherFunction = AIFunctionFactory.Create(
    (string location) => location switch
    {
        "Seattle" => $"The weather in {location} is rainy, 55°F",
        "New York" => $"The weather in {location} is cloudy, 70°F",
        _ => $"Sorry, I don't have weather data for {location}."
    },
    "GetWeather",
    "Gets the current weather for a given location");

// Set up DI services with logging
var services = new ServiceCollection()
    .AddLogging(builder =>
    {
        builder.SetMinimumLevel(LogLevel.Debug);
        builder.AddConsole();
    })
    .BuildServiceProvider();

// Build the session with middlewares
var builder = new RealtimeSessionBuilder(session)
    .UseFunctionInvocation(configure: functionSession =>
    {
        functionSession.AdditionalTools = [getWeatherFunction];
        functionSession.MaximumIterationsPerRequest = 10;
        functionSession.AllowConcurrentInvocation = true;
    })
    .UseOpenTelemetry(configure: otel =>
    {
        otel.EnableSensitiveData = true;
    })
    .UseLogging();

IRealtimeSession wrappedSession = builder.Build(services);

4. Configuring the Session

// Update session options (voice, modalities, instructions, VAD, tools, etc.)
await wrappedSession.UpdateAsync(new RealtimeSessionOptions
{
    OutputModalities = ["audio"],
    Instructions = "You are a helpful assistant.",
    Voice = "alloy",
    VoiceSpeed = 1.0,
    TranscriptionOptions = new TranscriptionOptions("en", "whisper-1"),
    VoiceActivityDetection = new VoiceActivityDetection
    {
        CreateResponse = true,
    },
    Tools = [getWeatherFunction]
});

5. Sending Client Messages to the Server

Use a Channel<RealtimeClientMessage> to send messages asynchronously:

var clientMessageChannel = Channel.CreateUnbounded<RealtimeClientMessage>();

// Send text message
var textItem = new RealtimeContentItem(
    [new TextContent("Hello, what's the weather in Seattle?")],
    id: null,
    role: ChatRole.User
);
await clientMessageChannel.Writer.WriteAsync(new RealtimeClientConversationItemCreateMessage(item: textItem));
await clientMessageChannel.Writer.WriteAsync(new RealtimeClientResponseCreateMessage());

// Send audio data
await clientMessageChannel.Writer.WriteAsync(new RealtimeClientInputAudioBufferAppendMessage(
    audioContent: new DataContent($"data:audio/pcm;base64,{Convert.ToBase64String(pcmAudioBytes)}")
));
await clientMessageChannel.Writer.WriteAsync(new RealtimeClientInputAudioBufferCommitMessage());
await clientMessageChannel.Writer.WriteAsync(new RealtimeClientResponseCreateMessage());

6. Listening to Server Messages

Call GetStreamingResponseAsync to consume server messages as they arrive:

var cts = new CancellationTokenSource();

// Start streaming — pass client messages as input, receive server messages as output
await foreach (var serverMessage in wrappedSession.GetStreamingResponseAsync(
    clientMessageChannel.Reader.ReadAllAsync(cts.Token),
    cts.Token))
{
    switch (serverMessage)
    {
        case RealtimeServerOutputTextAudioMessage audioMessage:
            if (audioMessage.Type == RealtimeServerMessageType.OutputAudioDelta && audioMessage.Text != null)
            {
                // Process audio chunk (base64-encoded PCM)
                PlayAudio(audioMessage.Text);
            }
            else if (audioMessage.Type == RealtimeServerMessageType.OutputAudioTranscriptionDelta && audioMessage.Text != null)
            {
                // Display incremental transcription of assistant's speech
                Console.Write(audioMessage.Text);
            }
            break;

        case RealtimeServerInputAudioTranscriptionMessage transcription:
            if (transcription.Type == RealtimeServerMessageType.InputAudioTranscriptionCompleted)
            {
                // Display what the user said
                Console.WriteLine($"You: {transcription.Transcription}");
            }
            break;

        case RealtimeServerErrorMessage error:
            Console.WriteLine($"Error: {error.Error?.Message}");
            break;

        case RealtimeServerResponseCreatedMessage response:
            if (response.Usage != null)
            {
                Console.WriteLine($"Tokens — Input: {response.Usage.InputTokenCount}, Output: {response.Usage.OutputTokenCount}");
            }
            break;

        case RealtimeServerResponseOutputItemMessage outputItem:
            if (outputItem.Item is RealtimeContentItem contentItem)
            {
                foreach (var content in contentItem.Contents)
                {
                    if (content is FunctionCallContent functionCall)
                    {
                        Console.WriteLine($"Function Call: {functionCall.Name}({string.Join(", ", functionCall.Arguments ?? [])})");
                    }
                }
            }
            break;
    }
}

7. Ending the Session

// Signal no more client messages
clientMessageChannel.Writer.Complete();

// Cancel streaming
cts.Cancel();

// Dispose the session
wrappedSession.Dispose();

Microsoft Reviewers: Open in CodeFlow

---

Demo Application

A complete application consuming the new realtime interfaces can be found at: RealtimeProposalDemoApp

[Copilot]

Pull request overview

This PR introduces an experimental Realtime Client / Session abstraction for Microsoft.Extensions.AI, including middleware-style session pipelines (logging, OpenTelemetry, function invocation) and an initial OpenAI realtime provider, while refactoring function-invocation logic to be shared across chat and realtime flows.

Changes:

• Add IRealtimeClient / IRealtimeSession abstractions plus realtime message/option types (audio, transcription, response items, errors, etc.).
• Add RealtimeSessionBuilder pipeline + middleware implementations (LoggingRealtimeSession, OpenTelemetryRealtimeSession, FunctionInvokingRealtimeSession).
• Refactor shared function invocation into reusable internal components (FunctionInvocationProcessor, helpers, logger), used by both chat and realtime.

Reviewed changes

Copilot reviewed 62 out of 63 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
test/Libraries/Microsoft.Extensions.AI.Tests/Realtime/RealtimeSessionExtensionsTests.cs	Unit tests for IRealtimeSession.GetService<T>() extension behavior.
test/Libraries/Microsoft.Extensions.AI.Tests/Realtime/RealtimeSessionBuilderTests.cs	Unit tests for RealtimeSessionBuilder pipeline behavior and ordering.
test/Libraries/Microsoft.Extensions.AI.Tests/Realtime/LoggingRealtimeSessionTests.cs	Unit tests validating logging middleware behavior across methods and log levels.
test/Libraries/Microsoft.Extensions.AI.Tests/Realtime/FunctionInvokingRealtimeSessionTests.cs	Unit tests for function invocation behavior in realtime streaming.
test/Libraries/Microsoft.Extensions.AI.Tests/Realtime/DelegatingRealtimeSessionTests.cs	Unit tests for base delegating session behavior (delegation, disposal, services).
test/Libraries/Microsoft.Extensions.AI.Tests/Microsoft.Extensions.AI.Tests.csproj	Includes shared TestRealtimeSession in test compilation.
test/Libraries/Microsoft.Extensions.AI.OpenAI.Tests/OpenAIRealtimeSessionTests.cs	Unit tests for OpenAI realtime session basic behaviors and guardrails.
test/Libraries/Microsoft.Extensions.AI.OpenAI.Tests/OpenAIRealtimeClientTests.cs	Unit tests for OpenAI realtime client creation and service exposure.
test/Libraries/Microsoft.Extensions.AI.Abstractions.Tests/TestRealtimeSession.cs	Test double for IRealtimeSession with callback hooks.
test/Libraries/Microsoft.Extensions.AI.Abstractions.Tests/Realtime/RealtimeSessionOptionsTests.cs	Tests for RealtimeSessionOptions and related option types.
test/Libraries/Microsoft.Extensions.AI.Abstractions.Tests/Realtime/RealtimeServerMessageTests.cs	Tests for server message types and their property roundtrips.
test/Libraries/Microsoft.Extensions.AI.Abstractions.Tests/Realtime/RealtimeContentItemTests.cs	Tests for RealtimeContentItem construction and mutation.
test/Libraries/Microsoft.Extensions.AI.Abstractions.Tests/Realtime/RealtimeClientMessageTests.cs	Tests for client message types and their properties.
test/Libraries/Microsoft.Extensions.AI.Abstractions.Tests/Realtime/RealtimeAudioFormatTests.cs	Tests for RealtimeAudioFormat behavior.
src/Libraries/Microsoft.Extensions.AI/Realtime/RealtimeSessionExtensions.cs	Adds GetService<T>() extension for IRealtimeSession.
src/Libraries/Microsoft.Extensions.AI/Realtime/RealtimeSessionBuilderRealtimeSessionExtensions.cs	Adds AsBuilder() extension for sessions.
src/Libraries/Microsoft.Extensions.AI/Realtime/RealtimeSessionBuilder.cs	Implements session middleware/pipeline builder.
src/Libraries/Microsoft.Extensions.AI/Realtime/OpenTelemetryRealtimeSessionBuilderExtensions.cs	Builder extension to add OpenTelemetry middleware to a realtime session.
src/Libraries/Microsoft.Extensions.AI/Realtime/LoggingRealtimeSessionBuilderExtensions.cs	Builder extension to add logging middleware to a realtime session.
src/Libraries/Microsoft.Extensions.AI/Realtime/LoggingRealtimeSession.cs	Delegating session middleware that logs calls and streaming messages.
src/Libraries/Microsoft.Extensions.AI/Realtime/FunctionInvokingRealtimeSessionBuilderExtensions.cs	Builder extension to add function invocation middleware.
src/Libraries/Microsoft.Extensions.AI/Realtime/FunctionInvokingRealtimeSession.cs	Implements tool/function invocation loop for realtime streaming.
src/Libraries/Microsoft.Extensions.AI/Realtime/AnonymousDelegatingRealtimeSession.cs	Anonymous delegate-based middleware for streaming interception.
src/Libraries/Microsoft.Extensions.AI/OpenTelemetryConsts.cs	Extends OpenTelemetry constants for realtime and token subcategories.
src/Libraries/Microsoft.Extensions.AI/Common/FunctionInvocationStatus.cs	Shared internal status enum for invocation outcomes.
src/Libraries/Microsoft.Extensions.AI/Common/FunctionInvocationProcessor.cs	Shared processor implementing serial/parallel invocation with instrumentation.
src/Libraries/Microsoft.Extensions.AI/Common/FunctionInvocationLogger.cs	Shared logger messages used by chat and realtime invocation flows.
src/Libraries/Microsoft.Extensions.AI/Common/FunctionInvocationHelpers.cs	Shared helpers (activity detection, elapsed time, tool map creation).
src/Libraries/Microsoft.Extensions.AI/ChatCompletion/FunctionInvokingChatClient.cs	Refactors chat function invocation to use shared processor/helpers/logger.
src/Libraries/Microsoft.Extensions.AI.OpenAI/OpenAIRealtimeClient.cs	Adds OpenAI realtime client implementation that creates/initializes sessions.
src/Libraries/Microsoft.Extensions.AI.OpenAI/OpenAIClientExtensions.cs	Adds AsIRealtimeClient extension for OpenAI client integration.
src/Libraries/Microsoft.Extensions.AI.OpenAI/Microsoft.Extensions.AI.OpenAI.csproj	Adds internals visibility for tests and Channels dependency (non-net10).
src/Libraries/Microsoft.Extensions.AI.Evaluation.Reporting/CSharp/Microsoft.Extensions.AI.Evaluation.Reporting.csproj	Comment formatting change.
src/Libraries/Microsoft.Extensions.AI.Abstractions/UsageDetails.cs	Adds realtime-specific token breakdown fields.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Tools/ToolChoiceMode.cs	Adds tool choice mode enum for realtime use.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/VoiceActivityDetection.cs	Adds VAD options type.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/TranscriptionOptions.cs	Adds transcription configuration type.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/ServerVoiceActivityDetection.cs	Adds server VAD settings.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/SemanticVoiceActivityDetection.cs	Adds semantic VAD settings.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/RealtimeSessionOptions.cs	Adds session configuration options (audio formats, tools, tracing, etc.).
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/RealtimeSessionKind.cs	Adds session kind enum (realtime vs transcription).
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/RealtimeServerResponseOutputItemMessage.cs	Adds server message for output items.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/RealtimeServerResponseCreatedMessage.cs	Adds server message for response lifecycle/usage metadata.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/RealtimeServerOutputTextAudioMessage.cs	Adds server message for output text/audio streaming.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/RealtimeServerMessageType.cs	Adds server message type enum.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/RealtimeServerMessage.cs	Adds base server message type.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/RealtimeServerInputAudioTranscriptionMessage.cs	Adds server transcription message type.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/RealtimeServerErrorMessage.cs	Adds server error message type.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/RealtimeContentItem.cs	Adds realtime conversation item wrapper.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/RealtimeClientResponseCreateMessage.cs	Adds client response request message type (modalities/tools/etc.).
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/RealtimeClientMessage.cs	Adds base client message type.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/RealtimeClientInputAudioBufferCommitMessage.cs	Adds client message for committing audio input buffer.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/RealtimeClientInputAudioBufferAppendMessage.cs	Adds client message for appending audio input buffer.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/RealtimeClientConversationItemCreateMessage.cs	Adds client message for creating a conversation item.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/RealtimeAudioFormat.cs	Adds audio format specification type.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/NoiseReductionOptions.cs	Adds noise reduction options enum.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/IRealtimeSession.cs	Adds realtime session interface.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/IRealtimeClient.cs	Adds realtime client interface.
src/Libraries/Microsoft.Extensions.AI.Abstractions/Realtime/DelegatingRealtimeSession.cs	Adds base delegating session implementation.

[shyamnamboodiripad]

Signing off on behalf of eval (so that the whitespace change in Reporting.csproj does not block merge)

[stephentoub]

I'm unclear as to the semantic of this. RealtimeSessionOptions is a mutable class. If I start setting properties on that while the session is active, is that going to result in immediate changes in behavior?

[tarekgh]

No, the abstraction includes an Update Session operation that must be called to update the session. I need a type to use when updating the session (it must be a writable object), and I also want to expose the same information to anyone requesting it at any time, in that case, it can be read-only.
The reason is that, in the middleware layer, I need access to the session properties. OpenAI models allow updating the session after it has been created, but I believe not all providers allow that. Therefore, in most scenarios, I expect that once the session is created with the desired configuration, it will not change much afterward.

I am trying to avoid having two types for that. Do you have a better idea handling that?

[tarekgh]

Maybe we can make the session options using init instead of setters. This will make the object immutable. I believe this will solve the confusion and will be a clearer design. I'll try that and see how it goes.

[stephentoub]

I'm not sure about the word "Inject"... is that standard terminology used by the providers? Is this just "Send"? How does this relate to GetStreamingResponseAsync... is this only valid when someone is actively enumerating?

[stephentoub]

Should the session itself be enumerable?

[stephentoub]

Do any providers have the notion of "Auto"?

[stephentoub]

Is this mime/media type? I believe we spell that out as MediaType elsewhere, like in DataContent

[stephentoub]

Oh! This isn't using the OpenAI library's realtime support? Why not?

[tarekgh]

Somehow, I had the impression of not taking a dependency on third-party libraries. Looks like I was wrong 🥹. I'll look at that and update. Thanks!

[tarekgh]

I looked at OpenAI SDK, looks the latest package 2.8.0 doesn't have the updates for the Realtime model. I am seeing they have merged the PR openai/openai-dotnet#928 two days ago. I think we need to wait them publish a new version then we can consume it. I'll try to watch that.