feat: RealtimeVoiceChannel — speech-to-speech over Twilio Media Streams (POC)

[xinghaohuang91]

Summary

TAC now supports two voice architectures, and this PR adds the second one
plus the observability to compare them side by side:

• Realtime — speech-to-speech. Raw call audio is bridged directly to a
  speech-to-speech model (OpenAI Realtime) over Twilio Media Streams. The model
  does its own transcription, reasoning, synthesis, and turn detection.
• Cascaded (ConversationRelay) — the existing path. Twilio does ASR/TTS and
  TAC hands each transcribed turn to any LLM (OpenAI, Bedrock, LangChain, …) for
  a text reply that ConversationRelay speaks back.

The core difference: does the audio ever become text? Cascaded turns speech
into text, runs an LLM, and synthesizes speech back (three models chained).
Realtime keeps everything as audio in a single model. Both channels export the
same OpenTelemetry span shape to Langfuse, so the two can be measured
apples-to-apples.

See REALTIME_VOICE.md for the full write-up.

---

Realtime — speech-to-speech (RealtimeVoiceChannel)

TAC is the bridge between Twilio telephony and OpenAI's speech-to-speech model.
Raw u-law audio flows both ways; TAC does no STT/TTS of its own.

flowchart LR
    caller([📞 Caller])

    subgraph twilio[Twilio]
        pstn[Programmable Voice<br/>PSTN / SIP]
        stream[Media Streams<br/>&lt;Connect&gt;&lt;Stream&gt;]
    end

    subgraph app[Your app · TAC]
        server[RealtimeVoiceServer<br/>FastAPI]
        channel[RealtimeVoiceChannel<br/>audio bridge]
        server --- channel
    end

    subgraph openai[OpenAI]
        realtime[Realtime API<br/>gpt-realtime<br/>STT · reason · TTS · VAD]
    end

    caller <-->|voice call| pstn
    pstn --> stream
    stream -->|HTTP: fetch TwiML| server
    stream <-->|WebSocket: u-law audio| channel
    channel <-->|WebSocket: u-law audio| realtime

Loading

How it works

• Twilio Programmable Voice answers the call and, per the TwiML TAC returns,
  opens a Media Streams WebSocket carrying raw call audio.
• RealtimeVoiceChannel accepts the Media Stream and relays audio to/from the
  OpenAI Realtime API — no text in the loop.
• The model does all the language work (STT, reasoning, TTS, turn detection) and
  streams audio back.
• Barge-in is model-native: on input_audio_buffer.speech_started TAC
  truncates the in-flight reply at the exact millisecond already played and
  clears Twilio's playback buffer.

Trade-offs: lowest latency, most natural (preserves tone/emotion/pacing),
real barge-in — but locked to a speech-to-speech model and the text is only a
by-product (harder to inspect/guardrail mid-turn).

Call flow

sequenceDiagram
    participant Caller
    participant Twilio as Twilio Media Stream
    participant Channel as RealtimeVoiceChannel
    participant OpenAI as OpenAI Realtime

    Note over Caller,OpenAI: Call connects
    Caller->>Twilio: dials number
    Twilio->>Channel: POST /twiml
    Channel-->>Twilio: <Connect><Stream url="wss://…/voice-realtime">
    Twilio->>Channel: WS open + "start" (call metadata)
    Channel->>OpenAI: WS open + session.update (u-law audio, server VAD)
    Channel->>OpenAI: response.create (speak welcome greeting)
    OpenAI-->>Channel: response.output_audio.delta (greeting audio)
    Channel-->>Twilio: media (greeting audio)
    Twilio-->>Caller: 🔊 "Hello! How can I help you today?"

    Note over Caller,OpenAI: Caller speaks
    Caller->>Twilio: 🎤 audio
    Twilio->>Channel: media (base64 u-law)
    Channel->>OpenAI: input_audio_buffer.append
    OpenAI-->>Channel: response.output_audio.delta (reply audio)
    Channel-->>Twilio: media (reply audio)
    Twilio-->>Caller: 🔊 reply

    Note over Caller,OpenAI: Barge-in (caller interrupts)
    Caller->>Twilio: 🎤 starts talking over the reply
    Twilio->>Channel: media
    Channel->>OpenAI: input_audio_buffer.append
    OpenAI-->>Channel: input_audio_buffer.speech_started
    Channel->>OpenAI: conversation.item.truncate (cut reply at played point)
    Channel-->>Twilio: clear (drop buffered audio)

Loading

---

Cascaded — ConversationRelay (VoiceChannel)

Twilio's ConversationRelay does ASR and TTS; TAC hands the transcribed turn to
any LLM and streams the text reply back for Twilio to speak. The LLM is
invoked in the app's on_message_ready callback, not inside TAC.

flowchart LR
    caller([📞 Caller])

    subgraph twilio[Twilio]
        pstn[Programmable Voice<br/>PSTN / SIP]
        relay[ConversationRelay<br/>ASR · TTS · VAD]
    end

    subgraph app[Your app · TAC]
        server[TACFastAPIServer<br/>FastAPI]
        channel[VoiceChannel]
        callback[on_message_ready<br/>your LLM / agent]
        server --- channel
        channel --- callback
    end

    subgraph llm[Your LLM]
        model[OpenAI · Bedrock ·<br/>LangChain · …<br/>text in / text out]
    end

    caller <-->|voice call| pstn
    pstn --> relay
    relay -->|HTTP: fetch TwiML| server
    relay <-->|WebSocket: text tokens| channel
    callback <-->|text prompt / streamed reply| model

Loading

How it works

• Twilio ConversationRelay transcribes caller speech, opens a WebSocket to TAC
  carrying text, and synthesizes the text reply back to the caller.
• VoiceChannel manages the conversation lifecycle and hands each transcribed
  turn to the app's on_message_ready callback.
• The app's LLM produces the text reply, streamed back token by token — swap
  models freely; TAC never touches the model call.

Trade-offs: you see the text every turn, so you can log, moderate, RAG,
route tools deterministically, guardrail, and swap models — but latency stacks
(ASR + LLM + TTS) and paralinguistics are lost at the text boundary.

---

Comparing the two in Langfuse

Both channels export the same span shape (via src/tac/core/tracing.py), so
the architectures can be measured head-to-head. response_latency is the direct
comparison point — "caller finishes talking → assistant's first audio/token back":

Realtime Voice Call                          (RealtimeVoiceChannel)
└── twilio_media_stream.call
    ├── openai_realtime.connect          (model connect + session config)
    ├── openai_realtime.response_latency (caller stops talking -> first audio back)
    ├── openai_realtime.barge_in         (truncated?, played_ms)
    └── openai_realtime.tool_call        (tool_name)

ConversationRelay Voice Call                 (VoiceChannel)
└── conversation_relay.call
    ├── conversation_relay.response_latency  (final transcript -> first token sent)
    ├── model_invocation                      (LLM call; time_to_first_token_ms) *
    └── conversation_relay.barge_in

Two spans differ by design, because of where the model lives:

• model_invocation (*) is emitted by the example, not by TAC. In the
  cascaded path the app invokes the LLM inside on_message_ready, so TAC can't
  time it — voice_cascaded.py wraps the call itself using the new
  VoiceChannel.trace_context(conv_id) to nest under the call trace. It's the
  cascaded stand-in for openai_realtime.connect and breaks out the model's
  share of response_latency.
• ConversationRelay has no tool_call span for the same reason: tool calls
  run inside the callback (the LLM framework's own agent loop), not inside TAC.

getting_started/examples/features/voice_cascaded.py is a fair counterpart to
voice_realtime.py: same model tier, greeting, tone, and tool; no memory on
either side; relay-only mode — so the only variable between traces is the voice
architecture itself. See
getting_started/examples/observability-demo/README.md.

---

What's included

Realtime (new channel):

• RealtimeVoiceChannel (src/tac/channels/realtime/) — the audio bridge, with
  barge-in via conversation.item.truncate + Twilio clear.
• RealtimeVoiceChannelConfig — model / voice / instructions / welcome greeting.
• generate_stream_twiml — <Connect><Stream> TwiML.
• RealtimeVoiceServer (src/tac/server/) — dedicated FastAPI host (separate
  from TACFastAPIServer), exposing POST /twiml and a /voice-realtime WebSocket.
• Example app, tests, and a new optional realtime extra (websockets).

Cascaded observability + comparison:

• OpenTelemetry spans on VoiceChannel (conversation_relay.call /
  .response_latency / .barge_in), mirroring the realtime channel.
• VoiceChannel.trace_context() — public helper for nesting app-emitted spans
  (e.g. a model-invocation span) under the call trace.
• voice_cascaded.py example + updated observability-demo docs.

Notable implementation details

• Realtime uses the GA request shape (no OpenAI-Beta header; nested
  audio.input/audio.output; response.output_audio.delta). The beta shape
  fails against gpt-realtime with close code 4000.
• Audio format is audio/pcmu (u-law) to match Twilio telephony — no transcoding.
• Barge-in tracking resets on response.done so truncation offsets stay valid
  across turns.
• Tracing spans are keyed by conversation_id so concurrent calls never share
  state; a console exporter is the no-credentials fallback.

Out of scope (this POC)

• Caller memory/profile injection on the realtime path — intentionally deferred.
• WebSocket-upgrade signature validation (the TwiML request is validated; the WS
  is not yet). Fine for testing; revisit before production.

Type of Change

• New feature

Checklist

• Tests added/updated
• Documentation updated
• Tested E2E (live call: greeting, conversation, and barge-in verified)

SDK Parity

This is the Python SDK.

• Change is Python-specific (no TypeScript update needed)

POC — flagging for discussion before parity work.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Note

Copilot couldn't run its full agentic review because no GitHub Actions runner was available. Make sure your repository has a runner available to run Copilot's review, or add a copilot-setup-steps.yml file specifying one with the runs-on attribute. See the docs for more details.

Adds a POC “RealtimeVoiceChannel” that bridges Twilio Media Streams (u-law audio) to the OpenAI Realtime WebSocket API for low-latency speech-to-speech voice calls, plus a dedicated FastAPI server endpoint and supporting docs/tests.

Changes:

• Introduces RealtimeVoiceChannel + config + TwiML generator for <Connect><Stream>.
• Adds RealtimeVoiceServer (FastAPI + uvicorn) exposing POST /twiml-realtime and a /voice-realtime WebSocket bridge.
• Adds tests, example script, and a new optional realtime extra (websockets).

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/test_realtime_voice_channel.py	Adds unit tests for TwiML generation, channel bridging behavior, and server route registration.
src/tac/server/realtime_server.py	New FastAPI server hosting TwiML + WebSocket endpoints for the realtime voice path.
src/tac/server/init.py	Exposes RealtimeVoiceServer via the server package API.
src/tac/channels/realtime/twiml.py	Adds TwiML generator for Twilio Media Streams <Connect><Stream>.
src/tac/channels/realtime/config.py	Adds Pydantic config for model/voice/instructions + telephony audio format constants.
src/tac/channels/realtime/channel.py	Implements the Twilio↔OpenAI dual-WebSocket audio relay with barge-in truncation.
src/tac/channels/realtime/init.py	Adds public exports for the realtime channel package.
src/tac/channels/init.py	Re-exports realtime channel symbols at the top-level channels package.
pyproject.toml	Adds realtime optional extra with websockets.
getting_started/examples/features/voice_realtime.py	Adds runnable example for the realtime voice server/channel.
REALTIME_VOICE.md	Adds POC documentation + sequence diagram and setup instructions.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[richardchen874-sys]

The raw-audio bridge between Twilio Media Streams and a speech-to-speech model is a strong direction, especially because it avoids the STT -> text agent -> TTS chain that usually adds latency and failure points.

One suggestion: add a per-call budget and fallback policy early, even for the POC. Realtime audio can become expensive fast, and many production voice flows only need the premium realtime model during the live turn while summaries, classification, and follow-up actions can be routed to cheaper text models afterward.

I am testing an OpenAI-compatible multi-model API layer around official Chinese models, and voice-agent routing/cost boundaries are one of the areas I am watching closely. For this channel, is latency or audio-token cost the bigger blocker for moving beyond POC?