Skip to content

feat(room_io): mix audio from every participant into one AgentSession - #6855

Open
Darshak03 wants to merge 14 commits into
livekit:mainfrom
Darshak03:feat/mix-participant-audio-input
Open

feat(room_io): mix audio from every participant into one AgentSession#6855
Darshak03 wants to merge 14 commits into
livekit:mainfrom
Darshak03:feat/mix-participant-audio-input

Conversation

@Darshak03

@Darshak03 Darshak03 commented Aug 14, 2026

Copy link
Copy Markdown

Closes #6795

Problem

AgentSession can only listen to one participant at a time: _ParticipantInputStream keeps a single _stream/_publication, so a second participant's track replaces the first instead of joining it. Use cases like an AI interview with human takeover need N participants → 1 session → 1 shared chat context.

Approach

Opt-in mixing. With AudioInputOptions.mix_participants=True, RoomIO subscribes to the microphone of every accepted participant and mixes them with rtc.AudioMixer (the same primitive BackgroundAudio uses) into the single audio input the session already consumes. One STT/LLM/TTS pipeline, one chat context, no API surface beyond the flag:

session.start(
    room=ctx.room,
    room_options=RoomOptions(audio_input=AudioInputOptions(mix_participants=True)),
)

Changes

  • voice/room_io/_input.py_ParticipantAudioInputStream keeps a _MixedSource per participant (channel + track stream + forward task) and feeds each channel into an rtc.AudioMixer; the mixed output becomes the session's input. The only change to the shared single-participant path is a _sink(participant) hook in the base class, which still returns _data_ch — video and non-mixed audio behave exactly as before.
  • voice/room_io/room_io.py — every accepted participant is added to the mix on connect and removed on disconnect. Kind filtering is unchanged, and the agent's own avatar worker (ATTRIBUTE_PUBLISH_ON_BEHALF) is excluded so the agent can't hear itself. The linked participant is still the first one and only drives the outputs (audio, transcription, chat text).
  • voice/room_io/types.py — the mix_participants option.

Notes:

  • AGC runs once on the mixed output rather than per stream (one shared AudioProcessingModule interleaved across speakers would be wrong).
  • A noise-cancellation selector now builds a processor per participant, owned by that participant's stream; a directly-passed FrameProcessor instance keeps today's shared lifetime.
  • set_participant() is a no-op for the input in this mode — listening covers everyone; toggle it with the existing audio-input enable/disable.

Not included

Per-turn speaker attribution, also mentioned in the issue. Mixing collapses everyone into one stream, so a single STT cannot label who spoke — that needs per-participant recognition, which is a separate change.

Tests

tests/test_room_io.py:

  • test_mix_participants_sums_every_participant_audio — two participants' frames come out of the input summed into one frame.
  • test_mix_participants_tracks_room_membership — RoomIO mixes both humans, skips its own avatar worker, links only the first, and drops a participant on disconnect.

uv run pytest --unit passes (the pre-existing test_sent_tokenizer failure and test_room.py errors are environment-related and reproduce on main).

Adds `AudioInputOptions.mix_participants`. When enabled, RoomIO subscribes to
the microphone of every accepted participant and mixes them (rtc.AudioMixer)
into a single input stream, so one session hears the whole room with a shared
chat context. The linked participant still drives the outputs.

Closes livekit#6795
devin-ai-integration[bot]

This comment was marked as resolved.

Darshak03 and others added 2 commits August 14, 2026 15:22
Address review feedback on the mixing input:

- ChanClosed is now caught inside the base `_forward_task`, below the
  `@log_exceptions` boundary. A mixed participant leaving closes their sink
  mid-forward, which the decorator logged as a full error traceback before the
  subclass could suppress it. The outer suppress stays for the pre-connect and
  flush sends, which sit outside that boundary.
- Warn when `mix_participants` is combined with a directly-passed
  `FrameProcessor`: one stateful filter cannot serve N concurrent speakers.
  A selector already builds one per participant.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XQ72uVTX79ocrm3KJ3LKYh
Two more review findings:

- The frame sink was resolved twice, once in the audio override and again in
  the base loop. A participant removed from the mix between those two lookups
  fell back to `_data_ch`, sending raw frames past the mixer and the AGC.
  The sink is now bound once, where the forward task is created, and the
  `_sink` hook is gone.
- With mixing, the linked participant is one of N contributors, but their
  disconnect still closed the session and left the transcription output
  pointed at the departed identity. RoomIO now relinks to another participant
  still in the mix; `close_on_disconnect` fires only when none remain.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XQ72uVTX79ocrm3KJ3LKYh
devin-ai-integration[bot]

This comment was marked as resolved.

A registered stream that delivers nothing does not merely log: the mixer waits
stream_timeout_ms on every stream for every block, so it drops below real time
and the live speakers fall behind for as long as it stays registered. Measured
with one 50ms block and one silent stream: 2s of speech arrives as 1.05s, and
the gap grows linearly — a muted observer in a 20 minute interview would leave
the audio minutes behind.

- A source joins the mixer when its track starts producing and leaves in
  _close_mixed_source, so participants who never publish never register.
- A muted track is not read at all (torn down on track_muted, resubscribed on
  track_unmuted) rather than left filling a channel nobody drains. Buffered
  frames are dropped on the way out so they can't resurface as stale audio.
- Detached input no longer starves the mixer per source: sources keep feeding
  it and _forward_mixed drops the mixed output, which was already gated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XQ72uVTX79ocrm3KJ3LKYh
devin-ai-integration[bot]

This comment was marked as resolved.

…nect audio

- Relinking after the linked participant leaves no longer notifies the session.
  That hook re-arms _aec_warmup_remaining, which was already spent, so the next
  speaking turn would substitute silence for everyone's audio and refuse
  interruptions. A relink is not a new call.
- A mixed participant's noise-cancellation processor is kept on its source and
  passed to the forward task, so its pre-connect buffer is filtered like its
  live track. Previously the mixed path never populated self._processor, and
  the buffered opening words reached the STT unfiltered.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XQ72uVTX79ocrm3KJ3LKYh
devin-ai-integration[bot]

This comment was marked as resolved.

…ntly

A source was handed to the mixer before any of its audio was flowing, so it
starved the mix while its pre-connect buffer loaded (up to 3s) and then dumped
the whole recording into a channel the mixer drains one block at a time. The
backlog never cleared: measured with a 2s buffer, the speaker already in the
call ran 1.4s behind and stayed there.

- Pre-connect audio goes straight to the session instead of through the mixer.
  It was recorded before the participant joined, so it is not concurrent with
  anyone and mixing it only builds a lasting offset.
- A source registers with the mixer when its live loop starts and unregisters
  in a finally, so a track that ends without being unpublished no longer drags
  the mix down either.
- Each subscribe gets a fresh channel and the old one is closed before its
  writer is cancelled, so late frames cannot resurface after an unmute. Mixer
  bookkeeping is keyed on the channel, so a task that is still shutting down
  cannot unregister the channel its successor just registered.
- The trailing silence that flushes the STT is only emitted when nothing is
  left mixing; injecting it mid-call would punch a hole in everyone else.
- aclose waits for pending source teardown and refuses to resurrect the mixer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XQ72uVTX79ocrm3KJ3LKYh
devin-ai-integration[bot]

This comment was marked as resolved.

- The publication advertises TF_PRECONNECT_BUFFER for the lifetime of the
  track, but PreConnectAudioHandler drops the buffer after the first read. On
  the resubscribe that follows an unmute, the second wait therefore blocked for
  the full pre_connect_audio_timeout while the new stream backed up, then
  logged a spurious timeout. Flushed track sids are now remembered and skipped.
- Pre-connect frames go straight to the session rather than through the mixer,
  so they missed the detached check _forward_mixed applies to mixed output.
  They now honour it, and audio input disabled means disabled again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XQ72uVTX79ocrm3KJ3LKYh
devin-ai-integration[bot]

This comment was marked as resolved.

…rst run

_forward_mixed_source read source.chan and source.processor on its first
execution, one loop iteration after the task was created. Room events can be
dispatched several per iteration, so a mute/unmute or track switch could
replace both before the task ever ran. The stale task then bound the
successor's channel: it wrote the old track's audio into the live mix and, on
cancellation, unregistered the channel the live task was feeding, dropping that
speaker from the conversation for good.

Both values are now passed as arguments, and the coroutine reads nothing back
off the source.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XQ72uVTX79ocrm3KJ3LKYh
devin-ai-integration[bot]

This comment was marked as resolved.

_next_mixed_participant accepted any mixed participant, and _mix_sources holds
an entry for every accepted participant including observers that publish no
microphone. When the last speaker left, the session relinked to one of those
and close_on_disconnect never ran, leaving a session nobody could talk to.

The test is now a published microphone rather than a live one. Someone who is
merely muted is a person who can unmute, and closing on them mid-conversation
would be a worse failure than the stale session this fixes; a participant with
no microphone at all cannot speak and no longer keeps the session alive.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XQ72uVTX79ocrm3KJ3LKYh
devin-ai-integration[bot]

This comment was marked as resolved.

…nversation

Sending the buffer straight to the session avoids the mixer backlog, but the
session reads the same channel the mixer writes to, so a newcomer's pre-join
words were interleaved frame by frame with whoever was already speaking. The
buffer is now dropped when other participants are already mixed in: losing one
newcomer's opening words beats garbling everyone's transcript. Nothing is lost
in the single-speaker case, which is where the feature matters.

Also stop reporting a closed sink as an error. ChanClosed derives from
Exception, so the broad handler in _flush_pre_connect swallowed it and logged a
traceback, which made the callers' suppress dead code. It is re-raised now, and
_forward_mixed treats the session channel closing as shutdown too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XQ72uVTX79ocrm3KJ3LKYh
devin-ai-integration[bot]

This comment was marked as resolved.

samples_per_channel is per channel, so the half-second flush frame allocated
only one channel's worth of samples: rtc.AudioFrame rejects it outright, and
any AudioInputOptions(num_channels=2) setup failed to close out a turn. The bug
predates this PR but the helper it was moved into is reached from more paths.

utils.audio.silence_frame already gets this right and is used elsewhere in the
package, so _silent_frame now delegates to it rather than repeating the byte
arithmetic.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XQ72uVTX79ocrm3KJ3LKYh
devin-ai-integration[bot]

This comment was marked as resolved.

_flush_pre_connect awaited the buffer for up to pre_connect_audio_timeout only
to drop every frame at the _attached check, stalling that source's entry into
the mixer, and marked the track as flushed so a later subscribe could never get
it. It now returns before touching the handler, so the buffer stays available
once audio input is re-enabled.

The marker stays set before the read for the remaining paths: wait_for_data
drops the buffer in its own finally whatever the outcome, so retrying after a
timeout or a teardown can only block for the timeout and deliver nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XQ72uVTX79ocrm3KJ3LKYh
devin-ai-integration[bot]

This comment was marked as resolved.

When other participants are already mixed in, the buffer is discarded once it
loads. Awaiting it first left the joiner's live track buffering unread in the
AudioStream's unbounded queue for up to pre_connect_audio_timeout, and the
mixer takes one block per stream per iteration paced by the live speakers, so
that head start never drained: the joiner stayed seconds behind for the rest of
the call, having gained nothing from the wait.

The check now runs before the await as well as after it. The second one still
matters: somebody can start speaking while the buffer loads.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XQ72uVTX79ocrm3KJ3LKYh
devin-ai-integration[bot]

This comment was marked as resolved.

PreConnectAudioHandler only drops its entry in wait_for_data's finally, so
skipping the read to avoid delaying a mid-conversation joiner left the decoded
recording held until the session ended. It is now read and discarded on a
background task: off the forwarding path, so the joiner still starts mixing
immediately, and cancelled on aclose, where the same finally releases it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XQ72uVTX79ocrm3KJ3LKYh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support multiple participants in a single AgentSession with a shared chat context

1 participant