feat(room_io): mix audio from every participant into one AgentSession - #6854
feat(room_io): mix audio from every participant into one AgentSession#6854Darshak-Jankat wants to merge 1 commit into
Conversation
Adds `AudioInputOptions.mix_participants`. When enabled, RoomIO subscribes to the microphone of every accepted participant and mixes them (rtc.AudioMixer) into a single input stream, so one session hears the whole room with a shared chat context. The linked participant still drives the outputs. Closes #6795
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| # the sink of a mixed participant is closed when they leave, mid-forwarding | ||
| with contextlib.suppress(aio.ChanClosed): |
There was a problem hiding this comment.
🟡 Leaving a participant produces a scary error traceback in the logs
The safeguard that is meant to silently ignore a departing participant's closed audio pipe is placed outside the code that reports errors (contextlib.suppress(aio.ChanClosed) at livekit-agents/livekit/agents/voice/room_io/_input.py:460), so every time a mixed participant leaves or the session shuts down while their audio is flowing, a full error traceback is written to the logs.
Impact: Normal participant departures and normal session shutdown look like crashes in production logs, making real failures harder to spot.
Mechanism: log_exceptions on the base forward task fires before the suppress
_ParticipantInputStream._forward_task is decorated with @log_exceptions(logger=logger) (livekit-agents/livekit/agents/voice/room_io/_input.py:135-136), and log_exceptions logs via logger.exception(...) and then re-raises (livekit-agents/livekit/agents/utils/log.py:16-23).
When a mixed participant is removed, remove_participant closes their channel (_input.py:319) while the per-participant forward task may be awaiting/entering await sink.send(frame) inside the base loop (_input.py:160). aio.Chan.send raises ChanClosed once close() has been called (livekit-agents/livekit/agents/utils/aio/channel.py:91-93). The base task logs the traceback first; only afterwards does the subclass's contextlib.suppress(aio.ChanClosed) swallow the exception. The same happens in aclose (_input.py:511-515), which closes each source channel before cancelling its task.
To actually get silent handling, the ChanClosed must be caught inside (or around) the send in the base _forward_task, i.e. below the log_exceptions boundary.
Prompt for agents
In livekit-agents/livekit/agents/voice/room_io/_input.py, _ParticipantAudioInputStream._forward_task wraps its body (including the call to super()._forward_task) in contextlib.suppress(aio.ChanClosed) to tolerate a mixed participant's channel being closed while frames are being forwarded. However the base _ParticipantInputStream._forward_task is decorated with @log_exceptions, which logs the exception with a full traceback via logger.exception before re-raising it, so the suppression happens too late: every participant removal (remove_participant closes the channel) and every aclose() with active mixed audio emits a bogus error traceback. Handle ChanClosed inside the base forward loop (e.g. break out of the loop when the sink is closed, or catch ChanClosed around the send) so it never crosses the log_exceptions boundary, keeping the suppress only as a secondary safety net.
Was this helpful? React with 👍 or 👎 to provide feedback.
| noise_cancellation = self._noise_cancellation | ||
| auto_close_noise_cancellation = False | ||
| if callable(noise_cancellation): | ||
| noise_cancellation = noise_cancellation(NoiseCancellationParams(participant, track)) | ||
| if isinstance(noise_cancellation, rtc.FrameProcessor): | ||
| if self._mix_participants: | ||
| # each mixed participant gets its own processor, tied to its stream | ||
| auto_close_noise_cancellation = isinstance(noise_cancellation, rtc.FrameProcessor) | ||
| elif isinstance(noise_cancellation, rtc.FrameProcessor): | ||
| self._update_processor(noise_cancellation) | ||
| else: | ||
| self._update_processor(None) |
There was a problem hiding this comment.
🟡 A single noise-cancellation processor is fed by all speakers at once, degrading audio
When mixing is enabled and noise cancellation was supplied as a ready-made processor object, that one processor is handed to every participant's audio stream (rtc.AudioStream.from_track(..., noise_cancellation=noise_cancellation) at livekit-agents/livekit/agents/voice/room_io/_input.py:438-445), so several speakers' audio is pushed through the same stateful filter simultaneously and can come out distorted.
Impact: With multi-participant mixing, noise cancellation can degrade or mangle everyone's audio instead of cleaning it, hurting transcription quality.
Mechanism: shared FrameProcessor instance across N concurrent streams
In _create_stream (_input.py:425-445), the per-participant processor is only created when noise_cancellation is callable (a selector); in that case mixing sets auto_close_noise_cancellation=True so each stream owns its own processor. But when the user passes a rtc.FrameProcessor instance directly, self._noise_cancellation is not callable and the very same object is passed to each AudioStream.from_track — one per mixed participant (_on_track_available, _input.py:391).
A FrameProcessor carries per-stream state and per-stream metadata: the runtime calls _on_stream_info_updated(room_name=..., participant_identity=..., publication_sid=...) per stream (see the test double at tests/test_room_io.py:87-96), so with several streams the metadata of the last one wins, and frames from different speakers are interleaved into a single filter instance. This is exactly the reason the PR stopped sharing one AudioProcessingModule across speakers (_input.py:417-419), but the same reasoning was not applied to a directly-passed processor.
Possible handling: when mix_participants is enabled, reject/warn on a directly-passed FrameProcessor instance (asking for a selector instead), or create a per-participant processor.
Prompt for agents
In _ParticipantAudioInputStream._create_stream (livekit-agents/livekit/agents/voice/room_io/_input.py), when mix_participants is enabled and noise_cancellation was given as a concrete rtc.FrameProcessor instance (not a selector callable), the exact same processor object is passed to every participant's rtc.AudioStream. Frame processors are stateful and receive per-stream metadata via _on_stream_info_updated, so sharing one across N concurrent participant streams interleaves unrelated speakers through one filter — the same hazard the PR avoids for the shared AudioProcessingModule. Decide on a supported behaviour: either log a warning / raise at construction time when mix_participants is combined with a directly-passed FrameProcessor (directing users to a NoiseCancellationSelector so each participant gets its own processor), or build a per-participant processor internally.
Was this helpful? React with 👍 or 👎 to provide feedback.
|
Superseded by #6855, reopened from the correct account. Same commit. |
2 participants
Closes #6795
Problem
AgentSessioncan only listen to one participant at a time:_ParticipantInputStreamkeeps a single_stream/_publication, so a second participant's track replaces the first instead of joining it. Use cases like an AI interview with human takeover need N participants → 1 session → 1 shared chat context.Approach
Opt-in mixing. With
AudioInputOptions.mix_participants=True,RoomIOsubscribes to the microphone of every accepted participant and mixes them withrtc.AudioMixer(the same primitiveBackgroundAudiouses) into the single audio input the session already consumes. One STT/LLM/TTS pipeline, one chat context, no API surface beyond the flag:Changes
voice/room_io/_input.py—_ParticipantAudioInputStreamkeeps a_MixedSourceper participant (channel + track stream + forward task) and feeds each channel into anrtc.AudioMixer; the mixed output becomes the session's input. The only change to the shared single-participant path is a_sink(participant)hook in the base class, which still returns_data_ch— video and non-mixed audio behave exactly as before.voice/room_io/room_io.py— every accepted participant is added to the mix on connect and removed on disconnect. Kind filtering is unchanged, and the agent's own avatar worker (ATTRIBUTE_PUBLISH_ON_BEHALF) is excluded so the agent can't hear itself. The linked participant is still the first one and only drives the outputs (audio, transcription, chat text).voice/room_io/types.py— themix_participantsoption.Notes:
AudioProcessingModuleinterleaved across speakers would be wrong).FrameProcessorinstance keeps today's shared lifetime.set_participant()is a no-op for the input in this mode — listening covers everyone; toggle it with the existing audio-input enable/disable.Not included
Per-turn speaker attribution, also mentioned in the issue. Mixing collapses everyone into one stream, so a single STT cannot label who spoke — that needs per-participant recognition, which is a separate change.
Tests
tests/test_room_io.py:test_mix_participants_sums_every_participant_audio— two participants' frames come out of the input summed into one frame.test_mix_participants_tracks_room_membership— RoomIO mixes both humans, skips its own avatar worker, links only the first, and drops a participant on disconnect.uv run pytest --unitpasses (the pre-existingtest_sent_tokenizerfailure andtest_room.pyerrors are environment-related and reproduce onmain).🤖 Generated with Claude Code
https://claude.ai/code/session_01XQ72uVTX79ocrm3KJ3LKYh