feat: Microsoft Teams calls by StandIn (livekit-plugins-standin) - #6861
feat: Microsoft Teams calls by StandIn (livekit-plugins-standin)#6861alaamh wants to merge 4 commits into
Conversation
Put any LiveKit agent on a real Microsoft Teams call. StandIn (https://standin.komaa.com) is the hosted service that joins the Teams call; this plugin answers StandIn's per-call dial inside the agent worker, creates one LiveKit room per call in the user's own project, dispatches their agent by name, and relays the audio both ways. No Teams media stack, no Graph API, and the worker file stays the shape of every other example: importing the plugin registers it, STANDIN_SECRET arms it, and nothing starts except through cli.run_app(server). - livekit-plugins/livekit-plugins-standin: the plugin. TeamsCall attaches the Teams surface (caller identity, call context, the governor's goodbye) inside the entrypoint; ChatChannel ships the client side of the coming text lane. - examples/msteams: a complete single-file worker. - Hardened: HMAC handshake with an age-pruned single-use replay guard, capacity checks before crypto, pre-start watchdog, audio-idle backstop, shielded idempotent teardown, drain-aware accept gate. Verified with a live Microsoft Teams call answered end to end, plus a hermetic unit suite, mypy --strict, and ruff. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…signing time Addresses both review findings: - rtc.AudioStream and rtc.AudioSource own FFI subscriptions and internal tasks that only aclose() releases; cancelling the pump task freed neither, leaking one of each per finished call. Both are now closed in the pump's finally and in _teardown, before the socket and room teardown so a hang there cannot skip them. - The single-use handshake cache keyed entries on arrival time while verification accepts timestamps up to REPLAY_WINDOW_MS in the future, so a future-dated capture could be pruned while its signature was still valid. Entries are now keyed on the signing timestamp, making retention match validity exactly, and pruning runs on a time throttle instead of a size watermark. Two regression tests pin both behaviors.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…odbye survives a failed interrupt - The listener now defaults to port 9442, the port every StandIn plugin uses, and STANDIN_HOST overrides the bind address (default 0.0.0.0 for containers behind an ingress; 127.0.0.1 when only a local tunnel should reach it). READMEs and the funnel example follow. - The two remaining log sites that printed the caller-supplied call id raw now go through _safe(), like every other site. - interrupt() and say() no longer share one suppress block: interrupt() raises when nothing is speaking yet or the current speech disallows interruptions, and the goodbye must still be spoken. Regression test added. - AgentSession is generic upstream now; annotate AgentSession[Any] so mypy --strict is clean again.
| if call_id in self._calls: | ||
| return web.Response(status=409, text="call already has a live session") | ||
|
|
||
| # 2 MB bounds a single inbound message, matching the sibling providers: | ||
| # audio is ~856 B base64 per frame, and the protocol caps video.frame | ||
| # JPEGs to fit this envelope (sent sparsely, dropped when busy). | ||
| ws = web.WebSocketResponse(heartbeat=None, max_msg_size=2 * 1024 * 1024) | ||
| await ws.prepare(request) | ||
|
|
||
| call = _Call(self, call_id, ws) | ||
| self._calls[call_id] = call |
There was a problem hiding this comment.
🟡 Two calls dialed for the same call id at the same time can both be accepted, and one ending untracks the other
The check that refuses a second session for the same call id happens before the connection is actually accepted (await ws.prepare(request) at livekit-plugins/livekit-plugins-standin/livekit/plugins/standin/bridge.py:749-753), so two dials arriving together both pass it, and when the first one finishes it removes the second one's bookkeeping entry, leaving a live call that the listener no longer knows about.
Impact: A duplicate or rapidly retried dial can leave an untracked call running, so its room and agent job keep burning capacity and the listener's active-call count (and the capacity limit built on it) becomes wrong.
Check-then-act across the WebSocket upgrade await
_upgrade performs if call_id in self._calls: return 409 and the capacity check len(self._calls) >= self._max_connections (bridge.py:705-708, bridge.py:743-744) but only inserts into self._calls after await ws.prepare(request) (bridge.py:750-753), which yields to the event loop. Concurrent handlers therefore all observe the pre-insert state.
For the duplicate-callId case the second handler overwrites self._calls[call_id]; when the first _Call tears down, _teardown's finally calls self._bridge._release(self._call_id) (bridge.py:282) and _release pops by id unconditionally (bridge.py:773-774), removing the second, still-live call from the map. That call is then invisible to active_calls, to the capacity check, to the 409 guard, and to CallBridge.aclose()'s drain loop (bridge.py:678-683), so its room and dispatched agent are never torn down at worker shutdown.
Inserting a placeholder into self._calls before awaiting prepare, and making _release only pop when the stored object is the same _Call, closes both halves.
Prompt for agents
In livekit-plugins/livekit-plugins-standin/livekit/plugins/standin/bridge.py, CallBridge._upgrade checks `call_id in self._calls` (409 guard) and the max_connections capacity limit, but only registers the new _Call in self._calls after `await ws.prepare(request)`. That await yields to the event loop, so concurrent upgrades for the same callId (or a burst at the capacity boundary) all pass the checks before any of them registers; the later registration overwrites the earlier one. Additionally, _Call._teardown always calls bridge._release(call_id), and _release pops the entry unconditionally, so the first call finishing removes the second, still-live call from self._calls - that call then escapes active_calls accounting, the 409 guard, and the shutdown drain in CallBridge.aclose(), leaking a room and an agent job. Consider reserving the slot (inserting a placeholder or the _Call object) before awaiting ws.prepare, and making _release remove the entry only when the mapped object is the same _Call instance.
Was this helpful? React with 👍 or 👎 to provide feedback.
…elated marker churn)
| with contextlib.suppress(Exception): | ||
| session.interrupt() | ||
| with contextlib.suppress(Exception): | ||
| session.say(text) |
There was a problem hiding this comment.
🟡 Farewell line is silently dropped for voice agents that use a realtime model
The closing line the service asks the agent to speak is discarded without any log when speaking it fails (session.say(text) swallowed by contextlib.suppress(Exception) at livekit-plugins/livekit-plugins-standin/livekit/plugins/standin/call.py:231-232), which is exactly what happens for the shipped example's setup, so the caller hears nothing before the call is torn down.
Impact: Callers get cut off in silence instead of hearing the goodbye, and operators see no error explaining why.
Why say() fails for a realtime-only session and why the failure is invisible
AgentActivity.say raises RuntimeError when the session has no TTS and the realtime model does not advertise supports_say (livekit-agents/livekit/agents/voice/agent_activity.py:1417-1427). The OpenAI realtime plugin never sets supports_say, and RealtimeCapabilities.supports_say defaults to False (livekit-agents/livekit/agents/llm/realtime.py:84, livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/realtime/realtime_model.py:467-491). The shipped example builds AgentSession(llm=openai.realtime.RealtimeModel.with_azure(...)) with no TTS (examples/msteams/agent.py:51-60), so the default goodbye handler always hits that RuntimeError.
Because the call is wrapped in a bare contextlib.suppress(Exception), the failure is neither logged nor surfaced; the documented behaviour ("the default handler interrupts the current turn and says it") silently becomes a no-op. A generate_reply(instructions=...) fallback, or at minimum a log line, would make the failure visible/recoverable.
Prompt for agents
In livekit-plugins/livekit-plugins-standin/livekit/plugins/standin/call.py, TeamsCall._handle_goodbye calls session.say(text) inside contextlib.suppress(Exception). For AgentSessions built on a realtime model with no TTS (the configuration used by examples/msteams/agent.py), AgentActivity.say raises RuntimeError because RealtimeCapabilities.supports_say is False for the OpenAI realtime plugin. The result is that the governor's goodbye is never spoken and nothing is logged, so the advertised behaviour silently does nothing. Consider logging the failure (logger.warning/exception) instead of silently suppressing, and/or falling back to session.generate_reply(instructions=...) when say() is unsupported so the line still reaches the caller.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if asyncio.iscoroutine(result): | ||
| task = asyncio.ensure_future(result) | ||
| task.add_done_callback( | ||
| lambda t: t.exception() if not t.cancelled() else None # surface, never raise | ||
| ) |
There was a problem hiding this comment.
🟡 User-supplied async call-context handlers can be cancelled mid-run by garbage collection
The background job that runs a developer's asynchronous context/goodbye handler is started without keeping a reference to it (asyncio.ensure_future(result) at livekit-plugins/livekit-plugins-standin/livekit/plugins/standin/call.py:242), so Python can collect and cancel it before it finishes, meaning handler work can vanish silently.
Impact: An agent's own reaction to call context (or to the goodbye) may randomly never complete, with no error reported.
Mechanism and inconsistency with the rest of the plugin
The event loop keeps only weak references to tasks. TeamsCall._invoke creates the task and attaches a done-callback, but nothing holds a strong reference, so the task is eligible for garbage collection while still pending; CPython will then cancel/destroy it ("Task was destroyed but it is pending!").
The rest of this PR explicitly guards against this: service.py:127-131 keeps startup tasks in a module-level set with the comment "The loop holds only weak references to tasks; keeping ours in a module set stops the startup task from being garbage-collected mid-flight", and bridge.py:543-546 keeps a self._tasks set with a discard callback. _invoke is a @staticmethod so it has no instance set to use; adding a module-level set (or making it an instance method that reuses one) fixes it.
Prompt for agents
TeamsCall._invoke in livekit-plugins/livekit-plugins-standin/livekit/plugins/standin/call.py schedules a user's coroutine handler with asyncio.ensure_future but never stores the resulting task. Since the event loop holds only weak references, the task can be garbage-collected while still pending and be destroyed mid-execution. The same file's siblings already handle this correctly (see the module-level _startup_tasks set in service.py and the per-call _tasks set in bridge.py). Keep a strong reference to the spawned task (e.g. a module-level set, discarded in the done callback) so handler coroutines always run to completion.
Was this helpful? React with 👍 or 👎 to provide feedback.
2 participants
Put any LiveKit agent on a real Microsoft Teams call.
Your AI teammate in Microsoft Teams calls: a colleague dials your Teams bot (or joins
a test meeting), and the agent you already built with
AgentSessionanswers withvoice, tools, turn detection, everything. No Teams media stack, no Graph API, no
Windows media SDK, and nothing about your agent changes.
StandIn is the hosted service that owns the Teams side:
it joins the call as a compliant Teams bot and relays the audio. This plugin is the
other half, and it is fully standalone: it answers StandIn's per-call dial inside
your worker process, creates one LiveKit room per call in your own project,
dispatches your agent into it by name, and relays the audio both ways.
See it answer
A real Microsoft Teams call, answered by the example worker exactly as committed
here (Azure OpenAI realtime, greeting the caller by name):
Screenshot of the live call, and the original clip with audio:
teams-call.png | teams-call.mp4
What your agent becomes
This PR ships the plugin (
livekit-plugins/livekit-plugins-standin) and acomplete single-file example (
examples/msteams). What that unlocks for aLiveKit agent, today and next:
Available now
and usage questions live, grounded in your product documentation - the
grounding is your agent's own RAG/tool stack, unchanged.
participant, with participant-count and "stay quiet unless directly
addressed" context fed to the agent. Following the screen share arrives with
the vision surface on the roadmap.
markets. The audio relay is language-agnostic, so this is purely your model
stack's strength.
On the roadmap below
working?", it walks the caller through the steps or the fix on its video,
visually and out loud.
video while it works, so people see progress instead of waiting.
voice conversation.
The whole integration
One line inside your entrypoint. The file is shaped like every other agent
example, and nothing starts except through
cli.run_app(server):There is no bootstrap call: importing the plugin registers it with the worker,
and setting
STANDIN_SECRETarms it. The listener starts with the worker andstops with it.
callcarriescaller_name,tenant_id,user_id(the caller's AAD id when Teamsprovides one) and
direction, so the agent greets the caller by name before thefirst word. The governor's goodbye is handled for you: when StandIn is about to end
the call, the default handler interrupts the current turn and speaks the line, so
the caller actually hears it.
CallInfo.from_job(ctx).is_teams_callis False for jobs dispatched by anythingelse, so one worker can serve Teams, SIP and web rooms at the same time.
Try it in five minutes (Sandbox: no Microsoft setup at all)
The Sandbox tier gives you a throwaway Teams meeting on StandIn's own bot, so there
is nothing to register with Microsoft.
cd examples/msteams && uv syncSign in at standin.komaa.com, create a Sandbox
connection, and copy the connection secret.
Put the secret and your LiveKit project in
.env:Run the worker and expose its port (any tunnel works; Tailscale shown):
Register
wss://<your-host>/msteams/callingas the connection's Agent voiceURL in the portal, then join the sandbox meeting from any Teams client.
Your agent picks up. Talk to it.
Going real (Free tier: your own Teams bot)
The Free tier walks you through registering your own Azure bot, so callers dial
your bot in your tenant. The worker does not change at all: same
agent.py, same.env, same funnel. You pair the identity in the portal, pointthe Azure bot's calling webhook at the
standin.komaa.comendpoint the portalshows you, and register the same Agent voice URL. Then call your bot from Teams.
What the community gets
AgentSessioncombination works unchanged; the caller is an ordinary audiotrack in the room.
capacity checks before crypto, a pre-start watchdog, an audio-idle backstop
that ends dead calls, and bounded teardown everywhere.
state reach the agent as data topics.
ChatChannelships the client side of theTeams text lane, so answering chat will be one async handler when the hosted
channel opens. See the roadmap.
Verified
--strict, ruff clean.through a Tailscale funnel, with the agent greeting the caller by name and
holding a multi-turn voice conversation on an Azure OpenAI realtime model.
Roadmap
same worker. The plugin already ships
ChatChannel- one async handler in,reply text out, with per-conversation ordering, redelivery dedupe, and a
typing indicator while the agent thinks. The worker dials OUT to StandIn's
chat channel, so this lane needs no port either, and on managed connections
your agent never holds a Bot Framework credential; it goes live when the
hosted channel endpoint opens.
screen-share and camera) depends on a third-party avatar runtime and lands
separately. Those frames are ignored gracefully today.