fix(voice): answer every tool call - #2289
Conversation
🦋 Changeset detectedLatest commit: 160d9e3 The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| const otherOutputs = finalOutput.filter((item) => !isAgentHandoff(item)); | ||
| agentTask = handoffs[0]?.agent; | ||
| toolOutput = agentTask | ||
| ? otherOutputs.length === 0 | ||
| ? undefined | ||
| : otherOutputs.length === 1 | ||
| ? otherOutputs[0] | ||
| : otherOutputs | ||
| : otherOutputs; |
There was a problem hiding this comment.
🟡 A tool that returns a transfer together with a value inside a list silently loses the value
The value attached to an agent transfer is dropped (handoffs[0]?.agent at agents/src/voice/generation.ts:446) when the transfer is returned inside a list, so the model never sees the result the tool meant to report.
Impact: A tool that hands off and also returns data in a list answers the model with an empty result instead of the data.
Array branch ignores `AgentHandoff.returns` while the scalar branch honours it
In createToolOutput, the new array branch (agents/src/voice/generation.ts:426-454) extracts only handoffs[0]?.agent and builds toolOutput from the other array entries. For return [handoff({ agent, returns: 'ok' })], otherOutputs is empty, so toolOutput becomes undefined — the returns: 'ok' payload is discarded and replyRequired is set to false at agents/src/voice/generation.ts:487.
The non-array branch (agents/src/voice/generation.ts:454-457) does the opposite: it uses finalOutput.returns as the tool output. So the same handoff behaves differently depending on whether it is wrapped in an array.
A fix would merge any returns value of the matched handoff into otherOutputs before collapsing.
Prompt for agents
In createToolOutput (agents/src/voice/generation.ts), the newly added array-handling branch picks the handoff's agent but never uses that handoff's `returns` field, whereas the existing scalar branch does (`toolOutput = finalOutput.returns`). As a result, `return [handoff({ agent, returns: 'ok' })]` produces an empty tool output with replyRequired=false, silently dropping 'ok'. Consider folding `handoffs[0].returns` (when not undefined) into the collected `otherOutputs` before collapsing to a single value/array, so both shapes behave consistently.
Was this helpful? React with 👍 or 👎 to provide feedback.
| get hasToolReply() { | ||
| return functionCallOutputs.some((output) => output.replyRequired); | ||
| }, | ||
| cancelToolReply() { | ||
| for (const output of functionCallOutputs) output.replyRequired = false; | ||
| }, |
There was a problem hiding this comment.
🟡 Session report now contains an extra field and a non-serializable callback for tool-execution events
The tool-execution event is copied field-by-field into the uploaded report ({...event} at agents/src/voice/report.ts:160), which now also picks up the two newly added members, so the report carries a field that does not exist in the reference wire format plus a function value.
Impact: Consumers of the session report see an unexpected extra field for tool-execution events, and the report object holds a value that cannot be serialized.
New getter/method on FunctionToolsExecutedEvent leak into the wire format
createFunctionToolsExecutedEvent (agents/src/voice/events.ts:202-215) now returns an object with an own enumerable getter hasToolReply and an own method cancelToolReply. eventToJSON spreads the event (agents/src/voice/report.ts:160) and then runs toSnakeCaseDeep (agents/src/voice/report.ts:109-127), which copies every own enumerable entry. The result gains has_tool_reply: boolean and cancel_tool_reply: <function> for every function_tools_executed event pushed at agents/src/voice/report.ts:250. The Python model exposes these as a property/method, so they are absent from its model_dump().
Defining the two members as non-enumerable, or excluding them in eventToJSON's switch (like speechHandle is deleted for speech_created), would keep the wire shape aligned.
Prompt for agents
FunctionToolsExecutedEvent gained an own enumerable getter (hasToolReply) and an own method (cancelToolReply) in agents/src/voice/events.ts. agents/src/voice/report.ts eventToJSON spreads every event object and toSnakeCaseDeep copies all own enumerable entries, so serialized session reports now contain has_tool_reply and a cancel_tool_reply function value for function_tools_executed events — neither exists in the Python wire format. Consider making these members non-enumerable (Object.defineProperty) or explicitly stripping them in eventToJSON's per-type switch, similar to how speechHandle is deleted for speech_created.
Was this helpful? React with 👍 or 👎 to provide feedback.
| readonly hasToolReply: boolean; | ||
| cancelToolReply(): void; |
There was a problem hiding this comment.
🟡 Newly added public event members and helper function are undocumented
The two new members added to the tool-execution event and the new interrupted-output helper carry no documentation comments (hasToolReply/cancelToolReply at agents/src/voice/events.ts:191-192), which the repository's contribution rules require for every new method or interface member.
Impact: The public API documentation generated for the framework is missing entries for the newly exposed members.
Repository rule
CONTRIBUTING.md states: "If writing new methods/interfaces/enums/classes, document them. This project uses TypeDoc for automatic API documentation generation, and every new addition has to be properly documented."
Undocumented additions in this PR:
FunctionToolsExecutedEvent.hasToolReplyandFunctionToolsExecutedEvent.cancelToolReply()(agents/src/voice/events.ts:191-192)- the exported
interruptedToolOutput()helper (agents/src/voice/generation.ts:495)
FunctionCallOutput.replyRequired (agents/src/llm/chat_context.ts:560-561) does have a doc comment and is fine.
| readonly hasToolReply: boolean; | |
| cancelToolReply(): void; | |
| /** Whether any completed tool result still expects the model to reply. */ | |
| readonly hasToolReply: boolean; | |
| /** Suppress the model reply for every tool result in this batch. */ | |
| cancelToolReply(): void; |
Was this helpful? React with 👍 or 👎 to provide feedback.
Ports livekit/agents#6785 to the Node.js voice pipeline.
Every completed tool call now produces exactly one output. Interrupted realtime results are committed locally and synced to the provider, interrupted handoffs are recorded as silent errors, and
StopResponse/invalid outputs no longer leave calls unanswered. Realtime providers honorreplyRequiredwhere possible.Verification
pnpm test agents(2125 passed, 5 skipped)pnpm test plugins/google(42 passed, 2 skipped)pnpm exec vitest run plugins/phonic --passWithNoTests(no test files)pnpm buildcue-clivoice validation (sid_43f154e34713): interrupted assistant item,lookupOrdertool output, then a fresh assistant responseSource diff coverage
livekit-agents/livekit/agents/llm/chat_context.pyagents/src/llm/chat_context.ts: addedFunctionCallOutput.replyRequired, defaulting totrue, including serialization.livekit-agents/livekit/agents/llm/llm.pyPromise<FunctionCallOutput>and has no nullable equivalent example.livekit-agents/livekit/agents/llm/utils.pyagents/src/voice/generation.ts, where JS ownsStopResponseand invalid-output sanitization. Every completed call now gets an output.livekit-agents/livekit/agents/voice/agent_activity.pyagents/src/voice/agent_activity.ts: preserve and emit all completed interrupted outputs, sync interrupted realtime outputs, and derive reply behavior from each output.livekit-agents/livekit/agents/voice/events.pyagents/src/voice/events.ts: JS already used non-null parallel arrays; addedcancelToolReply()andhasToolReplybacked by output flags.livekit-agents/livekit/agents/voice/generation.pyagents/src/voice/generation.ts: non-optional outputs, output-level reply flags, invalid-output errors, multiple-handoff errors, and silent failed interrupted handoffs.livekit-agents/livekit/agents/voice/remote_session.pyagents/src/voice/remote_session.ts: removed the obsolete null-output filter.livekit-plugins/livekit-plugins-aws/livekit/plugins/aws/experimental/realtime/realtime_model.pylivekit-plugins/livekit-plugins-google/livekit/plugins/google/realtime/realtime_api.pyplugins/google/src/realtime/realtime_api.ts: SILENT scheduling where supported and explicit Gemini/Vertex warnings otherwise.livekit-plugins/livekit-plugins-google/livekit/plugins/google/utils.pyplugins/google/src/realtime/realtime_api.ts, because JS keeps function-response construction in the realtime session rather thanutils.ts.livekit-plugins/livekit-plugins-phonic/livekit/plugins/phonic/realtime/realtime_model.pyplugins/phonic/src/realtime/realtime_model.ts: silent outputs do not open a speaking turn.livekit-plugins/livekit-plugins-ultravox/livekit/plugins/ultravox/realtime/realtime_model.pytests/fake_realtime.pyagents/src/voice/realtime_tool_output_commit.test.ts.tests/test_plugin_google_realtime.pyplugins/google/src/realtime/realtime_api.test.ts, adapted to JS's private response-builder harness and SDK object shape.tests/test_realtime_cancel_tool_reply.pyagents/src/voice/realtime_tool_output_commit.test.ts.tests/test_tool_output_per_call.pyagents/src/voice/generation_tools.test.ts, where JS directly tests tool sanitization.tests/test_tool_results_preserved_on_interruption.pyagents/src/voice/agent_activity.test.tsandagents/src/voice/realtime_tool_output_commit.test.ts, matching JS's activity/realtime test split.Source PR: livekit/agents#6785
Ported from livekit/agents#6785
Original PR description
Problem: A tool call could end with no output at all — dropped when the user interrupted, or never produced when a tool raised
StopResponseor returned an invalid value. A realtime model holds each call it emitted open until it is answered, so Gemini Live stopped responding and latergenerate_reply()calls produced no generation (#6569). A pipeline LLM re-issues the call instead and runs its side effects a second time.Fix: Every call now gets exactly one output. An interrupted turn commits the results of tools that finished and delivers them to the realtime session; a handoff answers as an error, since the interruption left it unapplied.
Behavior:
FunctionCallOutput.reply_requiredsays when no reply is wanted, and each realtime plugin honours it as its provider allows — Gemini withSILENTscheduling, Ultravox withagent_reaction="listens", Phonic by not opening a turn — or warns that it cannot. That replaces the blanket warning about models which generate tool replies on their own.Breaking:
FunctionToolsExecutedEvent.function_call_outputsislist[FunctionCallOutput], no longer optional. A tool raisingStopResponsenow leaves a call and an empty output in history where it previously left nothing.Replaces #6600. Fixes #6569