fix(xai): align realtime Voice Agent defaults and API coverage - #2255
fix(xai): align realtime Voice Agent defaults and API coverage#2255rosetta-livekit-bot[bot] wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: 13f15af The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| } catch (error) { | ||
| delete this.responseCreatedFutures[eventId]; | ||
| if (!forceMessageSent) this.dropPendingSayTag(eventId); | ||
| if (doneFut.done) return await doneFut.await; | ||
| throw error; |
There was a problem hiding this comment.
🔴 Scripted speech stops working after the voice connection reconnects
A scripted-speech request that is still waiting when the connection drops leaves its tracking tag in the pending queue forever (the tag is only removed when the message was never sent, at plugins/xai/src/realtime/realtime_model.ts:216), so every later scripted-speech request is matched to the wrong reply and eventually fails.
Impact: After a reconnect, say() on xAI realtime times out (~10s) and the reply that should have been spoken is cancelled instead, so the agent stays silent.
Pending say tag queue desynchronizes because reconnect clears futures but not the FIFO tag list
On reconnect the base session rejects all pending response futures with Session reconnected and clears both responseCreatedFutures and discardedEventIds (plugins/openai/src/realtime/realtime_model.ts:1103-1110). In the xAI subclass, the rejected say() lands in the catch block: since forceMessageSent is true, dropPendingSayTag(eventId) is skipped and no stale-cleanup timer is scheduled (scheduleStaleSayCleanup is only invoked from discardSay and the timeout path, plugins/xai/src/realtime/realtime_model.ts:202-210,255-263). The dead id therefore stays in pendingSayEventIds indefinitely.
The next say() pushes its own id behind the stale one. When the server sends response.created, handleResponseCreated shifts the stale id and stamps it as client_event_id (plugins/xai/src/realtime/realtime_model.ts:322-328); the base handler finds no matching future, so the new say() never resolves and rejects after 10s, at which point its id is added to discardedEventIds and the following response is cancelled.
A fix would clear pendingSayEventIds (and staleSayTimers) whenever the session reconnects, and drop the tag on any terminal failure of a sent say, not just aborts before sending.
Prompt for agents
In plugins/xai/src/realtime/realtime_model.ts, the FIFO correlation list pendingSayEventIds can retain dead entries. When the underlying OpenAI base session reconnects it rejects every entry in responseCreatedFutures with 'Session reconnected' and clears responseCreatedFutures/discardedEventIds, but nothing clears the xAI-side pendingSayEventIds or staleSayTimers. Because say()'s catch block only drops the pending tag when the force_message was never sent (forceMessageSent === false), a say that was in flight at reconnect time leaves its id in pendingSayEventIds permanently. handleResponseCreated then shifts that dead id onto the next response.created, so subsequent say() calls never resolve (they time out after 10s) and the response after that gets cancelled as 'discarded'. Consider hooking the session-reconnected path (or overriding whatever runs on reconnect) to reset pendingSayEventIds/staleSayTimers, and make say() drop or schedule cleanup of its tag on any terminal rejection, not only on pre-send aborts.
Was this helpful? React with 👍 or 👎 to provide feedback.
| say( | ||
| _text: string | ReadableStream<string>, | ||
| _options: { signal?: AbortSignal } = {}, | ||
| ): Promise<GenerationCreatedEvent> { | ||
| throw new Error(`${this.constructor.name} does not implement say(); use a TTS model instead`); | ||
| } |
There was a problem hiding this comment.
🟡 New public methods and protocol types added without documentation comments
The newly added scripted-speech entry point is exported without any documentation comment (say() at agents/src/llm/realtime.ts:173), which the repository's contribution rules require for every new public method, interface or class.
Impact: The generated API documentation for the new public surface is empty, and contributors have no stated contract for the new method.
Affected additions
CONTRIBUTING.md states: "If writing new methods/interfaces/enums/classes, document them. This project uses TypeDoc for automatic API documentation generation, and every new addition has to be properly documented."
Undocumented new public API in this PR:
RealtimeSession.say()inagents/src/llm/realtime.ts:173-178(no TSDoc describing behaviour, thrown error, or thesignaloption)RealtimeSession.say()override inplugins/xai/src/realtime/realtime_model.ts:159-225ForceMessageItemCreateandForceMessageCreateEventinplugins/openai/src/realtime/api_proto.ts:397-401and429-432
(The new supportsSay capability is documented and is fine.)
Was this helpful? React with 👍 or 👎 to provide feedback.
| private discardSay(eventId: string): void { | ||
| delete this.responseCreatedFutures[eventId]; | ||
| if (!this.discardedEventIds.has(eventId)) { | ||
| this.sendEvent({ type: 'response.cancel' }); | ||
| this.discardedEventIds.add(eventId); | ||
| this.scheduleStaleSayCleanup(eventId); | ||
| } | ||
| this.ensurePendingSayTag(eventId); | ||
| } |
There was a problem hiding this comment.
🟡 Cancelled scripted speech can silently suppress the agent's reply to the user
When a scripted speech request is cancelled after it was sent, its tracking tag is deliberately kept in the pending queue (ensurePendingSayTag at plugins/xai/src/realtime/realtime_model.ts:262), so whichever reply the server produces next — including a genuine reply to the user — is thrown away.
Impact: For up to 10 seconds after a cancelled scripted line, the agent can silently drop its answer to what the user just said.
Mechanism: discarded tag is applied to an arbitrary next response
discardSay adds the event id to discardedEventIds and re-inserts it into pendingSayEventIds (plugins/xai/src/realtime/realtime_model.ts:255-263). handleResponseCreated stamps the head of pendingSayEventIds onto any incoming response.created that lacks a client_event_id (plugins/xai/src/realtime/realtime_model.ts:322-328). With server VAD (create_response: true, the xAI default at plugins/xai/src/realtime/realtime_model.ts:21-28), the server autonomously creates a response as soon as the user finishes speaking. If such a response is the first one to arrive after the cancellation, the base handler sees a discarded id and cancels/discards it (plugins/openai/src/realtime/realtime_model.ts:1462-1472), so the user's turn goes unanswered. The tag is only released by the 10s cleanup timer (plugins/xai/src/realtime/realtime_model.ts:265-277).
A safer correlation would only discard responses whose creation timestamp/order can actually be attributed to the cancelled force_message (e.g. discard only until the very next response, and drop the tag as soon as any response has been observed).
Was this helpful? React with 👍 or 👎 to provide feedback.
e89bf78 to
f1d4957
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
| for (const [index, item] of newChatCtx.items.entries()) { | ||
| const remoteItem = remoteCtx.getById(item.id); | ||
| if ( | ||
| remoteItem && | ||
| JSON.stringify(remoteItem.toJSON(true)) !== JSON.stringify(item.toJSON(true)) && | ||
| !diffOps.toRemove.includes(item.id) && | ||
| !diffOps.toCreate.some(([, id]) => id === item.id) | ||
| ) { | ||
| diffOps.toRemove.push(item.id); | ||
| diffOps.toCreate.push([newChatCtx.items[index - 1]?.id ?? null, item.id]); | ||
| } | ||
| } |
There was a problem hiding this comment.
🟡 Conversation history items are needlessly deleted and re-sent to the model on every history update
Existing history entries are compared against the model's copy using fields the model never returns (JSON.stringify(remoteItem.toJSON(true)) at plugins/openai/src/realtime/realtime_model.ts:737), so entries such as interrupted replies never compare equal and get deleted and re-uploaded on every single history update.
Impact: Each agent handoff or history change makes the assistant re-send the same messages to the provider, adding avoidable latency and round trips and risking history-update timeouts.
Why the comparison never converges: client-only metadata is not mirrored back
computeChatCtxDiff (agents/src/llm/utils.ts:701) is id-based, and this new block adds a content-equality pass. However, the remote mirror is built from server echoes via openAIItemToLivekitItem (plugins/openai/src/realtime/realtime_model.ts:2222-2260), which only sets id, role, and content. ChatMessage.toJSON (agents/src/llm/chat_context.ts:396-419) additionally serializes interrupted, metrics, transcriptConfidence, and extra, and FunctionCallOutput serializes isError.
The agent's own context sets those fields (e.g. interrupted at agents/src/voice/agent_activity.ts:3509, metrics at agents/src/voice/agent_activity.ts:2532). Therefore, for any interrupted assistant message (or error tool output), the JSON strings differ permanently: after the delete+create round trip the server echo still lacks interrupted: true, so the next updateChatCtx schedules the same delete+create again, indefinitely.
A robust check should compare only the fields actually sent to the provider (role plus text/image content), e.g. reuse the message-equality helper that ignores client-only metadata, or compare the serialized livekitItemToOpenAIItem payloads.
Prompt for agents
In plugins/openai/src/realtime/realtime_model.ts, createChatCtxUpdateEvents now augments the id-based diff with a content-change check that compares JSON.stringify(remoteItem.toJSON(true)) against JSON.stringify(item.toJSON(true)). The remote mirror is reconstructed from server echoes by openAIItemToLivekitItem, which only carries id, role and content, while the agent's local items also carry client-only metadata (interrupted, metrics, transcriptConfidence, extra for messages; isError for function call outputs). As a result, items like interrupted assistant messages will always compare as changed, so they are deleted and re-created on the server on every updateChatCtx call, and the comparison never converges. Change the comparison so it only considers state actually mirrored by the provider (for example compare role plus the text/image content, or compare the payload produced by livekitItemToOpenAIItem) so items are only recreated when the provider-visible representation truly differs.
Was this helpful? React with 👍 or 👎 to provide feedback.
| const generation = await realtimeSession.say(text, { signal: abortController.signal }); | ||
| await this.realtimeGenerationTask(speechHandle, generation, {}, abortController); |
There was a problem hiding this comment.
🟡 Scripted speech on realtime models keeps waiting after the user interrupts
The scripted-speech step waits for the model's reply without watching for an interruption (await realtimeSession.say(...) at agents/src/voice/agent_activity.ts:1104), so a barge-in during that wait does not release the turn until the model replies or the internal 10 second timeout fires.
Impact: If a user interrupts right after the agent is asked to speak a scripted line, the agent can appear stuck and the next turn is delayed by up to ten seconds.
Divergence from the realtime reply path
SpeechHandle.interrupt() only resolves an interrupt future (agents/src/voice/speech_handle.ts:290-300); it does not abort the speech task's AbortController. Cooperative tasks must therefore race the pending work against the interrupt future.
The realtime reply task does exactly that (agents/src/voice/agent_activity.ts:3870-3876): it wraps generateReply() in speechHandle.waitIfNotInterrupted([generationPromise]), aborts the controller and returns when interrupted (also pre-attaching a catch to avoid an orphan rejection).
realtimeSayTask awaits realtimeSession.say() directly, so the task stays alive (and the speech handle stays not-done) until the xAI session resolves the say future — which only happens on response.created or the 10s say timeout in plugins/xai/src/realtime/realtime_model.ts:202-210.
Was this helpful? React with 👍 or 👎 to provide feedback.
fix(xai): align realtime Voice Agent defaults and add scripted speech Align xAI Realtime voice defaults, add native scripted speech support via say(), emit interim and one final user transcript per turn, and settle rejected realtime chat context updates. Bring realtime error handling to parity with the Python plugin: treat fatal codes (quota / auth / billing) as non-recoverable on a failed response.done, break the receive loop on them instead of reconnecting, and stop reporting input_audio_buffer_commit_empty as an error when server turn detection is on. Rebased onto main, which independently landed overlapping realtime work (isFatalError, chatCtxEventFutures, allSettled, protected members, exported generation internals, closeCurrentGeneration, the onmessage try/catch) plus a ToolContext class API and computeChatCtxDiff.toUpdate. Kept main throughout; only the deltas unique to this branch are retained: - protected _options/responseCreatedFutures/discardedEventIds (xAI subclass) - loggableEvent truncates the GA response.output_audio.delta payload - wsCloseFuture done-guard in onmessage, alongside main wsConn.close() - input_audio_buffer_commit_empty suppression under server turn detection - handleError responseCreatedFutures else-branch Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> @
47a9084 to
13f15af
Compare
| if (this.sessionConnectedAt > 0) { | ||
| const realtimeModel = this.realtimeModel; | ||
| this.emit('metrics_collected', { | ||
| type: 'realtime_model_metrics', | ||
| label: realtimeModel.label(), | ||
| requestId: 'session_close', | ||
| timestamp: Date.now(), | ||
| durationMs: 0, | ||
| sessionDurationMs: Date.now() - this.sessionConnectedAt, | ||
| ttftMs: -1, | ||
| cancelled: false, | ||
| inputTokens: 0, | ||
| outputTokens: 0, | ||
| totalTokens: 0, | ||
| tokensPerSecond: 0, | ||
| inputTokenDetails: { | ||
| audioTokens: 0, | ||
| textTokens: 0, | ||
| imageTokens: 0, | ||
| cachedTokens: 0, | ||
| }, | ||
| outputTokenDetails: { audioTokens: 0, textTokens: 0, imageTokens: 0 }, | ||
| metadata: { modelName: realtimeModel.model, modelProvider: realtimeModel.provider }, | ||
| } satisfies metrics.RealtimeModelMetrics); | ||
| } |
There was a problem hiding this comment.
🟡 Session-length usage numbers reported when a voice session ends are never collected
The end-of-session usage report is announced (emit('metrics_collected', ...) at plugins/xai/src/realtime/realtime_model.ts:96) only after the framework has already stopped listening for usage reports from the session, so the numbers are discarded.
Impact: Session duration for xAI realtime sessions never shows up in metrics or usage totals.
Mechanism: listener is detached before the session is closed
AgentActivity._closeSessionResources removes the metrics listener from the realtime session (agents/src/voice/agent_activity.ts:5104) and only afterwards calls await this.realtimeSession?.close() (agents/src/voice/agent_activity.ts:5136). The new xAI close() override emits metrics_collected inside that close call, so by then no listener exists and onMetricsCollected (which feeds _usageCollector and the MetricsCollected event) is never invoked. The same holds for the handoff/detach path, which also calls off('metrics_collected', ...) before closing (agents/src/voice/agent_activity.ts:837).
Prompt for agents
plugins/xai/src/realtime/realtime_model.ts close() emits a 'metrics_collected' event carrying sessionDurationMs, but AgentActivity detaches its metrics listener from the realtime session before calling realtimeSession.close() (agents/src/voice/agent_activity.ts around lines 5104 and 5136, and the handoff detach path around line 837). As a result the session-close metric is always dropped. Either emit the session metric at a point where the framework is still subscribed (e.g. when the websocket connection ends / before the framework tears down, or from a session-level hook), or change the framework teardown order so the realtime session is closed before its metrics listener is removed.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Summary
grok-voice-latestwithgrok-transcribesay()support using xAIforce_message, including FIFO response correlation and cancellation cleanupSource diff coverage
Source diff coverage
livekit-plugins/livekit-plugins-xai/livekit/plugins/xai/realtime/__init__.py: not applicable.plugins/xai/src/realtime/index.tsalready exports the target-nativeRealtimeSessionimplementation.livekit-plugins/livekit-plugins-xai/livekit/plugins/xai/realtime/realtime_model.py: adapted toplugins/xai/src/realtime/realtime_model.ts, with the missing realtimesupportsSay/say()voice-pipeline infrastructure added inagents/src/llm/realtime.tsandagents/src/voice/agent_activity.ts, and reusable protocol/session seams added inplugins/openai/src/realtime/api_proto.tsandplugins/openai/src/realtime/realtime_model.ts. Python futures/tasks are represented with JS promises, abort signals, and stream cancellation.livekit-plugins/livekit-plugins-xai/livekit/plugins/xai/types.py: ported to the colocatedGrokRealtimeModelstype inplugins/xai/src/realtime/realtime_model.ts.tests/test_realtime/test_realtime.py: adapted toplugins/xai/src/realtime/realtime_model.test.ts. The target has no shared credential-gated realtime provider/WAV harness; its existing xAI chat-context test already covers history deletion, and the sourcesay()smoke is covered at the wire/generation boundary.tests/test_realtime/test_xai_realtime_model.py: ported toplugins/xai/src/realtime/realtime_model.test.ts, including defaults, session updates, live captions,say()cancellation/close handling, and FIFO correlation.Test plan
env -u LIVEKIT_URL -u LIVEKIT_API_KEY -u LIVEKIT_API_SECRET pnpm vitest run agents(1065 passed, 2 skipped)env -u OPENAI_API_KEY pnpm vitest run plugins/openai(56 passed, 7 skipped)pnpm vitest run plugins/xai(29 passed, 1 skipped)pnpm buildpnpm lint(passes with existing warnings)pnpm format:checkValidation notes
cue-clidispatched the xAI realtime voice agent and reachedAgentActivity.realtime_say; the configured xAI endpoint rejected the WebSocket handshake with HTTP 400 before framework assistant events could be emitted.export * asdeclarations and a missingplugins/bey/api-extractor.json; the touched packages build and emit declarations successfully.Upstream: livekit/agents#6755
Ported from livekit/agents#6755
Original PR description
Summary
grok-voice-latest(resolves togrok-voice-think-fast-2.0) and default input transcription togrok-transcribeforce_messagesupport viaRealtimeSession.say()/supports_say, emit live caption updates fromconversation.item.input_audio_transcription.updated, and forwardreasoning/ transcription / speed optionslivekit-xai-issues.mdTest plan
uv run pytest tests/test_realtime/test_xai_realtime_model.py -q --allow-uncategorized(20 passed)XAI_API_KEY=… uv run pytest tests/test_realtime/test_realtime.py -k xai -q --allow-uncategorized(16 passed)grok-voice-latest,session.update(reasoning / idle_timeout / grok-transcribe), cancel, force_message, deleteMade with Cursor