feat(voice): Deepgram Flux STT + OTel latency observability#80
Open
bsahajsinghani wants to merge 7 commits into
Open
feat(voice): Deepgram Flux STT + OTel latency observability#80bsahajsinghani wants to merge 7 commits into
bsahajsinghani wants to merge 7 commits into
Conversation
- Fix create_observation() payload format: wrap in {"observations": [...]}
so Memora API accepts the request (was sending flat dict → silent 400)
- Inject Memora memory into agent instructions via MemoryPromptBuilder.compose()
so the LLM actually uses observations/summaries from past calls
- Enable memory_mode="always" in voice_streaming.py example
- Wire up CI webhook: if CONVERSATION_INTELLIGENCE_CONFIGURATION_ID is set,
expose /ci-webhook route for near real-time custom operator ingestion
- Auto-close conversation via CO API on WebSocket disconnect so Memora
extraction triggers immediately instead of waiting for inactive timeout
- Add CI/memory section to getting_started/README.md explaining standard
vs alternate write paths, latency tips, and links to lifecycle docs
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
**STT model selection:**
- Add `transcription_provider` and `speech_model` fields to VoiceChannelConfig,
TwiMLOptions, and VoiceChannel so callers can pass e.g.
`{"transcription_provider": "deepgram", "speech_model": "flux"}` at construction
- Wire both fields through twiml.py → ConversationRelay attributes
(`transcriptionProvider`, `speechModel`)
- Deepgram Flux is turn-aware: waits for a complete utterance before emitting
a transcript, eliminating spurious mid-utterance STT chunks and the redundant
Recall+LLM calls they caused (~6:1 chunk:turn → 1:1 with Flux)
- Document all supported Deepgram and Google models in getting_started/README.md
with a note about the error-64101 pitfall for invalid model names
**OTel tracing:**
- Add src/tac/tracing.py: lightweight OTel setup module controlled by
OTEL_ENABLED / OTEL_ENDPOINT / OTEL_SERVICE_NAME env vars
- Provides start_call(), end_call(), turn_span(), memory_span(), llm_span(),
inject_traceparent() context managers used in channel.py
- Wire traceparent header propagation into BaseAPIClient so Recall HTTP calls
carry the active trace context (enables Memora correlation on Grafana)
- Export `tracing` from tac.__init__ so examples can call tracing.setup_tracing()
- Add LATENCY_OBSERVABILITY_PLAN.md: phased plan for full end-to-end latency
measurement (Jaeger local → missing spans → Voice Insights → VAD TTFA)
**STT chunk diagnostics (debug-only):**
- [STT] / [STT TURN N] / [STT SUMMARY] print statements in channel.py to
confirm chunk:turn ratio during benchmarking; not intended for production
Note: This is a personal benchmarking branch — no review needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fills in previously untracked timing gaps in the voice call lifecycle: - call.co_init: measures time TAC spends waiting for Conversation Orchestrator to create the conversation after ConversationRelay connects. This polling phase can take several hundred milliseconds depending on CO load and was previously invisible. - call.profile_lookup: measures the participant fetch and customer address resolution that happens immediately after conversation init. Covers the list_participants() HTTP call and customer profile ID resolution. - record_first_token(): adds a first_token_sent point-in-time event to the llm.completion span the moment the first streaming token is written to the WebSocket. This splits the LLM span into two readable phases: time waiting for the first token (TTFT) and time streaming the full response. Both co_init and profile_lookup are parented to the call root span rather than the turn span, since they only occur once at call start. Note: personal benchmarking branch — no review needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Records two actionable findings from Phase 1-2 Jaeger traces: - CO init + profile lookup accounts for ~1s at call start - memory.recall TAC-side is 4-7x Memora server-side; switching memory_mode to "once" would save ~700ms/turn from turn 2 onwards Note: personal benchmarking branch — no review needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Notes that the memory.recall overhead is not a region mismatch — TAC and Memora are co-located in us-east-1 (intra-AZ). The ~280ms gap per recall is TLS connection setup paid fresh on every call. Connection pooling + memory_mode "once" together would reduce per-turn recall from ~700ms to near zero after turn 1. Note: personal benchmarking branch — no review needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
bsahajsinghani
requested review from
ryanrishi,
ryanrouleau and
xinghaohuang91
as code owners
July 8, 2026 17:26
mdhingra1
approved these changes
Jul 8, 2026
mdhingra1
left a comment
mdhingra1
left a comment
There was a problem hiding this comment.
thanks for confirming the insights turn source
…rgets Added industry-standard latency thresholds, full experiment configuration table aligned with the intern project proposal, and additional optimization ideas to explore beyond the core experiments. Note: personal benchmarking branch — no review needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… coverage Breaks down the two previously opaque spans into individually measurable steps: memory.recall now contains: - memory.profile_lookup — lookup_profile() HTTP call - memory.profile_fetch — get_profile() HTTP call to fetch traits - memory.recall_api — the actual Memora vector search call llm.completion now contains: - llm.prompt_build — MemoryPromptBuilder + message deepcopy - llm.response_stream — WebSocket streaming loop, first to last token sent Note: personal benchmarking branch — no review needed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two additions built as part of a voice agent latency benchmarking investigation:
1. Deepgram Flux STT support
transcription_providerandspeech_modelfields toVoiceChannelConfig,TwiMLOptions, andVoiceChannelgetting_started/README.md2. OTel tracing foundation for latency benchmarking
src/tac/tracing.py: lightweight OTel setup controlled byOTEL_ENABLED/OTEL_ENDPOINT/OTEL_SERVICE_NAMEchannel.py:call(root) →call.co_init,call.profile_lookup,turn→memory.recall,llm.completion(withfirst_token_sentevent for TTFT)inject_traceparentpropagated intoBaseAPIClientHTTP headers for Memora correlationhttp://localhost:16686)Full benchmarking plan + insights from first traces:
LATENCY_OBSERVABILITY_PLAN.mdKey findings so far:
memory.recallTAC-side is 700ms–1.2s/turn vs ~165ms Memora server-side — root cause is TLS reconnection on every call, not region mismatch (both intra-AZ inus-east-1). Connection pooling inBaseAPIClient+memory_mode: "once"could bring this to near zero after turn 1.Type of Change
Checklist
SDK Parity