perf(voice): add latency instrumentation and parallelize memory fetch - #96
Draft
xinghaohuang91 wants to merge 1 commit into
Draft
perf(voice): add latency instrumentation and parallelize memory fetch#96xinghaohuang91 wants to merge 1 commit into
xinghaohuang91 wants to merge 1 commit into
Conversation
Adds INFO-level timing logs across the Voice channel's per-turn pipeline (setup-to-first-prompt, CO poll/init, memory retrieval, LLM callback, time-to-first-token, streaming completion) to make voice latency testable end-to-end against live calls. Also parallelizes TAC.retrieve_memory()'s get_profile() and retrieve_memory() calls via asyncio.gather, since the memory recall call only needs the already-known profile_id and doesn't depend on the profile fetch result. Includes a written analysis of latency test calls (cross-checked against Twilio ConversationRelay/Voice Insights event timelines) documenting findings and remaining optimization ideas. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
INFO-level latency logging across the Voice channel's per-turn pipeline (setup→first-prompt, CO poll/init, memory retrieval, LLM callback, time-to-first-token, streaming completion) so voice latency can be measured end-to-end against live calls.TAC.retrieve_memory()'sget_profile()andretrieve_memory()calls viaasyncio.gather, since the memory recall call only needs the already-knownprofile_id, not the profile fetch result.VOICE_LATENCY_ANALYSIS.mddocumenting live-call latency test results (cross-checked against Twilio ConversationRelay/Voice Insights event timelines), root causes identified, and remaining optimization ideas (notably:BaseAPIClientcreates a newhttpx.AsyncClientper request with no connection reuse — likely the next highest-impact fix, not yet implemented).Draft — opened to snapshot in-progress latency testing work; not ready for review yet, more optimization work planned as a follow-up.
Type of Change
Checklist
test_tac.py/test_voice_channel.pysuites pass unchanged)VOICE_LATENCY_ANALYSIS.md)voice_streaming.py+ Twilio Voice Insights)SDK Parity
🤖 Generated with Claude Code