Voice pipeline — Pipecat frame-based, LiveKit WebRTC transport
DOWNSTREAM audio in, speech out
VAD (Silero)
20-60ms
STT
100-250ms partial
LLM
150-400ms first token
TTS
100-200ms first audio
transport
30-80ms RTT
user
context alternates user/assistant; every stage can carry partial output
transports: Daily, LiveKit, SmallWebRTCTransport, FastAPI WebSocket, WhatsApp, SIP
UPSTREAM cancel, metrics, barge-in
transport detects interrupt cancel frame propagates back: TTS LLM STT
TTS drops pending audio; LLM cancels inference; STT resets for new turn
without UPSTREAM: agent keeps talking over the user
end-to-end latency budget
premium stack: 450-600ms (Vapi, Retell reported)
common stack: 800-1200ms
> 1500ms feels broken. sum your chain before shipping.
LiveKit Agents: MultimodalAgent (direct audio) vs VoicePipelineAgent (cascade with text control).