voice assistant pipeline — 7 components, 1 turn
1. mic
16 kHz mono
20 ms chunks
2. VAD
Silero
+ pre-roll
3. STT
Whisper / Parakeet
streaming
4. LLM + tools
GPT-4o / Claude
stream + JSON tools
5. TTS
Kokoro / Cartesia
stream from token 20
6. speaker
opus out
PLC enabled
7. user
ear
barge-in: if user speaks, VAD cancels TTS + LLM, restart from (2)
the three failure modes you will hit
1. first-word clip — VAD triggers late; user's "hey" is lost. Keep 200-400 ms pre-roll buffer.
2. barge-in confusion — LLM keeps generating after user interrupts. Wire VAD → cancel-LLM.
3. silence hallucinations — Whisper says "Thanks for watching" on silent warm-up. Always VAD-gate.
in production, log every turn + P50/P95/P99 per-stage histograms before shipping.
2026 reference stacks (pick by target latency + license)
LiveKit + Deepgram + GPT-4o + Cartesia 350-500 ms industry default
Pipecat + Whisper-stream + GPT-4o + Kokoro 500-800 ms DIY open-source
Moshi (full-duplex, single model) 200-300 ms different architecture — lesson 15
Vapi / Retell (managed) 300-500 ms fastest to ship
whisper.cpp + llama.cpp + Kokoro-ONNX offline edge / privacy
add wake-word gate (Porcupine) for always-on consumer devices; never log raw audio > 30 days.