audio gen: codec tokens + transformer or diffusion
waveform
24 kHz, 1-D
codec encoder
Encodec / DAC / SoundStream
RVQ tokens
K × 75 Hz indices
8 codebooks typical
codec decoder
tokens → wav
token-AR path (MusicGen, VALL-E)
decoder-only transformer
p(t_n | t_<n, text_prompt, voice_prompt)
streams naturally (~200 ms TTFB)
delayed-parallel: K offset streams
dominates speech in 2026
diffusion / flow path (Stable Audio, AudioLDM)
DiT on audio latents
x_t → x_0 via flow matching
faster total time for long clips
cleaner for music at >=30 s
dominates music generation in 2026
2026 production stack
TTS: ElevenLabs V3, OpenAI TTS, GPT-4o realtime, NaturalSpeech 3
Music: Suno v4, Udio, Stable Audio 2.5, MusicGen 3.3B
SFX: AudioCraft 2, ElevenLabs SFX, Stable Audio Open
Voice clone: XTTS v2 (open), ElevenLabs Pro (consent-verified)