neural audio codecs — four variations on RVQ, one semantic/acoustic split
EnCodec (Meta, 2022)
acoustic RVQ baseline
wav → conv+transformer+conv
→ RVQ (4-32 codebooks)
→ decoder → wav
75 Hz · 24 kHz · used by MusicGen
PESQ 3.2 @ 6 kbps
DAC (Descript, 2023)
reconstruction king
EnCodec +
L2-norm codebooks
periodic activations
improved losses
44.1 kHz full-band · PESQ 3.5 @ 6 kbps
SNAC (Siuzdak 2024)
multi-scale RVQ
coarse codes @ 12 Hz
medium @ 25 Hz
fine @ 50 Hz
hierarchical → AR-LM friendly
used by Orpheus-3B
Mimi (Kyutai 2024)
semantic + acoustic split
codebook 0 — WavLM distilled
(semantic)
codebooks 1-7 — acoustic
12.5 Hz · 4.4 kbps · powers Moshi
used by Sesame CSM
why this matters for language modeling over audio
10 s speech × 12.5 Hz × 8 codebooks = 1000 tokens → trivial transformer context
factorize: LM predicts semantic codebook 0 first (text-aligned), then acoustic codebooks 1-7
result: zero-shot voice cloning (semantic = content, acoustic conditioned on reference)
pick by the problem
music generation → EnCodec-24k · MusicGen, AudioLDM
highest fidelity → DAC-44.1k · audio restoration, editing
AR LM over speech → SNAC or Mimi · TTS, codec LM
full-duplex streaming → Mimi only · Moshi, Sesame CSM