Music Generation and Audio-Language Models · AI Engineering

Codec tokens, MusicGen, audio encoders and projectors, Thinker and Talker, TTFAB budgets and audio pitfalls.

Audio becomes codec tokens

Nodes and connections

Node	Role	Kind
Waveform	16-48 kHz	client
Encoder conv	per-timestep vector	process
RVQ	8 codebooks at 75 Hz	store
Generator	token-AR or diffusion	model
Codec decoder	tokens to audio	process
Audio out

From	To	Connection
Waveform	Encoder conv
Encoder conv	RVQ	vectors
RVQ	Generator	indices
Generator	Codec decoder	new tokens
Codec decoder	Audio out

Sources

phases/08-generative-ai/11-audio-generation/docs/en.md

Numbers that matter

Codec rate: 8 RVQ codebooks at 75 Hz give 600 tokens/sec.
Stream floor: at 75 tokens/second, generate ≥75 tokens/sec per user.
Talker floor: 50 Hz speech tokens need ≥50 tok/s from the Talker.
Codebook: 1024 or 2048 entries is typical.
MusicGen codec: EnCodec at 32 kHz with 4 codebooks.
Log-Mel shape: 30 s at 100 Hz frame rate gives (3000, 80).
Round trip: below 500ms, the user stops noticing the lag.

Sources

Two generator families

Token-autoregressive: a decoder-only transformer over flattened RVQ tokens.
Delayed parallel: MusicGen emits K codebook streams with per-stream offsets.
VALL-E: speech tokens from a text prompt and a 3-second voice sample.
Latent diffusion: Stable Audio Open works on compressed audio. Good for loops.
Flow matching: Stable Audio 2.5 uses it on continuous audio latents.
Trend: flow matching wins for music. Token-AR dominates speech.

Sources

MusicGen, melody and FAD

from audiocraft.models import MusicGen

model = MusicGen.get_pretrained("facebook/musicgen-small")300M, fast

model.set_generation_params(duration=10)seconds of audio

wav = model.generate(["upbeat synthwave with driving drums, 128 BPM"])

torchaudio.save("out.wav", wav[0].cpu(), 32000)32 kHz output

melody, sr = torchaudio.load("humming.wav")

wav = model.generate_with_chroma(keeps the tune, swaps timbre

["jazz piano cover"], melody.squeeze(), sr)

from frechet_audio_distance import FrechetAudioDistance

fad = FrechetAudioDistance()VGGish embedding distance

fad.get_fad_score("generated_folder/", "reference_folder/")lower is better

Sources

phases/06-speech-and-audio/09-music-generation/docs/en.md

Audio-language model template

Audio encoder Whisper encoder, BEATs, CLAP or WavLM makes features.
Projector a linear layer or MLP maps features into the LLM embedding space.
LLM a decoder reads interleaved text and audio tokens and writes text.
Stage 1 freeze encoder and LLM. Train the projector on ASR or captioning data.
Stage 2 full or LoRA fine-tune on instruction audio tasks.
Stage 3 optional. Add a speech decoder for voice-in and voice-out.

Sources

phases/06-speech-and-audio/10-audio-language-models/docs/en.md

Projector and audio Q-former

class AudioProjector(nn.Module):

def __init__(self, audio_dim=1280, llm_dim=4096):

super().__init__()

self.down = nn.Linear(audio_dim, llm_dim)audio to LLM width

self.act = nn.GELU()

self.up = nn.Linear(llm_dim, llm_dim)

def forward(self, audio_features):

return self.up(self.act(self.down(audio_features)))

qf = QFormer(n_queries=8, hidden=20)N=8 queries

tokens = qf.forward(frames)99 frames in, 8 tokens out

Sources

Log-Mel and encoder settings

Setting	Default	Note
Sample rate	16 kHz	Resample first
Window	25ms	`win_ms: int = 25`
Hop	10ms	`hop_ms: int = 10`
Mel filters	80	0-8000 Hz
Log compress	`math.log(1 + x)`	Dynamic range
Q-former queries	32 or 64	AF3 uses 64
Whisper encoder	12-layer	Frozen in the ALM pattern

Sources

Thinker and Talker

Nodes and connections

Node	Role	Kind
Mic and VAD	speech tokens	client
Thinker	7B-80B, TMRoPE	model
Talker	200M-1B	model
Speech decoder	SNAC, MoVQGAN	process
Speakers	audio samples	client

From	To	Connection
Mic and VAD	Thinker	audio tokens
Thinker	Talker	text tokens
Talker	Speech decoder	RVQ indices
Speech decoder	Speakers	stream

Sources

phases/12-multimodal-ai/20-omni-models-thinker-talker/docs/en.md

TTFAB budget at 7B

Setting	Default	Note
Mic to audio tokens	40-80ms	Streams
Prefill	100-200ms	Much more at 70B
First Thinker text token	40ms	Grows with Thinker size
Talker on first text token	20ms	At least 15ms in the lesson model
First speech tokens commit	40ms	Before residual-VQ decode
Residual-VQ decode	30ms	`rvq_decode = 30`
Speech waveform decode	50-80ms	`wave_decode = 70`
Total TTFAB	320-490ms	600-900ms at 70B

Sources

Troubleshooting

Symptom	Cause	Fix
Music loops past 30 s	AR models loop	Crossfade, or use ACE-Step
Tempo leaves the BPM	Models wander	BPM tags, librosa `beat_track`
Mushy sung words	Open models are weak on lyrics	Commercial API or fine-tune
Hallucinations on silence	Whisper encoder issue	VAD-gate
Speakers mixed past 10 minutes	Long-audio degradation	Diarize first, then summarize
Multi-audio near 25%	Random-chance level	Do not trust "which clip" tasks
Errors from RVQ layer 1 spread	RVQ error accumulation	Temperature 0 on higher layers
Artifacts at clip joins	Crossfade boundaries	Careful overlap-add

Sources

Tasks in code

Task	Code
Load the ALM processor	`AutoProcessor.from_pretrained("Qwen/Qwen2.5-Omni-7B")`
Generate an answer	`model.generate(**inputs, max_new_tokens=200)`
Load MMAU-Pro	`load_dataset("gamma-lab-umd/MMAU-Pro", split="test")`
Score MMAU-Pro officially	`evaluate_mmau_pro_comprehensive.py`
Cut audio into frames	`window_frames(wave, 16000, 25, 10)`
List TTFAB parts	`ttfab(cfg)`
Sample codec tokens	`generate(counts, style, start, length, rng, temperature=1.0)`

Sources

Q&A

When is a cascade enough?: Transcription is the goal, with no music and no emotion.
When do I need end-to-end?: Music, emotion, multi-speaker or complex audio reasoning.
What does a cascade lose?: Mood, speaker identity, timing and deepfake cues.
Why a small Talker?: A 7B Talker falls behind the 50 tok/s floor.
Why TMRoPE?: It gives each token an absolute timestamp. Vision, audio and text align.
Half or full duplex?: Qwen2.5-Omni is half-duplex by default. Moshi supports full-duplex.

Sources

Tips

To ship safely, generate instrumental only.
Tag each generation with watermarks and metadata.
Report each MMAU-Pro category separately.
Check the codec first. It caps output quality. DAC is the current open best.
For voice cloning, add abuse detection and opt-out lists.

Sources

Original source

---
id: audio-generation-alms
title: Music Generation and Audio-Language Models
code: MG
subtitle: Codec tokens, MusicGen, audio encoders and projectors, Thinker and Talker, TTFAB budgets and audio pitfalls.
hue: 285
sources: phases/06-speech-and-audio/09-music-generation, phases/06-speech-and-audio/10-audio-language-models, phases/08-generative-ai/11-audio-generation, phases/12-multimodal-ai/19-audio-language-whisper-to-af3, phases/12-multimodal-ai/20-omni-models-thinker-talker
source_hash: 453110680f89bafa
---

## Audio becomes codec tokens
type: architecture
from: phases/08-generative-ai/11-audio-generation/docs/en.md

```arch
node | wave | 1,1 | Waveform | 16-48 kHz | client
node | enc | 2,1 | Encoder conv | per-timestep vector | process
node | rvq | 3,1 | RVQ | 8 codebooks at 75 Hz | store
node | gen | 3,2 | Generator | token-AR or diffusion | model
node | dec | 2,2 | Codec decoder | tokens to audio | process
node | out | 1,2 | Audio out | |
edge | wave | enc |
edge | enc | rvq | vectors
edge | rvq | gen | indices
edge | gen | dec | new tokens
edge | dec | out |
```

## Numbers that matter
type: concepts
from: phases/08-generative-ai/11-audio-generation/docs/en.md, phases/06-speech-and-audio/09-music-generation/docs/en.md, phases/12-multimodal-ai/19-audio-language-whisper-to-af3/docs/en.md, phases/12-multimodal-ai/20-omni-models-thinker-talker/docs/en.md

- **Codec rate**: 8 RVQ codebooks at 75 Hz give 600 tokens/sec.
- **Stream floor**: at 75 tokens/second, generate ≥75 tokens/sec per user.
- **Talker floor**: 50 Hz speech tokens need ≥50 tok/s from the Talker.
- **Codebook**: 1024 or 2048 entries is typical.
- **MusicGen codec**: EnCodec at 32 kHz with 4 codebooks.
- **Log-Mel shape**: 30 s at 100 Hz frame rate gives (3000, 80).
- **Round trip**: below 500ms, the user stops noticing the lag.

## Two generator families
type: concepts
from: phases/08-generative-ai/11-audio-generation/docs/en.md, phases/06-speech-and-audio/09-music-generation/docs/en.md

- **Token-autoregressive**: a decoder-only transformer over flattened RVQ tokens.
- **Delayed parallel**: MusicGen emits K codebook streams with per-stream offsets.
- **VALL-E**: speech tokens from a text prompt and a 3-second voice sample.
- **Latent diffusion**: Stable Audio Open works on compressed audio. Good for loops.
- **Flow matching**: Stable Audio 2.5 uses it on continuous audio latents.
- **Trend**: flow matching wins for music. Token-AR dominates speech.

## MusicGen, melody and FAD
type: annotated-code
lang: python
from: phases/06-speech-and-audio/09-music-generation/docs/en.md

```python
from audiocraft.models import MusicGen
model = MusicGen.get_pretrained("facebook/musicgen-small")  # 300M, fast
model.set_generation_params(duration=10)  # seconds of audio
wav = model.generate(["upbeat synthwave with driving drums, 128 BPM"])
torchaudio.save("out.wav", wav[0].cpu(), 32000)  # 32 kHz output
melody, sr = torchaudio.load("humming.wav")
wav = model.generate_with_chroma(  # keeps the tune, swaps timbre
    ["jazz piano cover"], melody.squeeze(), sr)
from frechet_audio_distance import FrechetAudioDistance
fad = FrechetAudioDistance()  # VGGish embedding distance
fad.get_fad_score("generated_folder/", "reference_folder/")  # lower is better
```

## Audio-language model template
type: flow
from: phases/06-speech-and-audio/10-audio-language-models/docs/en.md

1. **Audio encoder**: Whisper encoder, BEATs, CLAP or WavLM makes features.
2. **Projector**: a linear layer or MLP maps features into the LLM embedding space.
3. **LLM**: a decoder reads interleaved text and audio tokens and writes text.
4. **Stage 1**: freeze encoder and LLM. Train the projector on ASR or captioning data.
5. **Stage 2**: full or LoRA fine-tune on instruction audio tasks.
6. **Stage 3**: optional. Add a speech decoder for voice-in and voice-out.

## Projector and audio Q-former
type: annotated-code
lang: python
from: phases/06-speech-and-audio/10-audio-language-models/docs/en.md, phases/12-multimodal-ai/19-audio-language-whisper-to-af3/code/main.py

```python
class AudioProjector(nn.Module):
    def __init__(self, audio_dim=1280, llm_dim=4096):
        super().__init__()
        self.down = nn.Linear(audio_dim, llm_dim)  # audio to LLM width
        self.act = nn.GELU()
        self.up = nn.Linear(llm_dim, llm_dim)
    def forward(self, audio_features):
        return self.up(self.act(self.down(audio_features)))
qf = QFormer(n_queries=8, hidden=20)  # N=8 queries
tokens = qf.forward(frames)  # 99 frames in, 8 tokens out
```

## Log-Mel and encoder settings
type: defaults
from: phases/12-multimodal-ai/19-audio-language-whisper-to-af3/docs/en.md, phases/12-multimodal-ai/19-audio-language-whisper-to-af3/code/main.py

| Setting | Default | Note |
|---|---|---|
| Sample rate | 16 kHz | Resample first |
| Window | 25ms | `win_ms: int = 25` |
| Hop | 10ms | `hop_ms: int = 10` |
| Mel filters | 80 | 0-8000 Hz |
| Log compress | `math.log(1 + x)` | Dynamic range |
| Q-former queries | 32 or 64 | AF3 uses 64 |
| Whisper encoder | 12-layer | Frozen in the ALM pattern |

## Thinker and Talker
type: architecture
from: phases/12-multimodal-ai/20-omni-models-thinker-talker/docs/en.md

```arch
node | mic | 1,1 | Mic and VAD | speech tokens | client
node | think | 2,1 | Thinker | 7B-80B, TMRoPE | model
node | talk | 3,1 | Talker | 200M-1B | model
node | dec | 3,2 | Speech decoder | SNAC, MoVQGAN | process
node | spk | 2,2 | Speakers | audio samples | client
edge | mic | think | audio tokens
edge | think | talk | text tokens
edge | talk | dec | RVQ indices
edge | dec | spk | stream
```

## TTFAB budget at 7B
type: defaults
from: phases/12-multimodal-ai/20-omni-models-thinker-talker/docs/en.md, phases/12-multimodal-ai/20-omni-models-thinker-talker/code/main.py

| Setting | Default | Note |
|---|---|---|
| Mic to audio tokens | 40-80ms | Streams |
| Prefill | 100-200ms | Much more at 70B |
| First Thinker text token | 40ms | Grows with Thinker size |
| Talker on first text token | 20ms | At least 15ms in the lesson model |
| First speech tokens commit | 40ms | Before residual-VQ decode |
| Residual-VQ decode | 30ms | `rvq_decode = 30` |
| Speech waveform decode | 50-80ms | `wave_decode = 70` |
| Total TTFAB | 320-490ms | 600-900ms at 70B |

## Troubleshooting
type: troubleshooting
from: phases/06-speech-and-audio/09-music-generation/docs/en.md, phases/06-speech-and-audio/10-audio-language-models/docs/en.md, phases/08-generative-ai/11-audio-generation/docs/en.md

| Symptom | Cause | Fix |
|---|---|---|
| Music loops past 30 s | AR models loop | Crossfade, or use ACE-Step |
| Tempo leaves the BPM | Models wander | BPM tags, librosa `beat_track` |
| Mushy sung words | Open models are weak on lyrics | Commercial API or fine-tune |
| Hallucinations on silence | Whisper encoder issue | VAD-gate |
| Speakers mixed past 10 minutes | Long-audio degradation | Diarize first, then summarize |
| Multi-audio near 25% | Random-chance level | Do not trust "which clip" tasks |
| Errors from RVQ layer 1 spread | RVQ error accumulation | Temperature 0 on higher layers |
| Artifacts at clip joins | Crossfade boundaries | Careful overlap-add |

## Tasks in code
type: task-table
span: 2
from: phases/06-speech-and-audio/10-audio-language-models/docs/en.md, phases/06-speech-and-audio/10-audio-language-models/code/main.py, phases/12-multimodal-ai/19-audio-language-whisper-to-af3/code/main.py, phases/12-multimodal-ai/20-omni-models-thinker-talker/code/main.py, phases/08-generative-ai/11-audio-generation/code/main.py, phases/08-generative-ai/11-audio-generation/docs/en.md

| Task | Code |
|---|---|
| Load the ALM processor | `AutoProcessor.from_pretrained("Qwen/Qwen2.5-Omni-7B")` |
| Generate an answer | `model.generate(**inputs, max_new_tokens=200)` |
| Load MMAU-Pro | `load_dataset("gamma-lab-umd/MMAU-Pro", split="test")` |
| Score MMAU-Pro officially | `evaluate_mmau_pro_comprehensive.py` |
| Cut audio into frames | `window_frames(wave, 16000, 25, 10)` |
| List TTFAB parts | `ttfab(cfg)` |
| Sample codec tokens | `generate(counts, style, start, length, rng, temperature=1.0)` |

## Q&A
type: qa
from: phases/12-multimodal-ai/19-audio-language-whisper-to-af3/docs/en.md, phases/12-multimodal-ai/20-omni-models-thinker-talker/docs/en.md

- **When is a cascade enough?** Transcription is the goal, with no music and no emotion.
- **When do I need end-to-end?** Music, emotion, multi-speaker or complex audio reasoning.
- **What does a cascade lose?** Mood, speaker identity, timing and deepfake cues.
- **Why a small Talker?** A 7B Talker falls behind the 50 tok/s floor.
- **Why TMRoPE?** It gives each token an absolute timestamp. Vision, audio and text align.
- **Half or full duplex?** Qwen2.5-Omni is half-duplex by default. Moshi supports full-duplex.

## Tips
type: tips
from: phases/06-speech-and-audio/09-music-generation/docs/en.md, phases/06-speech-and-audio/10-audio-language-models/docs/en.md, phases/08-generative-ai/11-audio-generation/docs/en.md

- To ship safely, generate instrumental only.
- Tag each generation with watermarks and metadata.
- Report each MMAU-Pro `category` separately.
- Check the codec first. It caps output quality. DAC is the current open best.
- For voice cloning, add abuse detection and opt-out lists.