Audio-LLM arc: Whisper (2022) to Audio Flamingo 3 (2025)
the pipeline: spectrogram -> encoder -> Q-former -> LLM
1. waveform
16 kHz mono
2. log-Mel spec
25ms win, 10ms hop
80 Mel bins
log compress
30s = 3000 frames
3. audio encoder
Whisper: speech strong
BEATs: music strong
AF-Whisper: concat
1 frame per 10ms
12-layer transformer
frozen at bridge train
4. audio Q-former
32-64 learnable queries
cross-attend over frames
output fixed-length tokens
training
stage 1: ITM + ITC + ITG
stage 2: instruction tune
5. LLM
Qwen2.5-7B
or Llama 3.1
output
captions
QA answers
with CoT
cascaded vs end-to-end task coverage
cascaded (Whisper -> LLM)
transcription: yes
summarization: yes
emotion: no
music genre: no
environmental: no
deepfake: no
MMAU ~0.50
end-to-end audio-LLM (AF3)
every cascaded task: yes
emotion / mood: yes
music / instruments: yes
environmental sounds: yes
temporal grounding: yes
on-demand CoT: +3-5 pts
MMAU 0.72 (open SOTA 2025)