audio-language model — three-component template (2026)
1. audio encoder
Whisper / BEATs / CLAP
→ (T, d_audio)
frozen in stage 1
2. projector
1-3 linear layers
d_audio → d_llm
trained first
3. LLM decoder
Llama / Qwen / Gemma
text + audio tokens
LoRA-fine-tuned stage 2
4. text OR speech
Qwen2.5-Omni: both
GPT-4o Audio: both
speech decoder optional
training recipe
stage 1 · freeze encoder + LLM; train projector on ASR / captioning pairs
stage 2 · LoRA on encoder + LLM; audio-QA / instruction-following data
stage 3 · (optional) bolt on a speech decoder for voice-in / voice-out
MMAU-Pro 2026 — open weights have closed the gap, except on multi-audio
Gemini 2.5 Pro ~60% overall 73.4% speech 51.9% sound 64.9% music ~22% multi
GPT-4o Audio 52.5% — — — 26.5% multi
Qwen2.5-Omni-7B 52.2% 57.4% speech 47.6% sound 61.5% music ~20% multi
Audio Flamingo 3 ~54% — (music specialist via AF-CLAP)
random chance on 4-way multiple choice = 25% — multi-audio is genuinely broken across all models