audio evaluation — one metric per task, one leaderboard per metric
ASR
primary: WER (normalized)
CER for tone langs
speed: RTFx
latency: first-token ms
Open ASR Leaderboard
Parakeet 6.05% avg WER
Whisper-v3-turbo 1.58% LS
Canary-Qwen 5.63% avg
TTS
primary: MOS / UTMOS
SECS (cloning)
round: ASR-WER on output
latency: TTFA ms
TTS Arena · Artificial Analysis
Inworld TTS-1.5 ELO 1236
Kokoro-82M ELO 1059
ElevenLabs v3 ELO 1179
speaker + diarization
primary: EER
minDCF
diariz: DER / JER
overlap: per-segment F1
VoxCeleb1-O / VoxSRC
ECAPA 0.87% EER
3D-Speaker 0.50% EER
pyannote Precision-2 < 0.40%
audio classification
exclusive: top-1, top-5
multi: mAP
imbalanced: macro F1
per-class recall
HEAR · AudioSet · ESC-50
BEATs-iter3 AudioSet 0.548
Audio-MAE Speech Cmd 99.0%
BEATs ESC-50 97.0%
music generation
primary: FAD (Fréchet Audio Distance)
CLAP (text-audio alignment)
consumer: blind-panel MOS / ELO
Suno v5 ELO 1293 (quality leader)
Udio v4 — producer tools + stems
MusicGen-large on MusicCaps ~4.5 FAD
always pair FAD with human MOS panel
open: ACE-Step XL / YuE / Stable Audio Open
audio-language + streaming
LALM reasoning: MMAU-Pro (1800 Q)
long-audio: LongAudioBench
captioning: FENSE / SPICE / CIDEr
Gemini 2.5 Pro ~60% MMAU-Pro overall
GPT-4o Audio 52.5% — Qwen2.5-Omni 52.2%
multi-audio subset: ~22% ALL models (near random)
streaming S2S: latency P50/P95 + WER
Moshi 200 ms L4 · GPT-4o Realtime ~300 ms