MIO — four modalities, one shared vocabulary, streaming decode
four tokenizers, one vocab, one transformer
text
BPE tokenizer
ids 0..31999
image
SEED-Tokenizer
ids 32000..36095
speech
SpeechTokenizer RVQ
8 codebook layers
music
Encodec-class
8192 entries
one transformer, one NTP loss
~48k vocabulary, text + image + speech + music
streaming friendly: token in, token out
four-stage curriculum + streaming decode path
Stage 1 alignment
text-image pairs
text-speech pairs
text-music pairs
Stage 2 interleaved
multi-modality docs
cross-modal context
OBELICS + podcast
Stage 3 speech-rich
extra audio data
speech quality lift
without text regression
Stage 4 SFT
VQA, narration
speech dialogue
any-to-any tasks
streaming decode path (target <500 ms TTFAB)
mic -> speech tokens (~40 ms)
prefill prompt (~80 ms at 8B)
first output token (~40 ms)
residual-VQ layers 1..7 parallel decode (~30 ms)
speech waveform decoder (~80 ms)
total TTFAB: ~270 ms (GPT-4o-class)
Moshi 160 ms, MIO 400-500 ms in published traces