Jamba — 1 Transformer layer per 7 Mamba layers, MoE on alternates
one Jamba block (8 layers)
Mamba (SSM)
+ MoE router
Mamba (SSM)
dense MLP
Mamba (SSM)
+ MoE router
Mamba (SSM)
dense MLP
Mamba (SSM)
+ MoE router
Mamba (SSM)
dense MLP
Mamba (SSM)
+ MoE router
Transformer (attention)
dense MLP
repeat this block N times
memory at 256k context (BF16)
pure Transformer MHA 32L
128 GB
pure Transformer GQA(8) 32L
32 GB
Jamba 1:3 hybrid 32L
32 GB
Jamba 1:7 hybrid 32L
16 GB
pure Mamba 32L
<1 MB
why the 1:7 ratio
1:1 (pure Transformer)
best quality, worst memory
1:3 (25% attention)
still KV-heavy, small quality win
1:7 (12.5% attention) ✓
AI21 sweet spot — 256k on one 80GB GPU
1:15 (6% attention)
in-context recall starts failing
0:N (pure Mamba)
state-tracking fails without attention
Mamba-3 (ICLR 2026)
complex state + MIMO; pushes ratio lower?
frontier hybrids in 2026
Jamba 1.5 Large: 398B/94B active, enterprise
next-gen: likely Mamba-3 + attention at 1:10+