edge inference — four targets, bandwidth sets the ceiling
Apple Neural Engine
M4 / A18 — up to 38 TOPS
unified memory, no copy
Core ML for native ANE
MPS via llama.cpp
iPhone 16 ≈ 8 tok/s
60 GB/s DRAM
7B Q4, decode memory-bound
Qualcomm Hexagon
SD X Elite — up to 45 TOPS
SD 8 Gen 4 flagship Android
QNN SDK, AI Hub converts
Windows ARM copilots
SD 8 Gen 3 ≈ 7 tok/s
77 GB/s DRAM
7B Q4, typical mid/high
WebGPU + WebLLM
browser, no install
Chrome Android 121+ / Safari iOS 26
Firefox Android catching up
~70-75% mobile coverage
M3 Max ≈ 41 tok/s
~70-80% of native
OpenAI-compatible JS API
NVIDIA Jetson
Orin / AGX / Thor
TensorRT Edge-LLM
EAGLE-3 + NVFP4
chunked prefill
AGX ≈ 40 tok/s
gpt-oss-20b vLLM
205 GB/s
bandwidth sets the ceiling — decode reads full weights each token
H100 HBM3: 3350 GB/s → Llama 8B Q4 ceiling ≈ 710 tok/s (observed 170)
M3 Max 400 GB/s → ceiling ≈ 85 tok/s (observed 55)
iPhone 16 60 GB/s → ceiling ≈ 12 tok/s (observed 8)
compute matters only when runtime efficiency drops below this ceiling
quantization picks per target
ANE: INT4 weights + FP16 activations (Core ML path)
Hexagon: QNN INT8/INT4 (AI Hub converters)
WebGPU: Q4 GGUF (browser memory ceiling ~3-4 GB)
Jetson Thor: NVFP4 + FP8 KV (Edge-LLM path)