Hogwild! inference — N workers, one shared KV cache
worker 1
reads full shared cache
decides what to write next
writes one token
repeat
worker 2
reads full shared cache
sees worker 1's tokens
pivots to unexplored work
writes one token
shared KV cache
A
A
B
A
B
B
A
C
A
C
tokens arrive asynchronously from all workers
RoPE positions let the cache stay stable under concurrent writes
each worker's next read sees everything
(including the other workers' just-written tokens)
speedup(N, p, c) = 1 / [(1 - p) + p/N + c * N / T_serial]
p = parallelizable fraction · c = per-worker coord cost · T_serial = single-worker task length