serial depth: the only thing a GPU actually cares about
RNN
depth = N
h_1
h_2
h_3
h_4
h_5
h_6
h_7
each box must wait for the one to its left — 7 steps of wall-clock
Transformer
depth = 1
x_1
x_2
x_3
x_4
x_5
x_6
x_7
softmax(QK^T / √d) V
every x_i contributes to every output in a single matmul — 1 step of wall-clock
same op count. different dependency graph. that is the entire thesis of Attention Is All You Need.