Replaying the sampled tokens lowers the gap
Mean absolute difference between the engine and trainer log probabilities, per step, for 12 steps with seed 1. Median 0.0153 with the full vocabulary and 0.0135 with top-p and top-k replay.
0.012
0.013
0.014
0.015
0.016
0
3
6
9
11
step
logprob gap, mean absolute difference
full vocabulary
top-p 0.95, top-k 64, replayed
Median gap 0.0153 to 0.0135, about 12 percent lower. One seed.