1.7B fits for short steps, and 4B does not
Blue bars: the trainer state of full fine-tuning with Adam, estimated at 16 bytes per parameter, before activations and the engine. Rose bars: the node memory that Ray read when it stopped the two 4B runs, 115.86 GB with the engine memory fraction at 0.4 and 119.18 GB at 0.2, out of 121.69 GB. Ray stops a worker above 95 percent of the node, about 115.6 GB.
0
30
60
90
120
GB
node, 121.7 GB
Ray stops a worker above 95 percent
about 10 GB
0.6B
about 27 GB
1.7B
about 64 GB
4B
4B, run 1
4B, run 2
115.9 GB
119.2 GB
trainer state, estimate
node memory at the Ray kill
Lowering the engine memory did not save 4B. All runs used 256-token answers.