Ray reads one memory for the whole GB10
The memory Ray read when it stopped the first 4B run, as one bar of 121.69 GB. The trainer process used 36.77 GB and the engine processes 4.06 GB. The rest of the 115.86 GB in use was not in the process list. Our reading is that most of it is GPU allocations. 5.83 GB was free.
NODE MEMORY WHEN RAY STOPPED THE TRAINER, 121.7 GB
trainer process, 36.8 GB
engine, 4.1 GB
the rest of the used memory, 75.0 GB
free, 5.8 GB
kill above 95 percent
On a GB10 the GPU and the CPU share this memory.
Our reading: GPU allocations count as used memory, so a Ray kill is real.
Process sizes are the resident memory that Ray listed for its top processes.