Where the start fails tells you which setting to change
A start can fail at three points. If loading the weights fails, the model is too big for the GPU, so use FP8 weights or more GPUs. If the memory check fails, other programs hold memory, so lower the memory fraction. If the KV check fails, the maximum length needs more cache than is free, so lower the maximum length.
THE LOG STOPS AT
loading weights
before the load line
memory check
free memory is less
KV check
max seq len needs
THE CAUSE
weights too big
for this GPU
memory is shared
other programs hold it
cache too small
for one full request
CHANGE THIS
FP8 weights
or --tensor-parallel-size
lower the fraction
--gpu-memory-utilization
shorter context
--max-model-len