Three questions pick the parallel layout
First ask whether the model and the KV cache you need fit on one GPU. If yes, add copies with data parallelism. If no, ask whether the split model fits inside one machine. If no, add pipeline parallelism across machines with tensor parallelism inside each. If yes, ask whether the GPUs have NVLink and the GPU count divides the model evenly. If yes, use tensor parallelism. If no, use pipeline parallelism inside the machine.
Fits one GPU with its cache?
data parallelism
--data-parallel-size N
Split model fits one machine?
NVLink and an even split?
pipeline + tensor
across machines
tensor parallel
--tensor-parallel-size
pipeline parallel
--pipeline-parallel-size
yes
no
yes
no
yes
no
More output from a
model that fits.