Workstation with one GPU¶

gevolution requires a minimum 2 x 2 process grid, hence at least four MPI ranks. A workstation with one NVIDIA GPU can run all four ranks on that GPU using NVIDIA Multi-Process Service (MPS).

Warning

Four ranks share the GPU memory and execution resources. Start with a small Ngrid, limit output, and monitor memory use. This layout is intended for development and small simulations, not peak performance.

Start MPS¶

Run the MPS control daemon once for the user session:

export CUDA_VISIBLE_DEVICES=0
export CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps-${USER}
export CUDA_MPS_LOG_DIRECTORY=/tmp/nvidia-log-${USER}
mkdir -p "$CUDA_MPS_PIPE_DIRECTORY" "$CUDA_MPS_LOG_DIRECTORY"
nvidia-cuda-mps-control -d

All MPI ranks and the control command must use the same MPS pipe directory.

MPS does not make MPI CUDA-aware. A typical workstation MPI installation uses the host-staged communication fallback. To force that portable path while testing:

export LATFIELD2_DISABLE_CUDA_AWARE_MPI=1

See CUDA-aware MPI and host staging for the runtime selection rules. On systems where MPS provisioning is managed by an administrator, follow the local policy instead.

Launch four ranks¶

export CUDA_VISIBLE_DEVICES=0
export CUDA_MPS_PIPE_DIRECTORY=/tmp/nvidia-mps-${USER}
export CUDA_MPS_LOG_DIRECTORY=/tmp/nvidia-log-${USER}
export OMP_NUM_THREADS=1

mpirun -np 4 \
  -x CUDA_VISIBLE_DEVICES \
  -x CUDA_MPS_PIPE_DIRECTORY \
  -x CUDA_MPS_LOG_DIRECTORY \
  -x OMP_NUM_THREADS \
  ./gevolution -n 2 -m 2 -s settings.ini

MPI environment-export syntax varies. For example, some launchers use -env, while Slurm normally propagates exported variables automatically.

Stop MPS¶

After all ranks have exited:

echo quit | nvidia-cuda-mps-control

Troubleshooting¶

Confirm nvidia-cuda-mps-control -d starts without errors.
Confirm every rank sees the same intended GPU.
Reduce Ngrid, particle tiling, and output if allocation fails.
Do not start one MPS daemon per rank.
Avoid sharing the GPU with unrelated workloads during validation runs.