Multi-GPU HPC¶

The preferred production layout is one MPI rank per GPU. Each rank owns one rod-shaped spatial domain, uses its GPU for compute-intensive operations, and uses OpenMP for host-side work.

Generic MPI launch¶

For four GPUs:

export OMP_NUM_THREADS=<cpu-cores-per-gpu>
export OMP_PLACES=cores
export OMP_PROC_BIND=close

mpirun -np 4 ./gevolution -n 2 -m 2 -s settings.ini

Use the site's GPU binding mechanism so each rank sees a different GPU.

Generic Slurm script¶

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4
#SBATCH --gpus-per-node=4
#SBATCH --cpus-per-task=16
#SBATCH --time=00:30:00

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
export OMP_PLACES=cores
export OMP_PROC_BIND=close

srun --gpu-bind=single:1 --cpu-bind=cores \
  ./gevolution -n 2 -m 2 -s settings.ini

Exact GPU-binding options depend on the Slurm and MPI configuration. Verify binding with a short run before allocating production resources.

CUDA-aware MPI is recommended but optional. Use the site's documented GPU-support environment variable only after confirming that its MPI installation accepts CUDA device buffers. To establish a host-staged baseline:

export LATFIELD2_DISABLE_CUDA_AWARE_MPI=1

See CUDA-aware MPI and host staging.

CSCS Alps / Grace Hopper¶

The tracked config/alps.mk build profile selects sm_90, the CSCS library paths, CLASS, HEALPix, and the Alps-only execution targets:

make CONFIG=config/alps.mk print-config
make CONFIG=config/alps.mk -j

Alps nodes have four Grace Hopper superchips. The production execution pattern is:

export OMP_NUM_THREADS=72
export OMP_PLACES=cores

srun -N 1 -n 4 --cpus-per-task=72 \
  --hint=exclusive --cpu-bind=socket \
  ./gpu-bind.sh ./gevolution -n 2 -m 2 -s settings.ini

Account, partition, constraint, and memory options are allocation-specific.

Selecting the process grid¶

For P GPUs, choose n * m = P, with n >= 2 and m >= 2. Prefer a reasonably balanced grid unless topology measurements suggest otherwise. The mesh must satisfy:

Ngrid % n == 0
(Ngrid / m) % 2 == 0

Examples:

MPI ranks	Layout	Compatible example
4	`-n 2 -m 2`	`Ngrid = 64`
16	`-n 4 -m 4`	`Ngrid = 128`
36	`-n 6 -m 6`	`Ngrid = 144`