Running simulations¶

Command-line interface¶

./gevolution -n <grid-dim-0> -m <grid-dim-1> -s <settings.ini> [options]

Option	Meaning
`-n`	First dimension of the LATfield2 process grid
`-m`	Second dimension of the LATfield2 process grid
`-s`	Simulation settings file
`-p`	Optional CLASS precision file; requires `HAVE_CLASS`
`-i`	Reserved for the currently unsupported external I/O server
`-g`	Reserved for the currently unsupported external I/O server

The number of compute ranks must be exactly n * m.

The LATfield2 external I/O server has not been ported to the GPU backend. Compiling with EXTERNAL_IO is deliberately blocked. The -i and -g branches remain in the source for a possible future implementation, but they cannot be used by the current executable.

No self-neighbours

n >= 2 and m >= 2 are mandatory. Communications inherited from LATfield2 assume that neighbouring ranks cannot be the sending rank.

Process layout and `Ngrid`¶

The simulation volume is cut along two dimensions. The remaining \(x\) direction is local and contiguous. For the distributed real-to-complex FFT:

Ngrid / n must be an integer
Ngrid / m must be an even integer

Choose a balanced process grid when possible. Before a large allocation, validate the layout with a small run using the same n, m, and representative compiler configuration.

CPU and GPU binding¶

The preferred production layout is:

one MPI rank per GPU;
all CPU cores near that GPU assigned to the rank;
OpenMP threads pinned to those CPU cores;
CUDA-aware MPI enabled when the system MPI stack has validated device-buffer support.

For a one-GPU workstation, use four ranks through NVIDIA MPS.

CUDA-aware MPI and host staging¶

CUDA-aware MPI is optional. Direct device-pointer communication is activated only when a recognised environment hint is enabled and LATFIELD2_DISABLE_CUDA_AWARE_MPI is not enabled. Otherwise communication that originates from device memory is staged through host memory.

The host-staged fallback covers LATfield2 FFT, halo, and particle exchanges as well as gevolution particle light-cone ID routing, balanced particle output, and HEALPix light-cone communication. It is the safe default for workstation MPI installations without CUDA support.

# Force host staging
export LATFIELD2_DISABLE_CUDA_AWARE_MPI=1
mpirun -np 4 ./gevolution -n 2 -m 2 -s settings.ini

On a validated CUDA-aware MPI installation, use the site-provided support variable or explicitly opt in:

export LATFIELD2_ENABLE_CUDA_AWARE_MPI=1
mpirun -np 4 ./gevolution -n 2 -m 2 -s settings.ini

The recognised variables and compiler configuration are listed under CUDA-aware MPI selection. NVIDIA MPS and CUDA-aware MPI solve different problems: MPS improves concurrent use of one GPU by several ranks, while CUDA-aware MPI controls whether MPI may access device buffers directly.

OpenMP¶

Typical environment:

export OMP_NUM_THREADS=<cores-assigned-to-rank>
export OMP_PLACES=cores
export OMP_PROC_BIND=close

Host-side Fourier loops and residual CPU work use OpenMP. More threads are not automatically better if they cross NUMA boundaries or oversubscribe CPUs.

Restarting and hibernation¶

Unsupported in the GPU executable

The settings parser still accepts hibernation parameters, and source for the older workflow remains in the repository. In main.cu, however, calls that write hibernation checkpoints are disabled. Do not rely on hibernation redshifts or hibernation wallclock limit for GPU runs. Reaching the wallclock limit currently stops the simulation without writing the announced checkpoint.

Reading supported initial data from disk remains available. It should not be described as an exact GPU checkpoint/restart workflow.

Logging and benchmarks¶

With -DBENCHMARK, gevolution prints:

initialisation and total runtime;
main-loop time;
projections;
snapshot, light-cone, and spectrum output;
particle kick and drift;
gravity solver and FFT time.

Record the executable revision, compiler definitions, process layout, GPU model, MPI stack, OpenMP settings, and output schedule with every benchmark.