Installation¶

gevolution 2.0 is written in CUDA C++ and targets NVIDIA GPUs. It does not support serial execution.

Required software¶

Component	Purpose
CUDA toolkit and `nvcc`	GPU compilation and runtime
MPI with C++ compiler wrapper	Distributed-memory execution
GPU-enabled LATfield2	Domain decomposition, fields, particles, FFTs, and communication
FFTW3	Host-side FFT stages and alignment fallback
GSL and CBLAS	Numerical utilities
HDF5	Field snapshots and initial data

For production, use parallel HDF5. A CUDA-aware MPI implementation is recommended for performance but is not required for correctness: when CUDA-aware MPI is not explicitly enabled, LATfield2 and gevolution stage device communication through host memory.

Optional software¶

Component	Enables
CLASS built as a library	On-the-fly transfer functions and linear radiation/fluid treatment
HEALPix C library	Pixelised metric light-cone output
HEALPix C++ and CFITSIO	Building the `lcmap` post-processing tool
NVIDIA Nsight Systems / Compute	GPU profiling

Build GPU-enabled LATfield2¶

Clone the GPU-ready LATfield2 version adjacent to gevolution or update the gevolution include path to its location. Configure LATfield2 consistently with the desired precision, FFT, HDF5, CUDA, and MPI stack.

git clone https://github.com/gevolution-code/LATfield2.git
git clone https://github.com/gevolution-code/gevolution-2.0.git
cd gevolution-2.0

LATfield2 is a header-oriented dependency, but its configuration and linked libraries must match gevolution. In particular, use compatible MPI and HDF5 builds.

Configure gevolution¶

The tracked makefile contains portable build rules. Machine-specific choices belong in a configuration file:

cp config/local.mk.example config/local.mk
# Edit config/local.mk for the target system.
make print-config
make -j

config/local.mk is ignored by Git. The build first loads generic defaults, then the selected configuration, while command-line assignments have highest precedence:

make CONFIG=config/alps.mk
make CONFIG=config/local.mk.example
make CUDA_ARCH=sm_86 ENABLE_HEALPIX=1

GPU targets require an explicit CUDA_ARCH. Automatic detection is avoided because many HPC login and build nodes cannot see the execution GPUs.

Variable	Purpose
`CUDA_ARCH`	CUDA code-generation target, such as `sm_86` or `sm_90`
`LATFIELD2_DIR`	Path containing `LATfield2.hpp`
`PRECISION`	`single` or `double`; selects the matching FFTW library
`PARALLEL_HDF5`	Define parallel-HDF5 support when set to `1`
`ENABLE_CLASS`	Define `HAVE_CLASS` and link CLASS when set to `1`
`ENABLE_HEALPIX`	Define `HAVE_HEALPIX` and link HEALPix C when set to `1`
`USE_PKG_CONFIG`	Discover common dependency flags through `pkg-config`
`EXTRA_DEFINES`	Additional definitions without the `-D` prefix

Standard Make variables CPPFLAGS, NVCCFLAGS, CXXFLAGS, LDFLAGS, and LDLIBS can add site-specific flags. NVCC, CXX, and MPICXX select the compilers.

The workstation example uses pkg-config for parallel HDF5, GSL, FFTW, and HEALPix. HPC profiles can set USE_PKG_CONFIG=0 and provide explicit include and library paths instead. make check-config validates the CUDA architecture, LATfield2 path, and requested pkg-config packages. Override PARALLEL_HDF5_PKG if the local parallel-HDF5 package uses a name other than hdf5-openmpi.

There is no nvcc switch that makes MPI CUDA-aware. CUDA device-buffer support is a property of the selected MPI library and is enabled for gevolution at runtime as described below.

Compiler definitions¶

Recommended or commonly used definitions:

Definition	Behaviour
`-DFFT3D`	Required LATfield2 distributed FFT support
`-DHDF5`	Required LATfield2 HDF5 support
`-DH5_HAVE_PARALLEL`	Parallel HDF5 output
`-DSINGLE`	Single-precision fields; link `fftw3f`
`-DPHINONLINEAR`	Include shortwave corrections; recommended for standard use
`-DBENCHMARK`	Print timing breakdowns
`-DEXACT_OUTPUT_REDSHIFTS`	Interpolate supported output to requested redshifts
`-DGRADIENT_ORDER=2`	Centred second-order spatial differences
`-DHAVE_CLASS`	Link CLASS
`-DHAVE_HEALPIX`	Enable pixelised metric light cones
`-DVELOCITY`	Enable coarse-grained velocity utilities
`-DFIXED_ICS`	Generate fixed-amplitude, random-phase basic initial conditions
`-DCHECK_B`	Compute a diagnostic copy of the vector perturbation
`-DCIC_PROJECT_TIJ`	Use CIC projection for anisotropic stress \(T_{ij}\), reducing numerical artefacts at additional compute cost; recommended with `TENSOR_EVOLUTION`
`-DTENSOR_EVOLUTION`	Dynamically evolve tensor modes
`-DANISOTROPIC_EXPANSION`	Enable Bianchi-I homogeneous tensor mode
`-DICGEN_RELIC`	Compile the optional RELIC/second-order initial-condition reader; this workflow is not production-supported by default
`-DCOLORTERMINAL`	Enable ANSI colour highlighting in terminal output
`-DPARTICLE_LC_BALANCED_IO=1`	Redistribute single-file particle light-cone output work between ranks to improve I/O balance
`-DPCL_EXTRA_CAPACITY=<n>`	Reserve additional per-rank particle capacity

Required definitions and definitions coupled to optional libraries are generated by the build rules. Add independent optional definitions through EXTRA_DEFINES:

ENABLE_CLASS := 1
ENABLE_HEALPIX := 1
EXTRA_DEFINES += FIXED_ICS
EXTRA_DEFINES += VELOCITY
EXTRA_DEFINES += CHECK_B

-DFIXED_ICS affects realisations generated by the basic IC generator. It replaces the random Box-Muller mode amplitude with its root-mean-square value while retaining the random phase and seed-dependent phase reproducibility. This suppresses finite-volume variance, especially on large scales, but changes higher-order statistics. It does not by itself generate an opposite-phase paired simulation.

-DHAVE_HEALPIX requires the HEALPix C library. -DPARTICLE_LC_BALANCED_IO=1 enables redistribution of single-file particle light-cone output between ranks; without it, each rank writes its own contiguous particle contribution. Both modes support non-CUDA-aware MPI through host staging.

CUDA-aware MPI selection¶

CUDA-aware MPI is selected at runtime, not by a gevolution compiler definition. The MPI library itself must have been built with CUDA device-buffer support. gevolution and LATfield2 then activate direct device-pointer MPI only when:

LATFIELD2_DISABLE_CUDA_AWARE_MPI is not enabled; and
at least one recognised positive support hint is enabled.

Recognised support hints are:

LATFIELD2_ENABLE_CUDA_AWARE_MPI
MPICH_GPU_SUPPORT_ENABLED
MV2_USE_CUDA
PSM2_CUDA
OMPI_MCA_opal_cuda_support
OMPI_MCA_mpi_cuda_support

An environment value is considered enabled unless it is exactly 0, false, or FALSE.

Hints are not a capability probe

Do not set a CUDA-aware MPI hint merely to request better performance. These variables tell gevolution and LATfield2 that passing device pointers directly to MPI is safe. Enabling one with an MPI library that lacks CUDA support can cause segmentation faults inside MPI.

To force the portable host-staged path:

export LATFIELD2_DISABLE_CUDA_AWARE_MPI=1

To explicitly enable direct device-pointer MPI after validating the system MPI:

export LATFIELD2_ENABLE_CUDA_AWARE_MPI=1

No additional nvcc switch is required for either runtime path. Rebuild gevolution, parallel HDF5, and related libraries against the same MPI installation when changing MPI implementations.

Output format controls¶

These definitions alter external Gadget-2 files. Use the same values when building related tools such as lccat, and record non-default choices with the simulation metadata.

Definition	Default	Behaviour
`-DGADGET_LENGTH_CONVERSION=<value>`	`0.001`	Gadget length unit in Mpc/h
`-DGADGET_MASS_CONVERSION=<value>`	`1.0e10`	Gadget mass unit in solar masses/h
`-DGADGET_VELOCITY_CONVERSION=<value>`	`3.335640952e-6`	Gadget velocity unit relative to the speed of light
`-DGADGET_ID_BYTES=<4-or-8>`	`8`	Select 32-bit or 64-bit Gadget particle IDs

Using four-byte IDs reduces output size but is only valid when every written particle ID fits in 32 bits. Mismatched conversion or ID definitions between writers and readers produce incorrect or unreadable files.

Advanced compile-time tuning¶

Most users should retain these defaults. Change them only to address a specific limit, memory constraint, or measured I/O bottleneck.

Definition	Default	Behaviour and guidance
`-DMAX_OUTPUTS=<n>`	`32`	Maximum number of scheduled snapshots, spectra, hibernation entries, and light cones represented by fixed-size arrays
`-DPARAM_MAX_LINESIZE=<n>`	`1024`	Maximum settings-file line length, including long comma-separated output schedules
`-DPARAM_MAX_LENGTH=<n>`	`256`	Maximum parsed parameter-name or value length
`-DMAX_PCL_SPECIES=<n>`	`6`	Maximum total particle species, including CDM and optional baryons
`-DCYCLE_INFO_INTERVAL=<n>`	`10`	Number of integration cycles between progress messages
`-DPCLBUFFER=<n>`	`1048576`	Particle I/O chunk size; larger values use more host/device staging memory
`-DPIXBUFFER=<n>`	`1048576`	Initial size and growth increment of each HEALPix device pixel buffer
`-DHEALPIX_SHELL_CHUNK=<n>`	`16`	Number of metric light-cone shells processed per projection chunk
`-DMAX_INTERSECTS=<n>`	`12`	Maximum periodic light-cone intersections retained for a domain; increase if the runtime warning reports missing intersections
`-DCLASS_K_PER_DECADE_FOR_PK=<n>`	unset	Default CLASS `k_per_decade_for_pk` when it is not supplied through a precision file
`-DLIGHTCONE_INTERPOLATE`	disabled	Select an alternative vector-field interpolation for metric light-cone output; experimental

Large output schedules

Increasing MAX_OUTPUTS alone may not be sufficient. A settings line containing hundreds of comma-separated output redshifts can exceed both PARAM_MAX_LINESIZE and the value buffer controlled by PARAM_MAX_LENGTH. Increase and test these limits together. Larger parser buffers also increase stack and fixed-buffer usage.

Developer-only tuning constants such as STACK_ALLOCATION_LIMIT and LIGHTCONE_IDCHECK_ZONE should not normally be changed by users. H5_DEBUG is useful only when diagnosing HDF5 failures because it leaves automatic HDF5 error reporting enabled.

The settings reference explains how these fixed limits interact with long output schedules and species lists.

EXTERNAL_IO is currently unsupported

The LATfield2 external I/O server has not yet been ported to the GPU backend. Defining EXTERNAL_IO causes main.cu to fail compilation with an explanatory error. Existing conditional source remains in place for a possible future port.

Compile¶

make print-config
make check-config
make -j

Additional targets include:

make unit-tests
make parser-tests
make lccat
make lcmap
make help

parser-tests, help, and print-config do not require a CUDA architecture. The lcmap target requires HEALPix C/C++, CFITSIO, and GSL. The CSCS-only run, profile, scaling-test, and Slurm GPU test targets are available only when CONFIG=config/alps.mk is selected.

Common build errors¶

LATfield2.hpp not found
Correct the LATfield2 include path.

CUDA architecture is unsupported
Set CUDA_ARCH=sm_XX for the installed GPU and CUDA toolkit.

HDF5 symbols are missing or MPI types conflict
Use matching MPI compiler wrappers and HDF5 libraries. Prefer the parallel HDF5 build for production.

CLASS or HEALPix symbols are missing
Either add their include/library paths and linked libraries, or disable the corresponding ENABLE_CLASS or ENABLE_HEALPIX setting.

Host compiler is rejected by nvcc
Select a CUDA-supported host compiler through -ccbin.