Developer guide

gevolution remains a comparatively compact particle-mesh code, but GPU work requires explicit attention to execution space, memory location, communication, and compile-time kernel instantiation.

Source map

File	Responsibility
main.cu	GPU executable control flow
gevolution.hpp	Metric and particle evolution algorithms
Particles_gevolution.hpp	GPU particle handler and Gadget-2 I/O
parser.hpp, metadata.hpp	Settings parsing and simulation metadata
ic_basic.hpp, ic_read.hpp	Supported initial-condition workflows
output.hpp	Snapshots, spectra, and light-cone output
tools.hpp	Spectra, diagnostics, and general numerical utilities
radiation.hpp, class_tools.hpp	Linear sources and CLASS integration
background.hpp	Background evolution
velocity.hpp	Optional coarse-grained velocity utilities

main.cpp and retained CPU-oriented helpers are not the source of truth for GPU feature support.

Main control flow

flowchart TD
    A[Parse CLI and settings] --> B[Initialise lattices, fields, FFT plans]
    B --> C[Generate or read initial conditions]
    C --> D[Construct stress-energy]
    D --> E[Solve metric sectors]
    E --> F[Write scheduled output]
    F --> G[Kick and drift particles]
    G --> H[Advance background and time]
    H --> D

LATfield2 concepts

• Lattice describes global and rank-local mesh geometry.
• Field stores scalar, vector, or tensor data on a lattice.
• PlanFFT connects real-space and Fourier-space fields.
• parallel manages the two-dimensional process grid and MPI communication.
• Halo updates communicate mesh boundaries.
• The GPU particle handler manages row sorting and particle exchange.

Adding a field or source

1. Add metadata and parser settings only when a user-facing control is needed.
2. Allocate the field on the correct real or Fourier lattice.
3. Include its memory footprint in resource planning.
4. Implement stress-energy or evolution updates in the appropriate execution space.
5. Add halo updates and FFT plans where required.
6. Add focused output and validation.

Writing GPU-callable update structs

GPU particle and mesh kernels use callable structs so operations are known at compile time. Mark called functions appropriately for host/device execution, pass only required fields and parameters, and avoid host-only APIs in device code.

Prefer lattice_for_each for rank-local real-space mesh loops. Fourier-space loops may remain OpenMP host loops when their fields live on host memory.

Adding a particle species

Define its metadata, initial conditions, stress-energy projection, kick/drift behaviour, output, and species-specific validation. Account for row sorting, exchange, maximum velocity, substeps, and extra device memory.

Adding output

Keep compute data on device until a bulk staging operation is worthwhile. Avoid per-element device/host transfers. For distributed output, define who owns each record and how buffers are communicated before file writes.

Debugging and profiling

• Check CUDA errors close to kernel launches.
• Use small meshes and deterministic seeds.
• Compile with symbols and relevant diagnostic definitions.
• Define H5_DEBUG when HDF5's full automatic error stack is needed.
• Use NVTX ranges already present in the executable.
• Profile timelines with Nsight Systems.
• Use Nsight Compute for selected kernels after timeline analysis identifies a kernel-level bottleneck.
• Compare against CPU/reference results where available.

STACK_ALLOCATION_LIMIT controls when several temporary line buffers switch between fast stack allocation and dynamic allocation. LIGHTCONE_IDCHECK_ZONE controls an internal particle-light-cone duplicate-ID checking region. Treat both as developer-level numerical/performance controls: changes require targeted correctness and memory testing.

Testing

The repository provides unit_tests.cu and a unit-tests make target. Use the same build profile as the executable, for example make CONFIG=config/local.mk unit-tests. New features should also have small integration cases that check scientific output, MPI exchange, and GPU behaviour across more than one domain.