Skip to content

Performance and scaling

Benchmarking a run

Compile with -DBENCHMARK and use a representative settings file. A useful benchmark includes the actual physics options and output cadence, because FFTs, CLASS sources, and I/O often dominate runtime.

Record:

  • git revision and compiler definitions;
  • GPU, CPU, node, and interconnect models;
  • CUDA, MPI, FFTW, HDF5, and LATfield2 versions;
  • rank layout, OpenMP threads, and binding;
  • Ngrid, particle counts, species, and output schedule;
  • whether MPI is CUDA-aware.

When comparing CUDA-aware and host-staged runs, record the relevant LATFIELD2_* or MPI support environment variables. The communication mode is selected at runtime rather than by a gevolution compiler definition.

Run enough integration steps to amortise initialisation and filesystem variability.

Interpreting the timing breakdown

With BENCHMARK, the executable reports total and component timings. Compare:

  • projection time;
  • particle kick and drift;
  • gravity solver;
  • FFT share of gravity-solver time;
  • snapshot, spectrum, and light-cone output;
  • initial-condition generation.

Particle updates can scale nearly ideally while total scaling stalls because distributed FFTs or I/O dominate.

Strong scaling

Strong scaling fixes the problem size and increases the number of GPUs. The draft gevolution 2.0 paper reports a representative \(1080^3\)-particle, \(1080^3\)-mesh step on 16 ranks:

Component Approximate time
Total step 2104 ms
\(\phi\) solver 295 ms
\(B_i\) solver 455 ms
\(\chi\) solver 1140 ms
Kick 12 ms
Drift 31 ms
Stress-energy construction 171 ms

The example spends roughly 1.9 seconds of a 2.1-second step in FFTs. Adding GPUs eventually produces diminishing returns when communication dominates.

Weak scaling

Weak scaling increases problem size with resource count. Particle updates and initial-condition generation can remain close to ideal, while FFT communication and filesystem bandwidth become increasingly important.

Single-node scaling may be unusually strong when CUDA-aware MPI can use intra-node GPU peer-to-peer links.

Practical resource planning

  1. Select the smallest mesh and particle sampling that resolves the scientific question.
  2. Estimate GPU memory for particles, fields, FFT workspaces, halos, and output buffers.
  3. Choose a valid, balanced process grid.
  4. Benchmark a short run with production physics and output.
  5. Increase rank count only while time-to-solution or memory requirements justify it.

FFT-dominated runs

  • Prefer fast GPU-aware interconnects and CUDA-aware MPI.
  • Use one rank per GPU.
  • Keep rank placement topology-aware.
  • Avoid unnecessary metric/output fields that trigger additional transforms.

CUDA-aware MPI is an optimisation, not a correctness requirement. The host-staged fallback may be appropriate on workstations and remains useful as a diagnostic baseline on HPC systems.

I/O-dominated runs

  • Reduce snapshot and light-cone cadence.
  • Prefer spectra for frequent monitoring.
  • Use multi-Gadget2 for large particle snapshots.
  • Use parallel HDF5 and the site's recommended filesystem.
  • Measure filesystem variability separately from compute scaling.

Memory limits

Particle arrays can grow when particles move unevenly between ranks. The PCL_EXTRA_CAPACITY compile-time definition reserves additional capacity. Increasing it reduces reallocations but consumes more device memory.

Output staging and FFT workspaces can create peak memory demands beyond the steady-state particle and field footprint.