Skip to content

FAQ and troubleshooting

The run fails during MPI initialisation

Confirm that the launched rank count equals n * m, and that both n and m are at least two. Layouts with a self-neighbour are unsupported. These conditions and the Ngrid divisibility rules are checked automatically before large simulation allocations.

The FFT or lattice initialisation fails

Check:

Ngrid % n == 0
(Ngrid / m) % 2 == 0

Also confirm that the mesh is cubic and the LATfield2 build has FFT3D.

No CUDA-capable device is found

Check driver/toolkit compatibility and CUDA_VISIBLE_DEVICES. Under a scheduler, verify GPU allocation and binding. For a one-GPU workstation, start MPS before launching four ranks.

Build configuration is incomplete

Run make print-config to inspect the resolved profile. GPU targets require an explicit CUDA_ARCH, for example sm_86 for an RTX 3060 or sm_90 for Grace Hopper. Set it in config/local.mk, select a tracked profile, or pass it on the command line.

When USE_PKG_CONFIG=1, make check-config reports missing core dependency packages. On systems without useful pkg-config metadata, set USE_PKG_CONFIG=0 and provide CPPFLAGS, LDFLAGS, and LDLIBS in the selected configuration.

Several ranks unexpectedly use the same production GPU

Fix scheduler or MPI GPU binding. Sharing is only the documented workstation fallback and should use MPS intentionally.

The run crashes inside MPI during GPU communication

An MPI stack without CUDA device-buffer support may segfault if a CUDA-aware support hint incorrectly enables direct device-pointer MPI. Force host staging and retry:

export LATFIELD2_DISABLE_CUDA_AWARE_MPI=1
mpirun -np 4 ./gevolution -n 2 -m 2 -s settings.ini

Inspect the environment for accidentally enabled hints:

env | grep -E 'LATFIELD2|MPICH_GPU|MV2_USE_CUDA|PSM2_CUDA|OMPI_MCA_.*cuda'

For Open MPI, verify that the installation was built with CUDA support before enabling direct device communication:

ompi_info --parsable --all | grep mpi_built_with_cuda_support

Do not treat LATFIELD2_ENABLE_CUDA_AWARE_MPI=1 as a request for automatic detection; it asserts that direct device-pointer MPI is supported. See CUDA-aware MPI selection.

MPS does not start or ranks cannot connect

  • Ensure all ranks use the same CUDA_MPS_PIPE_DIRECTORY.
  • Start one daemon, not one per rank.
  • Check permissions on pipe and log directories.
  • Stop stale user-owned MPS daemons before retrying.
  • Follow site policy on systems with managed MPS.

CUDA memory allocation fails

  • Reduce Ngrid, particle tiling, or the number of output fields.
  • Remember that all four MPS ranks share one workstation GPU's memory.
  • Reduce PCL_EXTRA_CAPACITY if it is excessive.
  • Account for FFT workspaces and output staging buffers.
  • Inspect particle imbalance between ranks.

HDF5 output fails

  • Create the output directory before launch.
  • Confirm HDF5 and MPI were built against compatible MPI implementations.
  • Use parallel HDF5 with H5_HAVE_PARALLEL for production.
  • Check quota, free space, and filesystem availability.

Compilation fails when EXTERNAL_IO is defined

This is intentional. The LATfield2 external I/O server has not yet been ported to the GPU backend, so main.cu rejects EXTERNAL_IO at compile time. Remove the definition and use the supported direct output paths. The retained conditional source is reserved for a possible future port.

CLASS integration fails

  • Compile and link CLASS consistently with the selected host compiler.
  • Define HAVE_CLASS.
  • Ensure transfer tables cover all simulation wavenumbers.
  • Use synchronous-gauge transfer functions for the basic generator.
  • Check the optional precision file passed with -p.
  • Avoid mixing independent Tk file or mPk file inputs with CLASS linear radiation/fluid perturbations unless their realisations are demonstrably consistent.

See Initial conditions and particles and Radiation and additional fluids.

HEALPix output fails

  • Define HAVE_HEALPIX and link the HEALPix C library.
  • Build lcmap with the required HEALPix C++ and CFITSIO libraries.
  • Check lightcone Nside, shell coverage, and available GPU memory.

Hibernation files are not written

The GPU executable does not support hibernation output. Non-empty hibernation schedules and positive wallclock limits are rejected during settings validation, before the simulation starts.

Several settings errors are printed together

This is intentional. Strict parsing and essential bounds checks collect independent errors before aborting. Fix every reported error, including malformed numeric values, list overflow, conflicting duplicates, and invalid process decomposition.

Scaling is slow

  • Check the benchmark timing breakdown.
  • Confirm one rank per GPU and correct CPU/NUMA binding.
  • Verify whether the run is using CUDA-aware or host-staged MPI.
  • Determine whether distributed FFT communication dominates.
  • Test without large output to isolate filesystem cost.
  • Compare process-grid shapes.

Output is unexpectedly large

Review snapshot and light-cone schedules, selected field components, particle tracer factors, field downgrade factor, HEALPix resolution, shell factor, and covering factor. The exact reduction and light-cone rules are listed in Output and light-cone settings.

A legacy settings file selects prevolution or FalconIC

Those generators are removed in gevolution 2.0. Generate supported input externally and use read from disk, or use the basic generator.