ChaNGA

Overview

ChaNGA (Charm N-body GrAvity solver) is a parallel code for cosmological N-body simulations. It uses the Charm++ runtime system for load balancing and communication.

Building ChaNGA on Camber HPC

Prerequisites

• No modules needed – default compilers all work
• Git, GCC, and Gfortran are available by default

Build Process

1. Clone and Build Charm++:

   mkdir -p manga && cd manga
   git clone -b v7.0.0 --depth 1 --single-branch https://github.com/UIUC-PPL/charm
   cd charm
   ./build ChaNGa multicore-linux-x86_64 gcc gfortran "-j$(nproc)"
   cd ..

2. Clone Utility Libraries:

   git clone --depth 1 https://github.com/N-BodyShop/utility.git

3. Clone and Build ChaNGA:

   git clone -b moving-mesh --depth 1 --single-branch https://github.com/N-BodyShop/changa.git
   cd changa
   ./configure --enable-moving-mesh --enable-cooling=ad
   make -j$(nproc)

Running ChaNGA Jobs

Common Error and Solution

Problem: ChaNGA jobs fail with:

“Multiple PEs assigned to same core. Set affinity options to correct or lower the number of threads.”

Root Cause: Multiple Processing Elements (PEs) are being bound to the same physical CPU core when SLURM schedules tasks without proper core spacing.

Correct SLURM Job Script

#!/bin/bash
#SBATCH --job-name="ChaNGA-simulation"
#SBATCH --partition=cpu-queue
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --time=48:00:00
#SBATCH --mail-type=BEGIN,END
#SBATCH --mail-user=your-email@cambercloud.com

# Set environment variables
export CHARM_DIR=$HOME/manga/charm
export CHANGA_HOME=$HOME/manga/changa

# Run ChaNGa
cd $CHANGA_HOME
./ChaNGa ++ppn 8 your_param_file.param

Key Configuration Points

• Explicitly specify --nodes and --ntasks-per-node to avoid PE affinity issues
• Use ++ppn parameter to match your SLURM task allocation
• Charm++ is strict about core binding and will abort if it detects oversubscription
• No CPU binding flags needed (avoid --cpu-bind in most cases)

Multi-Node Jobs

For multi-node simulations:

#!/bin/bash
#SBATCH --job-name="ChaNGA-multinode"
#SBATCH --partition=cpu-queue
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=16
#SBATCH --time=48:00:00

# Run on multiple nodes
./ChaNGa ++ppn 16 ++p 64 your_param_file.param

Where:

• ++ppn 16: 16 processing elements per node
• ++p 64: Total processing elements (4 nodes × 16 PEs/node)

Environment Variables

export CHARM_DIR=${HOME}/manga/charm
export CHANGA_HOME=${HOME}/manga/changa
export PATH="${CHANGA_HOME}:${PATH}"

Docker Container Option

Alternatively, use the pre-built ChaNGA container:

# Pull the container
spack load apptainer@1.3.4
apptainer pull docker://392513736110.dkr.ecr.us-east-2.amazonaws.com/changa:v2024.8.20

# Run with container
apptainer exec changa_v2024.8.20.sif ./ChaNGa ++ppn 8 your_param_file.param

Troubleshooting

Performance Issues

• Ensure PE count matches available cores
• Use ++ppn value equal to cores per node
• Monitor memory usage for large simulations

Common Errors

1. “Multiple PEs assigned to same core”: Use explicit SLURM task distribution
2. Build failures: Ensure all dependencies are available and paths are correct
3. Segmentation faults: Check parameter file format and initial conditions

Best Practices

• Start with single-node jobs to verify setup
• Use appropriate time limits for your simulation size
• Monitor job efficiency with seff <job_id>
• Save checkpoints for long-running simulations
• Test with small particle counts first