Galaxy Collision

You can find this application in the demos folder of your Jupyter notebook environment.

galaxy_collision.ipynb

In this tutorial, use the vanilla version of Gadget4 on Camber to simulate a galaxy collision.

Since the galaxy collision example comes with Gadget4 source code, we focus more on showcasing how you can with Gadget4 on Camber.

Set up Gadget4

First, we will need to clone the Gadget4 repo. You will notice that this is not the official Gadget4 repo. We made certain changes to the vanilla version of the code to make it work with the Camber platform.

To be specific, the following files are added/modified:

buildsystem/Makefile.comp.camber (added)

CC       =  mpicc   -std=c11  # sets the C-compiler
CPP      =  mpicxx  -std=c++11 # sets the C++-compiler
OPTIMIZE =  -ggdb -O3 -march=native  -Wall -Wno-format-security -L$(ZLIB_HOME)/lib 

ifeq (EXPLICIT_VECTORIZATION,$(findstring EXPLICIT_VECTORIZATION,$(CONFIGVARS)))
CFLAGS_VECTOR += -mavx2  # enables generation of AVX instructions (used through vectorclass)
CPV      =  $(CPP)
else
CFLAGS_VECTOR = 
CPV      =  $(CPP)
endif

buildsystem/Makefile.path.camber (added)

GSL_INCL   =  -I$(GSL_HOME)/include
GSL_LIBS   =  -L$(GSL_HOME)/lib  -Xlinker -R -Xlinker $(GSL_HOME)/lib
FFTW_INCL  =  -I$(FFTW_HOME)/include
FFTW_LIBS  =  -L$(FFTW_HOME)/lib  -Xlinker -R -Xlinker $(FFTW_HOME)/lib
HDF5_INCL  =  -I$(HDF5_HOME)/include
HDF5_LIBS  =  -L$(HDF5_HOME)/lib -Xlinker -R -Xlinker $(HDF5_HOME)/lib
HWLOC_INCL =  -I$(HWLOC_HOME)/include 
HWLOC_LIBS =  -L$(HWLOC_HOME)/lib -Xlinker -R -Xlinker $(HWLOC_HOME)/lib

Makefile (modified)

ifeq ($(SYSTYPE),"bwforcluster")
include buildsystem/Makefile.comp.gcc
include buildsystem/Makefile.path.bwforcluster
endif

+ ifeq ($(SYSTYPE),"camber")
+ include buildsystem/Makefile.comp.camber
+ include buildsystem/Makefile.path.camber
+ endif

ifndef LINKER
LINKER = $(CPP)
endif

These are standard changes a user have to make when setting up Gadget4 on a new platform. We’ve made them for you so you can just focus on running the simulation. If you wish to bring your own Gadget source code, however, you have to make these changes yourself, and maybe add more to what we already have.

!rm -rf gadget4
!git clone -b camber --single-branch --depth 1 https://github.com/CamberCloud-Inc/gadget4-camber.git gadget4

Cloning into 'gadget4'...
remote: Enumerating objects: 342, done.[K
remote: Counting objects: 100% (342/342), done.[K
remote: Compressing objects: 100% (285/285), done.[K
remote: Total 342 (delta 57), reused 308 (delta 52), pack-reused 0 (from 0)[K
Receiving objects: 100% (342/342), 1.74 MiB | 7.03 MiB/s, done.
Resolving deltas: 100% (57/57), done.
Updating files: 100% (301/301), done.

Run your simulation

First, let’s import camber.

import camber

Gadget4 requires you to build from the root level of the repository, and recommends creating separate directories for simulations through the usage of the DIR variable. The DIR variable indicates a path to the working directory relative from your current directory, which should be the root of your gadget repo. Also note the addition of the SYSTYPE environment variable, this is required during build so Gadget4 could compile with the right dependencies.

build = camber.mpi.create_job(
    command="make -j 8 DIR=examples/G2-galaxy",  # path relative to the root of gadget repo
    mount_dir="./gadget4",  # mount to the root level of the gadget repo, making it the working directory
    extra_env_vars={"SYSTYPE": "camber"},  # ensure you use the camber systype
    engine_size="XSMALL",
)

We can check on the status of build by running the cell below. It should complete within the minute.

build

CamberJob({"job_id": 7309, "status": "COMPLETED", "engine_size": "XSMALL", "engine_type": "MPI", "command": "make -j 8 DIR=examples/G2-galaxy", "with_gpu": false})

Gadget4 binaries compiled on the Camber platform are always MPI-aware. This means we can and should run them using mpirun. Let’s run the simulation using MPI over 16 cores on a SMALL instance.

Note that we are changing the mount directory, this is the recommended so that you could run the simulation directly next to your binaries.

run = camber.mpi.create_job(
    command="mpirun -np 16 ./Gadget4 param.txt",
    mount_dir="./gadget4/examples/G2-galaxy",  # mount to the root level of the build directory just now
    engine_size="SMALL",
)

Check the status of your simulation by executing the cell below. The simulation should only take a few minutes.

run

CamberJob({"job_id": 7310, "status": "COMPLETED", "engine_size": "SMALL", "engine_type": "MPI", "command": "mpirun -np 16 ./Gadget4 param.txt", "with_gpu": false})

Once the job goes into COMPLETED status, you could read or download its logs.

run.read_logs(tail_lines=25)

Final time=3 reached. Simulation ends. RESTART: Writing restart files. RESTART: Backing up restart files... RESTART: no pre-existing restart files for renaming were found. RESTART: Writing restart files group #1 out of 16... RESTART: Writing restart files group #2 out of 16... RESTART: Writing restart files group #3 out of 16... RESTART: Writing restart files group #4 out of 16... RESTART: Writing restart files group #5 out of 16... RESTART: Writing restart files group #6 out of 16... RESTART: Writing restart files group #7 out of 16... RESTART: Writing restart files group #8 out of 16... RESTART: Writing restart files group #9 out of 16... RESTART: Writing restart files group #10 out of 16... RESTART: Writing restart files group #11 out of 16... RESTART: Writing restart files group #12 out of 16... RESTART: Writing restart files group #13 out of 16... RESTART: Writing restart files group #14 out of 16... RESTART: Writing restart files group #15 out of 16... RESTART: Writing restart files group #16 out of 16... RESTART: done. load/save took 0.111144 sec, total size 7.9483 MB, corresponds to effective I/O rate of 71.5135 MB/sec endrun called, calling MPI_Finalize() bye!

ACCEL: Start tree gravity force computation… (1626 particles)

run.download_log()

Congratulations, you just ran your first Gadget simulation on Camber! It should be noted that there are other examples in the Gadget4 repository in the examples folder. You are welcome to try them out as well following the same flow from above.