Docs
Applications
Astronomy
Multi-MPI parameter sweeps

Multi-MPI parameter sweeps

You can find this application in the demos folder of your Jupyter notebook environment.

    • athena_read.py
    • athinput.blast
    • mpi_parameter_sweep.ipynb
    • plot_output.py
    • A parallel simulation of four spherical blasts

    A parallel simulation of four spherical blasts. Scroll down for the video.

    In this tutorial, use the create_scatter_job function to run Athena++ code to create parallel MPI simulations of a spherical blast wave.

    Compile the executable

    First import Camber:

    import camber

    If needed, clone the Athena repo:

    # working directory is where the notebook is located
!git clone https://github.com/PrincetonUniversity/athena.git

    Create an MPI job to compile Athena++. This ensures the code is compiled with the correct MPI environment:

    compile_job = camber.mpi.create_job(
    command="cd athena && python configure.py --prob=blast -mpi -hdf5 --hdf5_path=${HDF5_HOME} && make clean && make all -j$(nproc)",
    engine_size="SMALL"
)
    # check the status of the job
compile_job.status
    'COMPLETED'

    Run the simulation grid 

    # The blast problem begins with a central sphere of material with higher pressure than its surroundings.
# The prat parameter sets the ratio.
# In this case, four values ranging from a factor of 10 to a factor of 300.
params = {
    "prat": ["10", "30", "100", "300"]
}
    # Run scatter jobs using MEDIUM engines.
# We take advantage of Athena's functionality for over-writing a parameter in the input file frome the
# command line as well as the ability to set the runtime directory
jobs = camber.mpi.create_scatter_job(
    command_template="mpirun -np 16 athena/bin/athena -i athinput.blast problem/prat={prat} -d run{prat}",
    template_params_grid=params,
    engine_size="MEDIUM"
)
    # lets check in on how are jobs are doing
jobs
    [CamberJob({"job_id": 7012, "status": "COMPLETED", "engine_size": "MEDIUM", "engine_type": "MPI", "command": "mpirun -np 16 athena/bin/athena -i athinput.blast problem/prat=30 -d run30", "with_gpu": false}),
 CamberJob({"job_id": 7013, "status": "COMPLETED", "engine_size": "MEDIUM", "engine_type": "MPI", "command": "mpirun -np 16 athena/bin/athena -i athinput.blast problem/prat=10 -d run10", "with_gpu": false}),
 CamberJob({"job_id": 7014, "status": "COMPLETED", "engine_size": "MEDIUM", "engine_type": "MPI", "command": "mpirun -np 16 athena/bin/athena -i athinput.blast problem/prat=100 -d run100", "with_gpu": false}),
 CamberJob({"job_id": 7015, "status": "COMPLETED", "engine_size": "MEDIUM", "engine_type": "MPI", "command": "mpirun -np 16 athena/bin/athena -i athinput.blast problem/prat=300 -d run300", "with_gpu": false})]

    Once the jobs are completed, plot the results:

    # Prepare a directory for output images
!mkdir output_images
    # import a custom script for reading and plotting the hdf5 outputs, placing images in the output_images directory
# each frame is a 2D slice throught the center of the 3D cartesian grid. This assumes each simulation output
# is in its own runtime directory. We pass the same params dictionary used above for the scatter job.
from plot_output import plot_output
plot_output(params, "prat")

    Visualize results 

    from IPython.display import Video
Video("density.mov")
    <IPython.core.display.Video object>