Docs
Tutorials
Simulate fluid mechanics with MPI

Simulate fluid mechanics with MPI

You can find this tutorial in the demos folder of your Jupyter notebook environment.

    • athena_read.py
    • athinput.orszag-tang
    • fluid_dynamics_mpi.ipynb
    • plot_output.py
    • Camber frames of the Orszag tang plot

    frames 3, 23, 43, and 63 of a simulation of the Orszag-Tang vortex. Scroll to the end for a video

    This tutorial demonstrates a simulation and visualization of a well-known problem in magnetohydrodynamics. The simulation is parallelized using the Camber mpi package, and it uses the Athena library to execute Athena++ code on Camber’s OpenMPI infrastructure.

    Simulate a 2D Orszag-Tang vortex

    The following steps create an MPI cluster to compute the Orszag-Tang Vortex, a popular demonstration problem in fluid mechanics. After the output is generated, an imported Python script creates a visualization.

    Spin-Up MPI cluster

    The first step is to import Camber and initialize the MPI cluster:

    import camber
    # download Athena++ from public repo if not already present in your directory
!git clone https://github.com/PrincetonUniversity/athena.git
    Cloning into 'athena'...
remote: Enumerating objects: 47696, done.
remote: Counting objects: 100% (1141/1141), done.
remote: Compressing objects: 100% (512/512), done.
remote: Total 47696 (delta 745), reused 897 (delta 628), pack-reused 46555 (from 1)
Receiving objects: 100% (47696/47696), 28.32 MiB | 23.52 MiB/s, done.
Resolving deltas: 100% (36144/36144), done.
Updating files: 100% (660/660), done.

    Now create an MPI job to compile Athena++. This step ensures the code compiles with the correct MPI environment:

    # Now we create an MPI job to compile Athena++. This step ensure the code is compiled with the correct MPI environment
compile_job = camber.mpi.create_job(
    command="cd athena && python configure.py -b --prob=orszag_tang -mpi -hdf5 --hdf5_path=${HDF5_HOME} && make clean && make all -j$(nproc)",
    engine_size="SMALL"
)

    Check the compile_job output to monitor the job status:

    compile_job
    CamberJob({"job_id": 7176, "status": "COMPLETED", "engine_size": "SMALL", "engine_type": "MPI", "command": "cd athena && python configure.py -b --prob=orszag_tang -mpi -hdf5 --hdf5_path=${HDF5_HOME} && make clean && make all -j$(nproc)", "with_gpu": false})

    Once the job status turns to COMPLETED, use read_logs() to check the output:

    compile_job.read_logs(tail_lines=10)

    Run the MPI job

    After compilation finishes, use the create_job method to start the Orszag-Tang simulation using the Athena binary and input file:

    # run camber with a medium instance
run_job = camber.mpi.create_job(
    command="mpirun -np 16 athena/bin/athena -i athinput.orszag-tang",
    engine_size="MEDIUM"
)
    # check job status
run_job
    CamberJob({"job_id": 7177, "status": "COMPLETED", "engine_size": "MEDIUM", "engine_type": "MPI", "command": "mpirun -np 16 athena/bin/athena -i athinput.orszag-tang", "with_gpu": false})
    # Check the job progress once it begins running or has completed
run_job.read_logs(tail_lines=10)
    cycle=1770 time=9.9959319121303625e-01 dt=4.0680878696375267e-04
cycle=1771 time=1.0000000000000000e+00 dt=5.3994982319190274e-04

    Terminating on time limit
time=1.0000000000000000e+00 cycle=1771
tlim=1.0000000000000000e+00 nlim=-1
    

    zone-cycles = 116064256
cpu time used  = 6.6417440000000001e+00
zone-cycles/cpu_second = 1.7474966815944728e+07

    Once the job starts running, Camber generates a number of OrszagTang output files. This output serves as the input for visualization in the next section.

    When the job finishes, you can capture the output with the download_log method.

    run_job.download_log()
    To view these logs, check the folder where you are running this workload for a file called `job_.log`

    Read and visualize output

    Now use a plot function generate images for your output.

    First, if it doesn’t exist, create a file for your output images:

    !mkdir output_images
    Now import the plot function from your local files and run it on your input. This generates 100 PNG files in your `output_images` directory:
    # import a custom script for reading and plotting the hdf5 outputs, placing images in the output_images directory
from plot_output import plot_output
plot_output()
    ### Turn the images into a video

    The plot_output function from the preceding step also generates a movie from all the output images. Here is a 2D simulation of the Orszag-Tang Vortex:

    from IPython.display import Video
Video("density.mov")
    <IPython.core.display.Video object>

    Read more

    For more fluid dynamics and MPI, try the notebook Parallel Athena simulations on MPI.

    Launch parallel jobs with scatterApplications