Cheryl Lau, PhD

Welcome!

< BACK

Instructions on how to couple Phantom to CMacIonize

Disclaimer: This method was found with Bert’s help and has been tested on three machines (one PC and two HPCs), but there is no guarantee.

The following instructions assume you use gfortran on a linux machine.

Step 0: Download Phantom and CMacIonize

Create a fork repo of Phantom and CMacIonize from https://github.com/danieljprice/phantom and https://github.com/bwvdnbro/CMacIonize.

Clone the fork into your computer by running the following commands:

git clone https://github.com/<your git username>/phantom.git
git clone https://github.com/<your git username>/CMacIonize.git

Step 1: Deal with the HDF5

Check if HDF5 has already been installed. If not, download it from source by running this on the command line:

wget https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.12/hdf5-1.12.2/src/hdf5-1.12.2.tar.gz 
gzip -cd hdf5-1.12.2.tar.gz | tar xvf -

(or replace 1.12.2 with any other latest version of HDF5).

Then, go to the top directory of your HDF5 and run:

./configure --enable-fortran --enable-cxx
make
make install

Step 2: Go to your .bashrc

Put in the following lines, if it is not already there:

# Phantom
export SYSTEM=gfortran
export PHANTOM_DIR=<path to phantom dir>
export OMP_SCHEDULE="dynamic"
export OMP_STACKSIZE=512M
ulimit -s unlimited
export HDF5=no
export HDF5ROOT=<path to HDF5 dir>
export LD_LIBRARY_PATH=<path to HDF5 dir>/hdf5/lib:$LD_LIBRARY_PATH

# CMacIonize
export CMI_DIR=<path to CMacIonize dir>
export HDF5_ROOT=<path to HDF5 dir>/hdf5

# Activate compilers
module load openmpi/5.0.5
module load gcclibs/11.4.1
module load ucx/1.16.0

Don’t forget to source .bashrc.

Step 3: Go to CMacIonize top directory

Run the following lines:

mkdir build
cd build
cmake -DMAKE_BUILD_TYPE=Release -DMAX_NUMBER_OF_THREADS=192 -DACTIVATE_FORTRAN_BINDING=True -DOUTPUT_HEATING=True -DCMAKE_Fortran_COMPILER=/usr/bin/gfortran <path to CMacIonize dir>
make

Before running make, check that it has found HDF5 and the fortran compiler.

Now you should see that, inside CMacIonize/build/compilation, it has generated two files, named cmi_fortran_includes.txt and cmi_fortran_libs.txt.

Step 4: Deal with the Phantom Makefile

Open the Makefile in phantom/build.

Step 4.1: Add in new setup

Create a new setup that looks like this:

ifeq ($(SETUP), cmi)
#	Coupling Phantom to CMacIonize for adding ionizing radiation
    SETUPFILE=<your sim's setup file>.f90
    FPPFLAGS= -DPHOTOION
    SRCPHOTOION=utils_cmi.f90 kdtree_cmi.f90 hnode_cmi.f90 heating_cooling_cmi.f90 photoionize_cmi.F90
    CMACIONIZE=yes
endif
Step 4.2: Add in LD flags

Underneath the ifeq ($(MCFOST), yes) block, add the following. Copy the ld flags listed in cmi_fortran_libs.txt to here:

ifeq ($(CMACIONIZE), yes)
    LDFLAGS+= <whatever is written in cmi_fortran_libs.txt>
endif

For example, it could look like

ifeq ($(CMACIONIZE), yes)
    LDFLAGS+= -L/$(CMI_DIR)/build/lib -lCMIFortranLibrary -lCMILibrary -lLegacyEngine -lSharedEngine $(HDF5ROOT)/lib/libhdf5.so /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/x86_64-linux-gnu/libdl.a /usr/lib/x86_64-linux-gnu/libm.so  -lstdc++ -lc
endif

Important note! The specific compilation flags listed in cmi_fortran_libs.txt are machine-dependent. If you’re running the code on different machines, you will have to repeat step 3 and create an extra block for each machine you use. For example, I have the following for St Andrews’ HPC cluster Hypatia:

ifeq ($(CMACIONIZE_HYPATIA), yes)
    LDFLAGS+= -L/$(CMI_DIR)/build/lib -lCMIFortranLibrary -lCMILibrary -lLegacyEngine -lSharedEngine $(HDF5ROOT)/lib/libhdf5.so /usr/lib64/libz.so /usr/lib64/libdl.a /usr/lib64/libm.so /software/MPI/openmpi-5.0.5/lib/libmpi.so -lstdc++ -lc
endif

and in the setup block, I’d now put CMACIONIZE_HYPATIA=yes instead of CMACIONIZE=yes.

Step 4.3: Add in the source files

Look for the line SOURCES= physcon.f90 ${CONFIG} ${SRCKERNEL} io.F90… in the Makefile.

Add ${SRCPHOTOION} into the list of source codes. Compilation order matters! It has to be after kdtree.F90 and linklist_kdtree.F90, but before readwrite_infile.F90, force.F90, deriv.F90, step_leapfrog.F90 and initial.F90.

Again, look for SRCDUMP= physcon.f90 ${CONFIG} ${SRCKERNEL} io.F90… in the Makefile, and add ${SRCPHOTOION} into the list. You could put it after ${SRCPOT} ${SRCPHOTO}.

Step 4.4: Add in the includes

Underneath the phantom: checksystem checkparams $(OBJECTS) phantom.o block, add in the following. Copy the flags listed in cmi_fortran_includes.txt to here:

photoionize_cmi.o: photoionize_cmi.F90
	$(FC) -c $(FFLAGS) <whatever is written in cmi_fortran_includes.txt> $< -o $@

For example,

photoionize_cmi.o: photoionize_cmi.F90
	$(FC) -c $(FFLAGS) -fopenmp -I$(CMI_DIR)/build/include $< -o $@

Step 5: Install the photoionization-relevant source codes

Please checkout the following new modules from my phantom repo:

  1. photoionize_cmi.F90
  2. kdtree_cmi.f90
  3. hnode_cmi.f90
  4. heating_cooling_cmi.f90
  5. utils_cmi.f90
    Make sure they are in your repo’s phantom/src/main/ directory.

Here goes the tricky bit. We now modify the ‘guts’ of the code to link it to the photoionization mods.

Go to your own repo’s phantom/src/main and open the following files:

  1. initial.F90
  2. deriv.F90
  3. force.F90
  4. step_leapfrog.F90
  5. dtype_kdtree.F90
  6. kdtree.F90
  7. readwrite_infile.F90

Open the same files in my phantom repo. You will find extra blocks of codes that are enclosed in #ifdef PHOTOION. A simple ‘ctrl+F PHOTOION’ will do the job. Please copy all of these into your own Phantom repo at their respective locations in the code.

Since Phantom updates quite often, I cannot guarantee that these codes will work in all future versions. You will likely have to figure out how to let your own Phantom incorporate the photoionization modules should they fail to compile/run. I’d be very happy to help, of course!

Step 6: Compile everything

Create a new work directory somewhere outside of the phantom repo. Go to your new directory and write a local Makefile for the new setup that you created in step 4.1.

~/phantom/scripts/writemake.sh cmi > Makefile

Finally, compile everything.

make; make setup

Best of luck and happy trouble-shooting.