CHOMBO

CHOMBO

Installation

This page describes what I (Gethin) did to install and run CHOMBO. In the first instance, I worked on dyln (an x86_64 machine) and used the GNU C and Fortran compilers (gcc, g++ and gfortran).

First we will look at the prerquisites, an MPI implementation and HDF5, and then CHOMBO itself.

MPICH v1.2.6

Only an MPI 1.2 compliant C binding is required to run CHOMBO in parallel. There are problems getting gfortran to work with both MPICH v1.2.5 and v1.2.6, so I will omit it here. Beware, however of picking up e.g. the Sun fortran compiler as f77 and f90 and thus getting a mixed compiler MPICH build.

Get tar ball:

wget ftp://ftp.mcs.anl.gov/pub/mpi/old/mpich-1.2.6.tar.gz

Treating dylan as an SMP machine (multi-core with shared memory), we will build a shared memory MPI installation:

./configure --prefix=/gsb6/ggdagw/mpich/mpich-1.2.6/ch_shmem --with-device=ch_shmem -cc=gcc -c++=g++ \
-fc=gfortran -f90=gfortran -cflags="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" \
>& config.out
make
make testing
make install

The cflags enable files > 2GB.

Note that Fortran binding will not be built, as MPICH will complain that it can't access the command line through gfortran. Also that I needed to copt mpicxx to mpiCC in the install directory, for some reason..

HDF5 v1.4.4

Get from:

ftp://hdf.ncsa.uiuc.edu/HDF5/prev-releases

For Serial Build

 
./configure --prefix=/gsb6/ggdagw/hdf5/hdf5-1.4.4/serial --enable-production

For Parallel Build

Tested MPI-IO with the program Sample_mpio.c from PHDF5 instructions (appendix) in conjunction with shmem MPI build and tests passed OK.

Straight from the box, we would have a broken build, due to problems with mpicc and DL_OPEN support. So, we will make a cheeky edit. In the file config/commence.in, change the line that defines the LT_LINK_EXE variable to remove the -dlopen self -rpath $(libdir) options.

Note that we've disabled linking against shared libraries. We could have gone the other way and added '-fPIC to the build flags.

Now we configure and build:

CC=/gsb6/ggdagw/mpich/mpich-1.2.6/ch_shmem/bin/mpicc ./configure --disable-shared \
--prefix=/gsb6/ggdagw/hdf5/hdf5-1.4.4/parallel
make
make check
make install

Note the H5dump tests don't pass, but all the PHDF5 ones do, so we'll run with that for the moment.

HDF5 v1.6.7

We could use this too. For a parallel build, for example:

CC=/gsb6/ggdagw/mpich/mpich-1.2.6/ch_shmem/bin/mpicc ./configure --disable-shared \
--prefix=/gsb6/ggdagw/hdf5/hdf5-1.6.7/parallel
make
make check
make install

Seems to be OK.

CHOMBO

The Oct07 release is installed in /gsb6/ggdagw/chombo/Chombo-2.0.

Without HDF5

lib/mk/Make.defs.local:

DIM           = 2
DEBUG         = TRUE
OPT           = TRUE
PRECISION     = DOUBLE
PROFILE       = FALSE
CXX           = g++
FC            = gfortran
MPI           = FALSE
USE_HDF       = FALSE

make lib make test make run

Looks like the tests pass BUT uses -m32 during compilation and so a 32bit executable results. This is fine here (since x86_64 will also handle 32bit exes), but it causes a problem when we try and link against 64bit dependencies, such as HDF5 (see below for remedy).

With HDF5 compiled in serial mode

USE_64 = TRUE

adds -m64 to the GNU compiler flags and so object code is created for the x86_64 architecture, and so it will link against the HDF libraries which are built for x86_64. This is not what the documentation says the makefile variable is for! Hey ho. Add it in.

• e.g. export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/gsb6/ggdagw/hdf5/hdf5-1.4.4/serial/lib

required to find shared object files, else tests will not run. Tests look OK for both v1.4.4 and v1.6.7 of HDF.

With MPI and parallel HDF5

Add in to Make.defs.local:

MPICXX        = /gsb6/ggdagw/mpich/mpich-1.2.6/ch_shmem/bin/mpiCC
HDFMPIINCFLAGS= -I/gsb6/ggdagw/hdf5/hdf5-1.4.4/parallel/include
HDFMPILIBFLAGS= -L/gsb6/ggdagw/hdf5/hdf5-1.4.4/parallel/lib -lhdf5 -lz

cd chombo/lib
make lib
make test
make run

Tests look to pass OK for both HDF5v1.4.4 and v1.6.7.

Running the AMRUpwind example from Gethin's build

• Ensure that /gsb6/ggdagw/mpich/mpich-1.2.6/ch_shmem/bin is in your path
• copy ~ggdagw/chombo/Chombo-2.0/example to your location of choice
• cd AMRUpwind
• edit:
  • GNUmakefile such that: CHOMBO_HOME = ../../chombo/Chombo-2.0/lib
  • exec/GNUmakefile such that: CHOMBO_HOME = ../../../chombo/Chombo-2.0/lib
• make clean
• make example
• make run

Useful informatiomn on files within example

After compilation, you will find 4 directories created within the example directory. Each contains subdirectories based on the particular build (e.g., 2d.Linux.64.mpiCC.gfortran.DEBUG.OPT.MPI/). These are:

• d which contains paths for any files used in the compilation; most usefully .H etc files.
• f which contains fortan code after it has been parsed; particularly useful in the case of ChF files (.f).
• o which contains usual .o objects.
• p which also contains parsed versions of any fortran code (.cpre is same as .f above).

Changes to Makefiles for Fortran90

In Chombo-2.0/lib/mk/Make.example:

src_extensions := .cpp .ChF .F .f .c .f90
vpath %.cpp $(src_dirs)
vpath %.ChF $(src_dirs)
vpath %.F   $(src_dirs)
vpath %.f   $(src_dirs)
vpath %.c   $(src_dirs)
vpath %.f90 $(src_dirs)

In Chombo-2.0/lib/mk/Make.rules:

line 104:

src_wildcard := *.cpp *.ChF *.F *.f *.f90

add after line ~350:

o/$(config)/%.o : %.f90  d/$(config)/%.d
        $(QUIET)$(FC) $(FFLAGS) $(XTRAFFLAGS) $(fcompflag) $< $(fobjflag)$@

add after line ~413:

d/$(config)/%.d : %.f90 d/$(config)/.dummy
        @if [ x"$(QUIET)" != x"@" ]; then echo Depending $< ...; fi

Tested example/expHeat/parVersion by replacing heatfort.F with heatfort.f90.