We recommend using the Intel ifort, icc, and icpc (fortran, C, C++) compilers for production, because of their superior speed optimization. Compiling for MPI is done with mpiifort, mpiicc, mpiicpc (note the double ii), which are wrapper scripts around the compilers that link in the correct MPI libraries.
However, using gfortran for development and testing can be a very good idea, especially if / when problems arise, and are difficult to locate. Compiling the same code with different compilers gives a better chance to discover problems, or to exclude a compiler bug as the reason for a problem under investigation. Gfortran's MPI wrapper is (currently) called mpif90 (or mpifort in recent laptop versions; note the single i!).
The default setup loads these module commands to choose the Intel compiler and MPI library (see the "module" section for additional info) at login:
module load intel intelmpi
Even after this module command, you can use the mpif90 wrapper to compile with gfortran for MPI.
Recommended compiler options (applies to RAMSES, STAGGER, PP-CODE, and DISPATCH):
When compiling RAMSES or STAGGER the make macro OPTS may be used to choose bundled options for e.g. optimization or debugging. These option bundles are also recommended for ad hoc compilation:
make OPTS=full_debug options -- for comprehensive debugging, with significant impact on speed
ifort -g -traceback -fpe:0 -ftrapuv -check all -fp-stack-check -c ...
make OPTS=debug options -- for light-weight debugging, with only moderate impact on speed
ifort -g -traceback -fpe:0 -ftrapuv -fp-model source -fp-model except -check bounds,pointers,uninit -c ...
make OPTS= options (same as not specifying OPTS) -- default options that give good performance, without excessive compilation times
ifort -g -traceback -fpe:0 -O -prev-div -c ...
make OPTS=optimize options -- for maximum performance, with possibly long compile times
ifort -g -traceback -fpe:0 -O3 -axAVX -xSSE4.2 -prec-div -diag-disable remark -c ...
For quick reference on any of these options, do "ifort --help | less". For full details, do "man ifort", either on the command line, or in the search field of a browser. Specifically:
-fpe:0 ensures that the code stops if it encounters floating point errors
-prec-div avoids that floating point precision differs in vectorized and non-vectorized code (can give MPI-stripes)
-axAVX -xSSE4.2 optimizes for Ivy Bridge, if avaiable, otherwise for Westmere / Nehalem
Linking with SLURM for optimal performance and binding between MPI ranks and cores
This is handled automatically when using the default Intel + Intel MPI combination. With other MPI libraries it is up to the user.
Using GCC
If you wish to use GCC (Gfortran) then you may need a newer version than the one shipped with CentOS.
Type:
scl enable devtoolset-8 bash
to enable GCC 8.3. If you need a newer version than 8.3, you can try to contact the hpc-center and ask what options they have.
