Compiling Codes
Overview
There are three compiler suites available on Carver: Portland Group (PGI), Intel, and GCC. The PGI compilers are the default, to provide compatibility with other NERSC platforms. Because Carver uses Intel processors, some benchmarks have shown better performance when compiled with the Intel compilers. The GCC compilers are available primarily to facilitate building open-source tools, although they can also be used for scientific applications.
MPI
The only supported MPI implementation on Carver is Open MPI, which is descended from LAM. In particular, note that Open MPI is not part of the MPICH family of MPI implementations.
For each supported compiler suite, NERSC provides a version of Open MPI that is compatible with that compiler. The default is PGI. In order to use other compilers, it is necessary to swap both the compiler module and the MPI module. For example, to use the Intel compilers with Open MPI:
carver% module swap pgi intel
carver% module swap openmpi openmpi-intel
To use the GCC compilers with Open MPI:
carver% module swap pgi gcc
carver% module swap openmpi openmpi-gcc
The above swap commands may be required in your batch scripts as well, if you plan to submit calculations in future sessions based on executables compiled with intel or gcc.
Compiler Names
Compiler "wrappers" provided by Open MPI supply the correct compiler and linker flags for MPI applications. When compiling non-MPI programs (that is, either serial or shared-memory parallel applications), the "native" compilers may be used directly.
| Language | Open MPI | Native PGI | Native Intel | Native GCC |
| Fortran | mpif77, mpif90 | pgf77, pgf90, pghpf | ifort | gfortran |
| C | mpicc | pgcc | icc | gcc |
| C++ | mpiCC, mpic++, mpicxx | pgCC | icpc | g++, c++ |
Basic Examples
MPI
carver% mpif90 -fast -o example.x example.f90
OpenMP
carver% pgf90 -fast -mp -o example.x example.f90
Compiler Options
Open MPI defines a single option for the compiler wrappers:
carver% mpif90 -showme ...
The "showme" option shows the command line that would be executed, without actually invoking the underlying compiler.
All remaining compiler options depend on the underlying native compiler; complete deccriptions are available via the "man" command. Some common options are summarized below.
| PGI | Intel | GCC | Explanation |
|---|---|---|---|
| -fast | -O3 | -O3 | Produce high level of optimization |
| -mp | -openmp | -fopenmp | Activate OpenMP directives and pragmas in the code |
| -byteswapio | -convert big_endian | -fconvert=swap | Read and write Fortran unformatted data files as big-endian |
| -Mfixed | -fixed | -ffixed-form | Process Fortran source using fixed form specifications. |
| -Mfree | -free | -ffree-form | Process Fortran source using free form specifications. |
| -V | -V | --version | Show version number of the compiler. |
| not implemented | -zero | -finit-local-zero | Zero fill all uninitialized variables. |
| -mcmodel=medium | -mcmodel=medium | -mcmodel=medium | Allow data sections greate than 2GB |
Based on vendor recommendations and our own experiences with these compilers, we recommend these options to generate fast executables:
PGI: -fast
Intel: the compiler's default options, i.e. no explicit optimization options, gives a very high level of optimization
GCC: -O3 -ffast-math
Compiler Comparisons
Add the performance tables here.
