Programming on Jacquard

Compiling and Linking Applications

The default and recommended compilers are produced by PathScale. The Fortran compiler is named pathf90 and the C/C++ compiler is pathcc. Documentation is available online and via man pages on jacquard.

MPI programs compiled with the Pathscale compilers can be debugged with the totalview debugger.

The default optimization level is -O2 for both pathf90 and pathcc. In addition, the default architecture and processor flags are already set to the correct values for the Opteron chip. A better level of optimization can be attained with the "-O3" compiler flag, which can cause small numerical differences in results. An even higher level of optimization can be attained with the "-Ofast" flag, which can cause even greater numerical differences. This option also includes interprocedural analysis, so this flag should be included at link as well as compile time. For a detailed description of Tuning Options see Chapter 7 of the User Guide.

Users should avoid use of the -m32 and -m64 options for specifying memory address size. The default on the machine is -m64, 64-bit addressing, to allow each task to access all available memory. Compiling and/or linking with these flags will result in problems.

Jacquard is "little endian," and this will affect ability to read binary data transferred from IBM SP systems. Use the assign command for Fortran, or byteswap functions for C and C++. Version 2.2 of the PathScale compilers will include compiler line options for endian conversion. The following use of the assign command and the FILENV environment variable will allow Fortran codes on Jacquard to read binary Fortran output files from IBM SP systems:

setenv FILENV .assign 
assign -F f77.mips -N be g:su
assign -I -F f77.mips -N be g:du 

Parallel Programming

To compile and link MPI codes, use the commands mpif90, mpicc, or mpicxx for Fortran, C, and C++, respectively.

The MPI implementation is known as MVAPICH (developed by the MPI over InfiniBand Project at The Ohio State University. It is based on an MPI known as MPICH. There are some run-time parameters that can be adjusted as described in the MVAPICH tuning guide.

OpenMP 2.0 directives are supported by the PathScale Fortran and C compilers if the -mp is used to compile and link. The OMP_NUM_THREADS environment variable should be set to 2 at run time. OpenMP only works on a single node, so you cannot effectively use more than 2 threads on this system. OpenMP is only partially supported for C++, see the Users Guide and Release Notes for more details.

Using the ACML Optimized Math Library

AMD provides a highly optimized library of BLAS, LAPACK and FFT routines, the AMD Core Math Library (ACML). This library is installed on Jacquard and can be used with both the PathScale and GNU compilers via modules.

To use the ACML library with the PathScale compilers, compile and link in the following way:

$ module load acml
$ mpif90 program.f90 $ACML
$ mpicc program.c $ACML
$ mpicxx program.C $ACML

The $ACML flag must be both a compile and link option.

Mixing Fortran and C/C++

The Pathscale Fortran compiler adds a single underscore to the name of external symbols in the default configuration on jacquard, whereas the C compiler leaves the name as-is. You may find references to Fortran "double underscoring" in the Pathscale manual. The default Jacquard configuration has been set to "no-double-underscore".

When linking Fortran-C/C++ mixed-language codes, use the C/C++ compiler (e.g. mpicxx) and include the Fortran library in the list of link libraries as in -lpathfortran.