Porting Programs from the Crays to the SP

This document outlines the main differences between the applications programming environment on the IBM SP and the applications programming environments on the Cray T3E and SV1.

• Fortran
• Fortran Datatypes
• Fortran System Interface and Language Extension Libraries
• Fortran Input/Output
• C/C++
• C/C++ Datatypes
• OpenMP
• MPI
• MPI Datatypes
• MPI Error Handler
• Mathematical Libraries

Fortran

For a complete comparison of Cray and SP Fortran see Fortran Compiler Option Correspondences: Cray f90 and IBM xlf90. The main differences are noted below:

• On the Crays the Fortran compiler is invoked with the command "f90". The SP has several ways to invoke the compiler, depending on the type of behavior you want. See IBM XL Fortran Compiler: Introduction for details.
• The Cray compiler accepts files with suffixes ".f", ".f90", ".F", and ".F90". The SP compiler accepts only ".f", and ".F" by default, although it is easy to change this behavior with the -qsuffix option. See the SP xlf man page and IBM XL Fortran Compiler: Source Files and Preprocessing for more details.
• The Cray compiler defaults to a medium level of optimization, -O2, whereas the SP compiler defaults to no optimization, -noopt. We recommend either "-O3 -qstrict -qtune=pwr3 -qarch=pwr3" or "-O4". See IBM XL Fortran Compiler: Optimization for more details.
• The Cray compiler defaults to allowing a program to use a process's entire memory limit. The SP defaults to 32Kb for stack variables and 128 Mb for heap variables. See Memory Management on the SP for more details. To remove this limitation, use "-bmaxstack:0x10000000 -bmaxdata:0x80000000" as both compile and link options.

Fortran Datatypes

The default sizes and available datatypes of the SP and the two Cray architectures are different:

The SP XL Fortran compiler flag "-qrealsize=8" can be used to promote all default reals, and real constants, to 8 bytes. Integers and logicals can be similarly promoted using "-qintsize=8".

A portable alternative is to use the Fortran 90 KIND syntax.

Fortran System Interface and Language Extension Libraries

These are routines which extend the Fortran language in some way to interact with the operating system, and which have approximately equivalent versions on the Crays and the SP.

Check the documentation for both routines when porting. For the SP, see the "Classes of Intrinsic Procedures" and "Service and Utility Procedures" sections of the XL Fortran Language Reference manul on the NERSC IBM/SP Documentation Web Server. For some of the routines, the SP has identically named C-callable routines for which there are man pages. These man pages do not reflect correct Fortran use. For the T3E, see the appropriate man page.

On the SP, if you call a procedure not listed above (obviously excepting routines where the source library is clear, e.g. MPI, LAPACK, etc.) you may see unexpected behavior. For example, making a call to exit instead of exit_, will terminate your program, destroying all Fortran-buffered output without flushing it to a file.

Fortran Input/Output

Sequential Unformatted Files

With Cray's Flexible Format I/O (FFIO) it is possible for a Cray Fortran program to write out binary data to a sequential unformatted file readable on other architectures by means of the ASSIGN command. This conversion is done at run time and requires no source code modification. See the assign and intro_ffio man pages on the Cray systems and the Cray Application Programmer's I/O Guide for more information. To make a Fortran program on a Cray system create a sequential unformatted file readable by a Fortran program on the SP, use these assign arguments on both the T3E and the SV1 to change the run time environment of the program where unit is the Fortran unit number of the file to which you are writing the data:

assign -R 
assign -F f77 -N ieee_dp u:unit

The ieee_dp option will convert real or real*8 Cray data to real*8 SP data and integer*8 Cray data to integer*4 SP data.

The ieee option will convert real or real*8 Cray data to real*4 SP data and integer*8 Cray data to integer*4 SP data.

Direct Access Unformatted and MPI I/O Files

Direct access unformatted files and MPI I/O files can be freely transferred between the T3E and the SP because both systems have IEEE data representation. Direct access unformatted binary files and MPI I/O files from the SV1 are unreadable on the T3E and SP since they have a Cray data representation.

Namelist Input/Output

Users familar with the "assign -f77 u:<unit number>" on the T3E, which causes an old-style namelist input to be written or read, can set the following environment variable on the SP to obtain the same effect:

setenv XLFRTEOPTS="namelist=old"

C/C++

As with Fortran, the SP has several ways to invoke the compiler, depending on the type of behavior you want. These are the main features of the SP C/C++ environments:

• The SP C compiler is an ANSI standard C compiler described in the xlc and mpcc man pages. The "C for AIX" manual is available on the NERSC IBM/SP Documentation Web Server.
• IBM's fully ANSI/ISO C++ Standard compliant VisualAge C++ compiler is described in the "IBM Visual Age C++ Batch Compiler" manual on the NERSC IBM/SP Documentation Web Server.

C/C++ Datatypes

These are the default sizes of the standard C/C++ datatypes on the SP and the two Cray architectures:

OpenMP

OpenMP is supported on the SP with Fortran and C, but not with the Visual Age C++ compiler. There is a guide to using OpenMP on the SP at Using OpenMP on seaborg.

These are the main differences between the SP and the SV1 implementations of OpenMP:

• You must compile your code with the "-qsmp=omp" option to implement OpenMP directives in Fortran or C routines.
• OpenMP programs will run by default with a number of threads equal to the number of processors on a node, 16 in the case of seaborg.
• You can set the number of threads used by an OpenMP program at run time by setting the environment variable OMP_NUM_THREADS to the desired value.
• The NCPUS environment variable has no effect on the SP.

MPI

For details about the SP implementation of MPI, see the "MPI Programming Guide" and the "MPI Subroutine Reference" on the NERSC IBM/SP Documentation Web Server. These are the main differences between the SP and Cray implementation of MPI:

• Unlike the Cray compilers, the SP compilers know about the syntax of MPI calls, so there is significantly more argument checking on the SP. For example, there are programs that compile on the Crays but not on the SP, when an argument to an MPI call is of the wrong type.
• The SP provides full support for the MPI-2 I/O standard. Only a subset of this standard was supported on the Crays. To link with MPI I/O on the SP, you must use one of the "mpxlf*_r" compilers, e.g. mpxlf90_r.

MPI

The default sizes of MPI datatypes are different on the different machines:

If you use IBM XL Fortran compiler flags to increase the precision of default reals, this does not change the interpretation of MPI_REAL.

MPI defines additional datatypes with explicit sizes. These can be used to provide some portability:

MPI Error Handler

In some cases, error conditions are flagged on the SP and the program halted without identifying where the error occurs. In addition, no corefile is produced to help debugging. This can make it extremely difficult to find the routine which is causing the problem.

This behavior can be improved by installing a different MPI error handler. For example, in the main Fortran program:

program p2p
implicit none
include 'mpif.h'
external myhandler

integer ierrhandler, ierror

call MPI_INIT(ierr)

call mpi_errhandler_create(myhandler, ierrhandler, ierror)
call mpi_errhandler_set(MPI_COMM_WORLD, ierrhandler, ierror)
....

requests that the function myhandler be called on any MPI error. A simple implementation is shown below, myhandler.c:

#include <sys/signal.h>
#include <mpi.h>

void myhandler(MPI_Comm *comm, int *code, char *routine_name, int *flag, int *badval) {

  raise(SIGABRT);

}

Using this handler will cause a core file to be created along with a traceback pointing to the correct program line when an MPI error is encountered.

Mathematical Libraries

BLAS and LAPACK

BLAS levels 1-3, and major parts of the LAPACK library are available on the SP in the Engineering and Scientific Subroutine Library (ESSL). This library is not loaded by default when a program is linked, so it is necessary to include "-lessl" at link time to use these routines. This library is availble for Fortran, C, C++, and in the 64 bit environment. See the "Engineering and Scientific Subroutine Library (ESSL) Documentation" on the NERSC IBM/SP Documentation Web Server for details. In addition to the serial versions of these routines contained in the ESSL library, there is an SMP ESSL library which has multi-threaded versions of many of the routines. See the section "Which Library Do You Want to Use?" in the "Engineering and Scientific Subroutine Library (ESSL) Documentation" on the NERSC IBM/SP Documentation Web Server for more information. These are some signficant issues you may encounter when moving programs from the Crays to the SP:

• On the Crays, real*8 variables use BLAS calls beginning with "S", on the SP real*8 variables use BLAS calls beginning with "D", e.g.

  On the Crays:

        real*8 a(n), b(n), x
        ....
        x = sdot(n,a,1,b,1)
  

  On the SP:

        real*8 a(n), b(n), x
        ....
        x = ddot(n,a,1,b,1)
  

• On the SP a small number of routines are not compatible with Cray Scientific Library or LAPACK. For example, SGEEV computes eigenvalues and eigenvectors of a real general matrix. On the T3E, the Scientific library follows the LAPACK convention with the first two arguments being character variables 'N' or 'V'. On the SP, the ESSL routine DGEEV takes a different set of arguments, but does essentially the same job.

  Routines which are incompatible (where x may be C, D, S, or Z):

  xGEEV
  xSPEV
  xSPSV
  xHPEV
  xHPSV
  xGEGV
  xSYGV
  

  for these routines, we suggest you use the public domain LAPACK library on the SP. Link this before ESSL:

  module load lapack
  xlf90 -o a.out b.o $LAPACK -lessl
  

  LAPACK documentation can be found at LAPACK -- Linear Algebra PACKage

BLACS and PBLAS

The SP supports all of BLACS (Fortran style binding only) and levels 2 and 3 of PBLAS in the PESSL library. See The Parallel Engineering and Scientific Subroutine Library (PESSL) for more details on IBM PESSL. These routines are, in general, compatible with those on the T3E. These are some signficant issues you may encounter when moving programs from the T3E to the SP:

• On the T3E, Fortran real*8 variables use PBLAS calls beginning with "PS", on the SP Fortran real*8 variables use PBLAS calls beginning with "PD".
• A call to BLACS_GET, followed by either BLACS_GRIDINIT or BLACS_GRIDMAP is required on the SP. On the T3E, only a call to either BLACS_GRIDINIT or BLACS_GRIDMAP is required.

ScaLAPACK

IBM PESSL also supports ScaLAPACK 1.5 style calls, providing a subset of the Linear Equation Solvers and Eigensystem Analysis routines. The argument lists are unchanged from standard netlib ScaLAPACK. See The Parallel Engineering and Scientific Subroutine Library (PESSL) for more details on IBM PESSL.

Just as with BLACS and PBLAS, a call to BLACS_GET, followed by either BLACS_GRIDINIT or BLACS_GRIDMAP is required on the SP.

In addition, the public domain ScaLAPACK distribution is installed on both machines. See ScaLAPACK Home Page for more details.