NERSCPowering Scientific Discovery Since 1974

Coarray Fortran

Description

Coarray Fortran (CAF) is a parallel programming extension of the Fortran syntax that allows a programmer to view a single shared partitioned address space.

CAF is supported by Cray and Intel compilers on Edison.

Using CAF with Cray Compilers

How to compile

Edison default programming environment is Intel, so you need to swap to Cray programming environment by issuing the following module command:

% module swap PrgEnv-intel PrgEnv-cray

Then compile the CAF codes with the fortran compiler wrapper, ftn:

% ftn -o CAFProg CAFProg.f

Cray compiler enables the Coarray syntax by default, so the "-h caf" option is not explicitly needed on the compilation line.

How to run

Here is a smaple job script to run CAF codes that are compiled with Cray compilers on Edison:

#PBS -q regular
#PBS -l mppwidth=96
#PBS -l walltime=06:00:00
#PBS -N my_job
#PBS -j oe

cd $PBS_O_WORKDIR

aprun -n 96 ./CAFProg

Using CAF with Intel compilers

CAF codes compiled with Intel compilers run only under the Cluster Compatibility Mode (CCM).

How to compile

CAF can run both in shared memory (on a single node) and distributed memory (across multiple nodes) modes on Edison.

To compile a CAF code to run in the distributed memory mode (across), do

% ifort -coarray=distributed -coarray-config-file=~/cafconfig.txt -o CAFProg CAFProg.f

Where the "-coarray" option enables the coarray syntax. The ~/cafconfig.txt file defines the mpirun (from Intel MPI) command line options, and only read in at run time. So you can modify the configuration file after the compilation. Here is a sample ~/cafconfig.txt file:

-genvall -genv I_MPI_FABRICS=shm:tcp -machinefile ./nodefile -perhost 24 -n 96 ./CAFProg 

Where the -genvall passes all the environment variables from the head node to the remote nodes, the -genv I_MPI_FABRICS=shm:tcp  option selects the fabric(s) to use in priority order:  use shared memory within a node, TCP for remote. The -machinefile ./hostsfile  option provide the list of compute nodes on which to run, and the -n 96 denotes to launch 96 CAF images. The ./CAFProg is the name of the Coarray Fortran program.

How to run

#PBS -q ccm_queue
#PBS -l mppwidth=96
#PBS -l walltime=06:00:00
#PBS -N my_job
#PBS -j oe

cd $PBS_O_WORKDIR
module load ccm
module load impi

ccmrun ./CAFProg

Documentation

For questions about using CAF at NERSC contact the consultants at consult@nersc.gov.