NERSCPowering Scientific Discovery Since 1974

PGAS Language Codes (UPC, Coarray Fortran)

Introduction


 

PGAS (Partitioned Global Address Space) languages such as Unified Parallel C (UPC) and Coarray Fortran (CAF) are available with the Cray compilers.  To use them, swap to the Cray programming environment and compile as usual with the Cray wrappers ftn and cc.  See the examples below for command line usage.

module swap PrgEnv-pgi PrgEnv-cray

 

 

Using Large Pages

Due to a hardware requirement, SHMEM, UPC, and CAF applications on Cray XE systems that use more than 2 GB of memory per compute node for static data must back the static data with huge pages.  The 2 GB of static data per compute node is the sum of the static data used per core, multiplied by the number of cores used per compute node. It includes initialized data and BSS. When using static linking, it also includes all static library data. It does not include dynamically allocated memory. Additionally, Lustre uses a portion of this space on each node, so not all 2GB is available for user applications. This affects any language or application that uses DMAPP (which currently includes SHMEM, UPC and CAF).

It is advisable to map the static data section and private heap onto huge pages. The application should be linked with libhugetlbfs. 

-lhugetlbfs 

 

UPC

Unified Parallel C is a partitioned global address space (PGAS) language and an extension of the C programming language. It is designed for HPC systems and allows the programmer to view a single shared partitioned address space where each variable is associated with a single processor, but can be directly read and written by any processor. Although not a language specification, low overhead, one sided communication models are important for obtaining good performance results from UPC codes whose programming paradigm facilitates low level communication transactions. See the GASNet webpage for more information about the low level networking layer used in UPC implementations.

 


Programming Environments

UPC is available on Hopper using Cray C compiler through Cray programming environment software modules. First issue the following command:

hopper% module swap PrgEnv-pgi PrgEnv-cray

 

Then, compile a code with:

hopper% cc -h upc -o UPCProg UPCProg.c

The "-h upc" option is needed on the cc command line.

 

Batch Script

Use the aprun command to run jobs. This example uses 1 processor per node, with a total of 24 nodes.

#PBS -q debug
#PBS -l mppwidth=24
#PBS -l mppnppn=1
#PBS -l walltime=00:10:00
#PBS -j eo
#PBS -V

cd $PBS_O_WORKDIR
aprun -n 24 -N 1 ./UPCProg

 

Use the aprun command to run jobs. This example uses the default 24 processors per node, with a total of 1 node on Hopper.

#PBS -q debug
#PBS -l mppwidth=24
#PBS -l walltime=00:10:00
#PBS -j eo
#PBS -V

cd $PBS_O_WORKDIR
aprun -n 24 ./UPCProg

 

CAF

CAF is a parallel programming extension of the Fortran syntax that allows a programmer to view a single shared partitioned address space, just as UPC does. It is now part of the Fortran 2008 standard.

 


Programming Environments

CAF is available on Hopper using Cray compiler through Cray programming environment software modules. First issue the following command:

hopper% module swap PrgEnv-pgi PrgEnv-cray

Then, compile a code with:

hopper% ftn -h caf -o CAFProg CAFProg.f

The "-h caf" option is needed on the ftn command line.

 

Batch Script

Use the aprun command to run jobs. This example uses the default 24 processors per node, with a total of 1 node on Hopper.

#PBS -q debug
#PBS -l mppwidth=24
#PBS -l walltime=00:10:00
#PBS -j eo
#PBS -V

cd $PBS_O_WORKDIR
aprun -n 24 ./CAFProg