NERSCPowering Scientific Discovery Since 1974

CREST HPX on Edison and Cori

High Performance ParalleX on NERSC Edison and Cori


CREST HPX or HPX-5 is the High Performance ParalleX runtime library from the Center for Research in Extreme Scale Technology (CREST) at Indiana University for petascale/exascale computing. Building on CREST’s commitment to developing new approaches for achieving the highest levels of performance on current and next-generation supercomputing platforms, CREST HPX is provided to support the international high-performance computing community in addressing significant challenges involved in achieving exascale computing.

CREST HPX is a reduction to practice of the revolutionary ParalleX execution model, which establishes roles and responsibilities between layers in an exascale system and supports dynamic and adaptive resource management and scheduling. It is implemented in portable C99 and is organized around a cooperative lightweight thread scheduler, a global address space, an active-message parcel transport, and a group of globally addressable local synchronization object classes. Internally, the infrastructure is built on scalable concurrent data structures to minimize shared-memory synchronization overhead. The global address space and parcel transport are based on the innovative Photon network transport library, which supports low-level access to network hardware and provides RDMA with remote completion events for low overhead signaling. An alternative ISend/IRecv network layer is included for portability, along with a reference MPI implementation. CREST HPX is compatible with Linux running on Intel x86, MAC OSX and Xeon Phi processors and various ARM core platforms (including both ARMv7 and ARMv8/Aarch64).


CREST HPX is used for a broad range of scientific applications, helping scientists and developers write code that shows better performance on irregular applications and at scale when compared to more conventional programming models such as MPI. For the application developer, it provides dynamic adaptive resource management and task scheduling to reach otherwise unachievable efficiencies in time and energy and scalability.

Getting started on NERSC Machines

Edison, Cori and Babbage have a testbed version of CREST HPX installed. Contact Alice Koniges ( for information.

We illustrate the Edison and Cori builds.

Accessing NERSC Machines
Access Edison using
ssh -Y
Access Cori using
ssh -Y
If you plan to download CREST HPX, we suggest starting with the latest released version, which can be downloaded from

For Example:

tar -xzf hpx-2.2.0.tar.gz

Install autotools using the script provided in hpx-2.2.0.tar.gz using the following commands:

For Example:

./hpx-2.2.0/scripts/ $HOME/hpx-2.2.0/hpx/autotools
export PATH=$HOME/hpx-2.2.0/hpx/autotools/bin:$PATH

Load the modules

On Edison,

module unload darshan
module swap PrgEnv-intel PrgEnv-gnu
module load intel
module unload craype-hugepages8M
module load craype-hugepages4M

export CRAYPE_LINK_TYPE=dynamic
export HPX_THREADS=24

On Cori,

module swap PrgEnv-intel PrgEnv-gnu
module load intel
module load craype-hugepages4M

export CRAYPE_LINK_TYPE=dynamic
export HPX_THREADS=32

Configuring CREST HPX Build

The CREST HPX configuration will pull a number of dependencies from your environment. See ./configure --help for a complete list of the variables that are used.

On Edison,

Set the TESTS_CMD with launcher command to be used. For example:

export TESTS_CMD="/usr/bin/timeout 20m srun -N 2 -n 2 -c 24"

Configure CREST HPX with TBBMalloc allocator.

IMPORTANT: In the ./configure command, replace BDIR with the full path to where you want to install. Move to the hpx folder in the directory you have just untarred, and run the ./configure .

For example, you can set this path with an environment variable as below.

export BDIR=$HOME/HPX5
./configure --prefix=$BDIR CC=cc CXX=CC --with-tbbroot="$INTEL_PATH.$INTEL_MINOR_VERSION/tbb" --with-tbbarch=intel64 --disable-jemalloc --enable-tbbmalloc CFLAGS="-O3 -g" CXXFLAGS="-O3 -g" --enable-debug --enable-instrumentation --enable-hugetlbfs --enable-mpi --enable-pmi --enable-photon PHOTON_CARGS="--enable-ugni --disable-libfabric" --enable-agas --enable-hpx++ --enable-parallel-config

 On Cori,

Set the TESTS_CMD with launcher command to be used. For example:

export TESTS_CMD="/usr/bin/timeout 20m srun -N 2 -n 2 -c 32"
./configure --prefix=$BDIR CC=cc CXX=CC --with-libffi=contrib --with-tbbroot="$INTEL_PATH/linux/tbb" --with-tbbarch=intel64 --disable-jemalloc --enable-tbbmalloc CFLAGS="-O3 -g" CXXFLAGS="-O3 -g" --enable-debug --enable-instrumentation --enable-hugetlbfs --enable-mpi --enable-pmi --enable-photon PHOTON_CARGS="--enable-ugni --disable-libfabric" --enable-agas --enable-hpx++ --enable-parallel-config

 Configure CREST HPX with TBBMalloc allocator. Set BDIR to path to install.

./configure --prefix=$BDIR --host=x86_64-k1om-linux CCASFLAGS="-mmic" LDFLAGS="-mmic" CC=mpiicc CXX=mpiicpc --with-tbbarch=mic --disable-jemalloc --enable-tbbmalloc CFLAGS="-mmic -O3" CXXFLAGS="-mmic -O3" --enable-mpi --enable-parallel-config

Complete the build and install CREST HPX

make -j 8
make install

Run the regression tests 

On Edison and Cori
salloc -p debug -N 2 -t 00:30:00 --job-name=hpx_regression
make check

Assuming all tests passed, go to examples folder and run hello using the command

srun -n 2 -N 2 -c 24 ./hello
Hello World from 0.
There are example included in examples folder which can be run using the launcher command.

Linking the CREST HPX as an external library

The previous commands will build CREST HPX and install it into the given location. The external application build system relies on the pkg-config utility to specify the needed HPX-5 compilation and linking options, so it is important to add the correct path for HPX-5 to your PKG_CONFIG_PATH. While one is at it, it is useful to modify the following environment variables to point to the newly installed CREST HPX. For example, with bash:

export PATH=$BDIR/bin:$PATH

 and then you can get the proper LIBS and CFLAGS values by doing the following:

pkg-config --libs hpx
pkg-config --cflags hpx
Simple hello program
#include <stdio.h>
#include <hpx/hpx.h>

static HPX_ACTION_DECL(_hello);
static int _hello_action(void) {
printf("Hello World from %u.\n", hpx_get_my_rank());
static HPX_ACTION(HPX_DEFAULT, 0, _hello, _hello_action);

int main(int argc, char *argv[argc]) {
if (hpx_init(&argc, &argv) != 0)
return -1;
int e = hpx_run(&_hello);
return e;

 For example, to compile hello.c using CREST HPX, you could do:

cc hello.c -o hello `pkg-config --cflags hpx` `pkg-config --libs hpx`

Inside of a Makefile you can get the same values by doing the following:

LIBS = $(shell pkg-config --libs hpx)
CFLAGS = $(shell pkg-config --cflags hpx)

Note also that even when built with AGAS, CREST HPX defaults to PGAS at present so --hpx-gas=agas must be used at runtime to specify the use of AGAS. AGAS works only with IS/IR, so the runtime option --hpx-network=isir must be used as well.

Controlling CREST HPX Behavior at Run-time

There are a variety of options that a user may want to control not just at build-time but also at run-time for the CREST HPX runtime. Such options include, for example, the global memory heap size, which network interconnect to use, and whether to turn on logging. There are two ways to control these options: through environment variables and command line options.

To control the options through command line parameters, just somewhere to your command line (after the name of the CREST HPX executable) the option. For example, the command line to control the network transport layer is --hpx-transport. On a machine using srun as the job launcher, setting this option to use the Isend/Irecv network backend instead of the builtin Photon backend might look like this:

To run hello.c on Edison,

/usr/bin/timeout 20m srun -N 1 -n 1 -c 24 ./hello --hpx-transport=mpi --hpx-network=isir

To control options through the environment variables, simply set the environment variable that corresponds to the available command line option. The naming convention for environment variables is that they match the name of the command line parameters except that they are all upper-case and all hyphens are converted to underscores (so the command line parameter --hpx-transport is named HPX_TRANSPORT as an environment variable). If you’re using bash as your shell, the above command would be instead:

export HPX_TRANSPORT=mpi
/usr/bin/timeout 20m srun -N 1 -N 1 -C 24 ./hello
Run-time options

CREST HPX ships with an extensive set of runtime options. All applications that link against CREST HPX can use the --hpx-help option to get a list of these. In this section we will discuss some specific options that are important for most programs.

General runtime options

The parameters users will most often be interested in that need some explanation are --hpx-network=<network> and --hpx-gas=<model>.

CREST HPX supports two different network types that have different performance characteristics. They rely on two separate transport implementations. PWC (“put with completion”) is the preferred network and usually performs better. It uses the Photon transport layer (included as part of the CREST HPX distribution) and communicates via RDMA for better performance. The Isend/Irecv network uses MPI two-sided communication and is included for portability.

 To use one of the networks, it must have been enabled when CREST HPX was built. Selecting a network automatically uses the corresponding transport method.

--hpx-gas=<model> can be used to specify the memory model the runtime will use. pgas is the default unless CREST HPX was built in SMP mode or is being run in a non-distributed environment. smp will force CREST HPX to use the SMP model. agas will force CREST HPX to use AGAS. In order to use AGAS, the runtime must have been built with --enable-agas and the Isend/IRecv network must be used (--hpx-network=isir) as AGAS is not compatible with the PWC network at this time.