NERSCPowering Scientific Discovery Since 1974

Shared and Dynamic Libraries

The Hopper system can support applications that use dynamic shared libraries (DSL) on the compute nodes.  Some "out-of-the-box" applications require DSLs and some popular applications like Python use DSLs as well.

Using System Shared and Dynamic Libraries

"System" DSLs include those that support software packages found in "typical" Linux distributions, e.g. Python and Perl. To build an application that will use system DSLs on Hopper, you must do three things:

  • use the  -dynamic flag in the link step of your build
  • set the environment variable  CRAY_ROOTFS to have the value DSL in your job script or before you run
  • disable the cray-shmem module with command module unload cray-shem (unless you are making calls to the Cray SHMEM data-passing library)

The -dynamic flag instructs the compiler wrappers (i.e., ftn, cc, CC)  to prepare a dynamically linked executable. 

If your code does not use the Cray SHMEM data-passing library explicitly, and you do not unload the cray-shem module at both the compile time and run time you may receive a run-time error message such as (dmapp_dreg.c:391: _dmappi_dreg_register: Assertion `reg_cache_initialized' failed").

hopper% module unload cray-shmem           #if your code does not make Cray shmem library calls
hopper% ftn -dynamic -o dyn_example.x dyn_example.f90

Alternatively, you can set the environment variable XTPE_LINK_TYPE instead of using the compiler/linker flag.

Setting the environment variable CRAY_ROOTFS value of DSL tells the Cray runtime to make the system shared-object libraries available on the compute nodes.The syntax to set the CRAY_ROOTFS environment variable is

setenv CRAY_ROOTFS DSL #(for csh-like shells: csh, tcsh)
export CRAY_ROOTFS=DSL #(for sh-like shells: bash, sh, ksh)

An example batch script using the bash shell is

#!/bin/bash -l
#PBS -q debug
#PBS -l mppwidth=128
#PBS -l walltime=00:10:00
#PBS -N my_job
#PBS -j oe

cd $PBS_O_WORKDIR
module unload cray-shmem
export CRAY_ROOTFS=DSL

aprun -n 128 ./example.x

Notes:

Normally, the aprun command expects a (compiled) binary executable.  But you can also use aprun to launch a script on the compute nodes.  To execute a script on the compute nodes you need to add the option -a xt in your aprun command line. You would typically do this only if you want  your script to execute on a node or nodes dedicated to your job. An example of starting a python script on a compute node is

aprun -n 1 -a xt ./numpy_ex1.py

Using Your Own Shared and Dynamic Libraries

To build and use your own DSLs

  • compile and build libraries with the -shared -fPIC compiler options
  • link with the -dynamic flag

An example follows. First, compile several C source files and unload the cray-shmem module if you are not making calls to Cray SHMEM data-passing library routines:

hopper% module unload cray-shmem     
hopper% cc -shared -fPIC source1.c
hopper% cc -shared -fPIC source2.c

Create a dynamic library, my_c_lib.so, from the C objects:

hopper% cc -fPIC -shared -o my_c_lib.so source1.o source2.o 

Link the dynamic library with a driver routine:

hopper% cc -fPIC -dynamic -o my_dyn_code.x driver.c my_c_lib.so

You can also create a dynamic library from Fortran source.

hopper% ftn -dynamic -fPIC -o my_f_lib.so stuff.f /opt/cray/xt-asyncpe/5.23/bin/ftn: INFO: linux target is being used

To run this code:

  • Add a line to your batch script setting the LD_LIBRARY_PATH environment variable to the location of your shared objects. We highly recommend using a directory in $SCRATCH or $SCRATCH2 for performance reasons.
  • At runtime, make sure you have loaded the same modules you loaded to build the code.

Here's an example batch script.

#PBS -S /usr/bin/csh
#PBS -q debug
#PBS -l mppwidth=128
#PBS -l walltime=00:10:00
#PBS -N my_job
#PBS -j oe

cd $PBS_O_WORKDIR

setenv CRAY_ROOTFS DSL #If you want the system DSLs

setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:${SCRATCH}/MY_DIRECTORY

module load AAA #if AAA was used when building the executable

aprun -n 128 ./my_dyn_code.x

Use Shared Libraries for Applications Using Stack Memory > 2GB (on Hopper only

Some applications have large stack memory usage (> 2GB) typically with big data arrays declared within common blocks. Compiling these applications statically with ftn will get "relocation truncated..." error messages similar to:

relocation truncated to fit: R_X86_64_PC32 against symbol `__hpf_quiet' defined in COMMON section in /opt/pgi/11.2.0/linux86-64/11.2/lib/libpgf90.a(initpar.o)

A workaround is to compile the application with the "-mcmodel=medium" option but on Hopper it is required that you also compile dynamically with "-dynamic" option. The generated executable will be dynamic, and is only able to run on Hopper with the environment variable "CRAY_ROOTFS" set to DSL before the aprun command.

Performance Implications of Using Dynamic Shared Objects

Using dynamic libraries may introduce delays in application launch times. This delay is caused by the runtime linking process and the relative inefficiency of symbol lookup in distributed shared objects. Also,  there may be a small performance degradation during execution.

Other Notes

Shared libraries must not be linked with static library archives (*.a files). When creating a shared library, you can only depend on other shared libraries for resolving external references. If you need to reference a routine that resides in an archive library (i.e., somethinglib.a), you need to access the archive, extract that routine, and put the routine in a separate shared library, include it in the shared library being created, or load it as a relocatable object.

To verify that you have created a dynamically-linked executable use the file command:

hopper% file a.out.dyn  
a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
for GNU/Linux 2.6.4, dynamically linked (uses shared libs), not stripped

Two system utilities related to shared objects that you might want to know about are ldd and readelf. Use these to get information about the shared object that you've created.