NERSCPowering Scientific Discovery Since 1974

Shared and Dynamic Libraries

The Edison system can support applications that use dynamic shared libraries (DSL) on the compute nodes.  Some "out-of-the-box" applications require DSLs and some popular applications like Python use DSLs as well.

Using System Shared and Dynamic Libraries

"System" DSLs include those that support software packages found in "typical" Linux distributions, e.g. Python and Perl. To build an application that will use system DSLs on Edison, you must do two things:

  • use the  -dynamic flag in the link step of your build
  • set the environment variable  CRAY_ROOTFS to have the value DSL in your job script or before you run

The -dynamic flag instructs the compiler wrappers (i.e., ftn, cc, CC)  to prepare a dynamically linked executable. 

edison01% ftn -dynamic -o dyn_example.x dyn_example.f90

Alternatively, you can set the environment variable XTPE_LINK_TYPE instead of using the compiler/linker flag.

Setting the environment variable CRAY_ROOTFS value of DSL tells the Cray runtime to make the system shared-object libraries available on the compute nodes.The syntax to set the CRAY_ROOTFS environment variable is

setenv CRAY_ROOTFS DSL #(for csh-like shells: csh, tcsh)
export CRAY_ROOTFS=DSL #(for sh-like shells: bash, sh, ksh)

An example batch script using the bash shell is

#!/bin/bash -l
#PBS -q debug
#PBS -l mppwidth=128
#PBS -l walltime=00:10:00
#PBS -N my_job
#PBS -j oe

cd $PBS_O_WORKDIR
module unload xt-shmem
export CRAY_ROOTFS=DSL

aprun -n 128 ./example.x

Notes:

Normally, the aprun command expects a (compiled) binary executable.  But you can also use aprun to launch a script on the compute nodes.  To execute a script on the compute nodes you need to add the option -a xt in your aprun command line. You would typically do this only if you want  your script to execute on a node or nodes dedicated to your job. An example of starting a python script on a compute node is

aprun -n 1 -a xt ./numpy_ex1.py

Using Your Own Shared and Dynamic Libraries

To build and use your own DSLs

  • compile and build libraries with the -shared -fPIC compiler options
  • link with the -dynamic flag

An example follows. First, compile several C source files and unload the xt-shmem module if you are not making calls to Cray SHMEM data-passing library routines:

edusin01% cc -shared -fPIC source1.c 
edison01% cc -shared -fPIC source2.c

Create a dynamic library, my_c_lib.so, from the C objects:

edison01% cc -fPIC -shared -o my_c_lib.so source1.o source2.o 

Link the dynamic library with a driver routine:

edisn01% cc -fPIC -dynamic -o my_dyn_code.x driver.c my_c_lib.so

You can also create a dynamic library from Fortran source.

edison01% ftn -dynamic -fPIC -o my_f_lib.so stuff.f /opt/cray/xt-asyncpe/3.4/bin/ftn: INFO: linux target is being used

To run this code:

  • Add a line to your batch script setting the LD_LIBRARY_PATH environment variable to the location of your shared objects. We highly recommend using a directory in $SCRATCH for performance reasons.
  • At runtime, make sure you have loaded the same modules you loaded to build the code.

Here's an example batch script.

#PBS -S /usr/bin/csh
#PBS -q debug
#PBS -l mppwidth=128
#PBS -l walltime=00:10:00
#PBS -N my_job
#PBS -j oe

cd $PBS_O_WORKDIR

setenv CRAY_ROOTFS DSL #If you want the system DSLs

setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:${SCRATCH}/MY_DIRECTORY

module load AAA #if AAA was used when building the executable

aprun -n 128 ./my_dyn_code.x

Performance Implications of Using Dynamic Shared Objects

Using dynamic libraries may introduce delays in application launch times. This delay is caused by the runtime linking process and the relative inefficiency of symbol lookup in distributed shared objects. Also,  there may be a small performance degradation during execution.

Other Notes

To verify that you have created a dynamically-linked executable use the file command:

edison01% file a.out.dyn  
a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
for GNU/Linux 2.6.4, dynamically linked (uses shared libs), not stripped

Two system utilities related to shared objects that you might want to know about are ldd and readelf. Use these to get information about the shared object that you've created.