NERSCPowering Scientific Discovery Since 1974

Shared and Dynamic Libraries

The Edison system can support applications that use dynamic shared libraries (DSL) on the compute nodes.  Some "out-of-the-box" applications require DSLs and some popular applications like Python use DSLs as well.

Using System Shared and Dynamic Libraries

"System" DSLs include those that support software packages found in "typical" Linux distributions, e.g. Python and Perl. To build an application that will use system DSLs on Edison, you must do two things:

  • use the  -dynamic flag in the link step of your build

The -dynamic flag instructs the compiler wrappers (i.e., ftn, cc, CC)  to prepare a dynamically linked executable. 

edison01% ftn -dynamic -o dyn_example.x dyn_example.f90

There used to be an environment variable XTPE_LINK_TYPE that could be used instead of using the compiler/linker flag; however, Cray has changed this environment variable to CRAYPE_LINK_TYPE.  (The older environment variable is still valid on Hopper.)

An example batch script using the bash shell is

#!/bin/bash -l
#PBS -q debug
#PBS -l mppwidth=128
#PBS -l walltime=00:10:00
#PBS -N my_job
#PBS -j oe

cd $PBS_O_WORKDIR
module unload xt-shmem
# export CRAY_ROOTFS=DSL (not needed any more since it has been set to system default)

aprun -n 128 ./example.x

Notes:

Normally, the aprun command expects a (compiled) binary executable.  But you can also use aprun to launch a script on the compute nodes.  To execute a script on the compute nodes you need to add the option -a xt in your aprun command line. You would typically do this only if you want  your script to execute on a node or nodes dedicated to your job. An example of starting a python script on a compute node is

aprun -n 1 -a xt ./numpy_ex1.py

Using Your Own Shared and Dynamic Libraries

To build and use your own DSLs

  • compile and build libraries with the -shared -fPIC compiler options
  • link with the -dynamic flag

An example follows. First, compile several C source files and unload the xt-shmem module if you are not making calls to Cray SHMEM data-passing library routines:

edusin01% cc -shared -fPIC source1.c 
edison01% cc -shared -fPIC source2.c

Create a dynamic library, my_c_lib.so, from the C objects:

edison01% cc -fPIC -shared -o my_c_lib.so source1.o source2.o 

Link the dynamic library with a driver routine:

edisn01% cc -fPIC -dynamic -o my_dyn_code.x driver.c my_c_lib.so

You can also create a dynamic library from Fortran source.

edison01% ftn -dynamic -fPIC -o my_f_lib.so stuff.f /opt/cray/xt-asyncpe/3.4/bin/ftn: INFO: linux target is being used

To run this code:

  • Add a line to your batch script setting the LD_LIBRARY_PATH environment variable to the location of your shared objects. We highly recommend using a directory in $SCRATCH for performance reasons.
  • At runtime, make sure you have loaded the same modules you loaded to build the code.

Here's an example batch script.

#PBS -S /usr/bin/csh
#PBS -q debug
#PBS -l mppwidth=128
#PBS -l walltime=00:10:00
#PBS -N my_job
#PBS -j oe

cd $PBS_O_WORKDIR

setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:${SCRATCH}/MY_DIRECTORY

module load AAA #if AAA was used when building the executable

aprun -n 128 ./my_dyn_code.x

Performance Implications of Using Dynamic Shared Objects

Using dynamic libraries may introduce delays in application launch times. This delay is caused by the runtime linking process and the relative inefficiency of symbol lookup in distributed shared objects. Also,  there may be a small performance degradation during execution.

Other Notes

To verify that you have created a dynamically-linked executable use the file command:

edison01% file a.out.dyn  
a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
for GNU/Linux 2.6.4, dynamically linked (uses shared libs), not stripped

Two system utilities related to shared objects that you might want to know about are ldd and readelf. Use these to get information about the shared object that you've created.