NERSCPowering Scientific Discovery Since 1974

Shared and Dynamic Libraries

The Edison system can support applications that use dynamic shared libraries (DSL) on the compute nodes.  Some "out-of-the-box" applications require DSLs and some popular applications like Python use DSLs as well.

Using System Shared and Dynamic Libraries

"System" DSLs include those that support software packages found in "typical" Linux distributions, e.g. Python and Perl. To build an application that will use system DSLs on Edison, you must do two things:

  • use the  -dynamic flag in the link step of your build

The -dynamic flag instructs the compiler wrappers (i.e., ftn, cc, CC)  to prepare a dynamically linked executable. 

edison01% ftn -dynamic -o dyn_example.x dyn_example.f90

Instead of using the compiler/linker flag, -dynamic, you can use an environment variable, CRAYPE_LINK_TYPE to make the compiler wrappers to link dynamically.

An example batch script to run a dynamically linked executable is as follows:

#!/bin/bash -l
#SBATCH -p debug
#SBATCH -N 2
#SBATCH -t 30:00
#SBATCH -J my_job
#SBATCH -o my_job.o%j

srun -n 48 ./example.x

Using Your Own Shared and Dynamic Libraries

To build and use your own DSLs

  • compile and build libraries with the -shared -fPIC compiler options
  • link with the -dynamic flag

An example follows. First, compile several C source files:

edison01% cc -shared -fPIC source1.c 
edison01% cc -shared -fPIC source2.c

Create a dynamic library, my_c_lib.so, from the C objects:

edison01% cc -fPIC -shared -o my_c_lib.so source1.o source2.o 

Link the dynamic library with a driver routine:

edison01% cc -fPIC -dynamic -o my_dyn_code.x driver.c my_c_lib.so

You can also create a dynamic library from Fortran source.

edison01% ftn -dynamic -fPIC -o my_f_lib.so stuff.f /opt/cray/xt-asyncpe/3.4/bin/ftn: INFO: linux target is being used

To run this code:

  • Add a line to your batch script setting the LD_LIBRARY_PATH environment variable to the location of your shared objects. We highly recommend using a directory in $SCRATCH for performance reasons. 
  • At runtime, make sure you have loaded the same modules you loaded to build the code.

Here's an example batch script.

#!/bin/bash -l
#SBATCH -p debug
#SBATCH -N 2
#SBATCH -t 30:00
#SBATCH -J my_job
#SBATCH -o my_job.o%j

#load the modules that were used when building the executable
export LD_LIBRARY_PATH=<Your library directory>:${LD_LIBRARY_PATH}
srun -n 48 ./my_dyn_code.x


Performance Implications of Using Dynamic Shared Objects

Using dynamic libraries may introduce delays in application launch times. This delay is caused by the runtime linking process and the relative inefficiency of symbol lookup in distributed shared objects. Also,  there may be a small performance degradation during execution.

Other Notes

To verify that you have created a dynamically-linked executable use the file command:

edison01% file a.out.dyn  
a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
for GNU/Linux 2.6.4, dynamically linked (uses shared libs), not stripped

Two system utilities related to shared objects that you might want to know about are ldd and readelf. Use these to get information about the shared object that you've created.