NERSCPowering Scientific Discovery Since 1974

Working with Modules within Perl and Python

It can often be convenient to work with the modules system from within perl or python scripts.  You can do this!

Using Modules within Python

The EnvironmentModules python package gives access to the module system from within python.  The EnvironmentModules python package has a single function: module.  Using this function you can provide the same arguments you would to "module" on the command line.  The module() function accepts a list of arguments, like ['load','<modulename>']; or ['unload','<modulename>'].

>>> import EnvironmentModules as EnvMod
>>> EnvMod.module(['load','blast+'])

 It is important to understand that this is most effective for scripts which execute other code (e.g. from the subprocess package of python), and not necessarily for loading additional packages for python to use.  This is because the python process is already running and changing its environment won't necessarily give expected results.  For example, changes to PYTHONPATH and LD_LIBRARY_PATH are not immediately accepted.  LD_LIBRARY_PATH is only evaluated at process start-up time, and won't be re-evaluated later.  Thus if you load any python packages which rely on dynamically linked C-code, you should load those modules before python (oracle_client, for example).

Problem with LD_LIBRARY_PATH

Wrong way

>>> EnvMod.module('load','oracle_client')
>>> import cx_Oracle
Traceback (most recent call last):
  File "", line 1, in 
ImportError: libclntsh.so.11.1: cannot open shared object file: No such file or directory
>>>

Right way

genepool$ module load oracle_client
genepool$ python
Python 2.7.4 (default, Apr 17 2013, 10:26:13)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import cx_Oracle
...

This is happening because the cx_Oracle python package relies on the dynamically linked library libclntsh.so.11.1, which is found by the operating system by setting the correct path in LD_LIBRARY_PATH. Since LD_LIBRARY_PATH is only evaluated when python starts, the oracle_client modulefile (which sets LD_LIBRARY_PATH for oracle), needs to be loaded before python is started.  For some applications, it may be easier to bootstrap the scripting session with a scriptEnv.

Workaround for PYTHONPATH

The PYTHONPATH environment variable is set to sys.path when python starts.  There is an opportunity to adjust sys.path and still take advantage to changes to PYTHONPATH if you need this functionality:

>>> import os
>>> import sys
>>> import EnvironmentModules as EnvMod
>>> EnvMod.module(['load','biopython'])
>>> for x in os.getenv("PYTHONPATH").split(":"):
...   if x not in sys.path:
...     sys.path.append(x)
...
>>> from Bio import SeqIO
...

 

Using Modules within Perl

The EnvironmentModules perl package gives access to the module system from within perl.  The EnvironmentModules perl package has a single function: module.  Using this function you can provide the same arguments you would to "module" on the command line.  The module() function accepts a single string or array representing the arguments you would pass to the module command line tool.  Note that EnvironmentModules.pm is only installed in the modules version of perl, and thus you need to load the perl module before you can access EnvironmentModules.pm.

#!/usr/bin/env perl
use EnvironmentModules;

module("load blast+"); 

Please note that the similar to python, loading modules which manipulate LD_LIBRARY_PATH or PERL5DIR will not work as expected.  It is generally recommended to load the modules before entering into perl or python instances.

Another popular method, loading the module in system() immediately before running an executable should always be avoided.  This is not portable, and will only work if your users are using the bash shell:

#!/usr/bin/env perl
...
### DO NOT DO THIS!
system("module load blast+; blastn ..."); ### DO NOT DO THAT
### REALLY, DO NOT DO THIS!

 When you call system(), perl forks it as "/bin/sh -c '<your command'>".  /bin/sh does not get a new module environment loaded, so, the instance of /bin/sh will be relying on the shell operating perl to get the module functionality.  It turns out that the module functionality provided by bash is correct for /bin/sh, but not the module functionality provided in tcsh or csh.  If you do the above, your code will only work for other bash users.  Instead, you should either load modules before running perl (a wrapper script -- preferred), or use the EnvironmentModules mechanism shown above.