NERSCPowering Scientific Discovery Since 1974

Parallel Batch Scripts

Parallel Environments on Genepool

You can run parallel jobs that use MPI or OpenMP on Genepool as long as you make the appropriate changes to your submission script! To investigate the parallel environments that are available on Genepool, you can use

CommandDescription
qconf -sp <pename> Show the configuration for the specified parallel environment.
qconf -spl Show a list of all currently configured parallel environments.

Basic Parallel Example

If your job requires the default 5GB of memory per slot, you can do the following:

#!/bin/bash
#
# == Set SGE options:
#
# -- ensure BASH is used
# -- run the job in the current working directory (where qsub is called)
#$ -cwd
# -- run with the environment variables from the User's environment
#$ -v env_var_name=value
# -- specify an email address
#$ -M uname@lbl.gov
# -- specify when to send the email when job is (a)borted,
# -- (b)egins or (e)nds
#$ -m abe
# -- specifying a 12 hour runtime
#$ -l h_rt=12:00:00
# -- Use the following, so you can catch the errors thrown by SGE if your job fails
# -- to submit.
#$ -w e
####
# -- request 4 slots on the same node, it will get 5GB memory per slot for a total of 20GB
# -- for the entire job.  -pe is the flag for parallel environment, pe_slots is the basic
# -- parallel environment on Genepool.
#$ -pe pe_slots 4
# == Your job info goes here
./myprogram inputs

 

Whole-node Parallel Example 

If you want to run a threaded job on a whole node, use the -l exclusive.c complex.  Note: you do NOT need to specify a parallel environment with the -pe flag. 

#!/bin/bash
#
# == Set SGE options:
#
# -- ensure BASH is used
#$ -S /bin/bash
# -- run the job in the current working directory (where qsub is called)
#$ -cwd
# -- run with the environment variables from the User's environment
#$ -v env_var_name=value
# -- specify an email address
#$ -M uname@lbl.gov
# -- specify when to send the email when job is (a)borted,
# -- (b)egins or (e)nds
#$ -m abe
# -- specifying a 12 hour 30 minute runtime
#$ -l h_rt=12:30:00
# -- request a whole node
#$ -l exclusive.c
# -- Use the following, so you can catch the errors thrown by SGE if your job fails
# -- to run.
#$ -w e
####
# -- run a job with 16 threads, this is an example with OpenMP
# NOTE: You do NOT need to specify a parallel environment.
# == Your job info goes here
export OMP_NUM_THREADS=16
./myprogram inputs

 

Whole-node MPI example

If you would like to run an MPI job across more than one of the whole-nodes, you will need to specify the appropriate parallel environment.

#!/bin/bash
#
# == Set SGE options:
#
# -- run the job in the current working directory (where qsub is called)
#$ -cwd
# -- run with the environment variables from the User's environment
#$ -v env_var_name=value
# -- Specify the memory per MPI task to 120G
#$ -l ram.c=120G
# -- specify an email address
#$ -M uname@lbl.gov
# -- specify when to send the email when job is (a)borted,
# -- (b)egins or (e)nds
#$ -m abe
# -- specifying a 24 hour runtime
#$ -l h_rt=24:00:00
# -- Use the following, so you can catch the errors thrown by SGE if your job fails
# -- to run.
#$ -w e
####
# -- The parallel environment is specified with the -pe flag. The following
# -- requests 2 nodes with one MPI task per node
#$ -pe pe_1 2
# == Your job info goes here
module load openmpi
mpirun ./myprogram inputs

 

Parallel Large Memory Example

If you want to request more or less than the default 5GB of memory, then you can use -pe and -l ram.c to request the necessary resources for your job.

#!/bin/bash
#
# == Set SGE options:
#
# -- ensure BASH is used
# -- run the job in the current working directory (where qsub is called)
#$ -cwd
# -- run with the environment variables from the User's environment
#$ -v env_var_name=value
# -- specify an email address
#$ -M uname@lbl.gov
# -- specify when to send the email when job is (a)borted,
# -- (b)egins or (e)nds
#$ -m abe
# -- specifying a 24 hour runtime
#$ -l h_rt=24:00:00
# -- Use the following, so you can catch the errors thrown by SGE if your job fails
# -- to run.
#$ -w e
####
# -- request 4 slots on the same node, -pe is the flag for parallel environment, pe_slots is the basic
# -- parallel environment on Genepool. 
#$ -pe pe_slots 4
# -- request 10GB of memory per slot, for a total of 10*4= 40GB
#$ -l ram.c=10G
# == Your job info goes here
./myprogram inputs

Notes about high-memory jobs on Genepool

If you request more than 48GB of memory (# of slots * memory requested) and your jobs cannot run across multiple nodes, you will be routed to the high memory nodes, (currently those nodes with over 48GB of memory) and it is likely your jobs will wait longer in the queue.  This is due to the fact that there are only a few high-memory nodes on the system.

If your code is set up to run across multiple nodes and you are requesting a small amount of memory per slot, you can run with the

#$ -pe pe_fill <numslots>

 This will spread your job across multiple lower memory nodes and not force you to wait for a single high memory node.