NERSCPowering Scientific Discovery Since 1974

Batch Jobs


Batch jobs are jobs that run non-interactively under the control of a "batch script," which is a text file (shell script) containing a number of job directives and LINUX commands. Batch scripts are submitted to the "batch system," where they are queued awaiting free resources on Edison. The batch system on Edison is Slurm. Please visit the Introduction to Slurm page to learn more about Slurm at NERSC.

Preparing a Batch Script

On Edison, the sbatch command is used to submit a batch script to Slurm. A batch script is a shell script containing a bunch of sbatch options preceded with "#SBATCH" and a srun command that executes your code in parallel. The simplest batch script will look something like this: 

#!/bin/bash -l
#SBATCH -p debug
#SBATCH -t 00:10:00
#SBATCH -L SCRATCH #note: specify license need for the file systems your job needs, such as SCRATCH,project

srun -n 48 ./my_executable

Or look like this in the long format of the sbatch options:

#!/bin/bash -l
#SBATCH --partition=debug
#SBATCH --nodes=2
#SBATCH --time=00:10:00
#SBATCH --license=SCRATCH #note: specify license need for the file systems your job needs, such as SCRATCH,project

srun -n 48 ./my_executable

This script requests two nodes (-N) to use for 10 minutes (-t) using the debug partition (-p). This example illustrates the three basic parts of a batch script:

  • Shell choice. #!/bin/bash -l indicates that your batch script is written in bash. You must indicate the shell at the first line of your batch script,  otherwise your job submission will fail. You can use other shells, such as csh, tcsh, zsh, etc.
  • #SBATCH directives. These are the sbatch options preceded with "#SBATCH". These directives tell the batch system in which partition to use, and how many nodes to reserve for your job and for how long to reserve those nodes. The "-L" or "--license=" directive specifies the file systems your job needs (More details on requesting file system license can be found here). Directives can also specify things like what to name STDOUT filets, what account to charge, whether to notify you by email when your job finishes, etc. 
  • The srun command is used to start execution of your code with multiple processes.

Sbatch Options

The following table lists recommended and useful sbatch options. For the full list of the sbatch options, see the sbatch man page. 

Short FormatLong FormatDefaultDescription
-N count  --nodes=count One node will be used. Used to allocate count nodes to your job.
-t HH:MM:SS --time=HH:MM:SS 00:30:00 Always specify the maximum wallclock time for your job.
-p [partition]  --partition=partition debug Always specify your partition, which will usually be debug for testing and regular for production runs. See "Queues and Policies".
N/A --ntasks-per-node=count 32 (Cori) Use [count] of MPI tasks per node 
-c count  --cpus-per-task=count 2 Run count threads threads per MPI task (for a MPI/OpenMP hybrid code, to run in pure MPI mode, please set OMP_NUM_THREADS to 1)
-L list_of_filesystems
N/A.  global homes and global common are implicit default Specifies the file systems (list separated by comma) needed for your job. Available choices for Edison are: scratch1, scratch2 (or SCRATCH), scratch3, dna, project, projecta, projectb.  More details on requesting file system license can be found here
-J job_name  --job-name=name Job script name. Job Name: up to 15 printable, non-whitespace characters.
-A repo  --account=repo Your default repo Charge this job to the NERSC repository mXXX (necessary only if you have more than one NERSC repo)
-e filename  --error=filename <script_name>.e<job_id> Write STDERR to filename
-o filename  --output=filename <script_name>.o<job_id> Write STDOUT to filename. By default both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is replaced with the job allocation number. See the -i option for filename specification options.
-i   "/dev/null" is open on the batch script's standard input and both standard output and standard error are directed to a file of the name "slurm-%j.out", where the "%j" is the job id.  Instruct SLURM to connect the batch script's standard input directly to the file name specified in the "filename pattern".

The filename pattern may contain one or more replacement symbols, which are a percent sign "%" followed by a letter (e.g. %j).

Supported replacement symbols are:

Job allocation number.
Node name. Only one file is created, so %N will be replaced by the name of the first node in the job, which is the one that runs the script.





Email notification Valid event values are: BEGIN, END, FAIL, REQUEUE, ALL (equivalent to BEGIN, END, FAIL, REQUEUE, and STAGE_OUT), STAGE_OUT (burst buffer stage out completed), TIME_LIMIT, TIME_LIMIT_90 (reached 90 percent of time limit), TIME_LIMIT_80 (reached 80 percent of time limit), and TIME_LIMIT_50 (reached 50 percent of time limit). Multiple type values may be specified in a comma separated list. The user to be notified is indicated with --mail-user. Mail notifications on job BEGIN, END and FAIL apply to a job array as a whole rather than generating individual email messages for each task in the job array.
-D directory_name --workdir=directory_name  Directory specification Set the working directory of the batch script to directory before it is executed. The path can be specified as full path or relative path to the directory where the command is executed.
N/A --export=ALL This is on by default Export the current environment variables into the batch job environment.  This is the default behavior.

All options may be specified as either sbatch command-line options or as directives in the batch script as #SBATCH options.  Note: if you use both, any command line options will override the corresponding options in the batch script.

Srun Options

All (parallel) codes that execute with multiple compute nodes on Edison must be started with the "srun" command. Most of the srun options have the corresponding sbatch options.  The following table shows the commonly used srun options. for the full list of the srun options, please see the srun man page. 

Short FormatLong FormatDefaultDescription
-n  --ntasks N/A Specify  the  number  of tasks to run. 
-N --nodes   Request  that  a  minimum  of minnodes nodes be allocated to this job.
-c count  --cpus-per-task=count 2 Run count threads threads per MPI task 
  --cpu_bind   Bind tasks to CPUs. 

Choosing a partition

You can choose from two partitions for your job, debug and regular. The debug partition is meant mainly for short, interactive debugging purpose, so it has a higher priority but has a 30 minutes of wall clock limit. The regular partition is mainly for production runs. It can be run with different Quality of Service (QOS) (normal, low, premium) request. Different partitions with different QOS may have different charge rates. The main purpose of having different queues and QOS is to control scheduling priorities and set limits on the numbers of jobs of different sizes and also to achieve an optimal balance among fairness, wait times, run times, and DOE strategic goals.

When you submit your batch job you will usually chose one of these partitions

  • regular : Use this for almost all your production runs.
  • debug: Use this for small, short test runs

Here is an example of requesting the premium QOS (for the increased queue priority) with the regular partition.

#!/bin/bash -l
#SBATCH -p regular
#SBATCH --qos=premium
#SBATCH -t 06:00:00

srun -n 48 ./my_executable

See Queues and Scheduling Policies for more information about Edison queues. 

Job Submission

Once you have a batch script you submit it to the batch system using the "sbatch" command. For a batch script named "myscript.slurm",  type

edison01% sbatch myscript.slurm

from the directory that contains the script file (the directory is called the working directory). In addition to specify the sbatch options preceded with "#SBATCH" in the batch script,  you can also specify them on the sbatch command line. For example,

edison01% sbatch -p regular -t 01:00:00 myscript.slurm

In this example, it requests the regular partition for 1 hour.  When both the sbatch directives and command line options are used, the command line options override the corresponding options in the batch script.

Job Management

Slurm provides a rich set of commands to manage your jobs. You submit a job to the batch system using the sbatch command and use the squeue command to monitor its status in the queue. You use the scontrol command to view or update jobs, including holding/releasing a job, and modifying the submission parameters, and use the scancel command to delete jobs. You check your job's scheduling priority using the sprio command, and check your job's resource usage with the sstat command. Please check the man pages of these commands for more information about them. 

Job Accounting 

Once your job is completed, you can use the sacct command to see the accounting data for your job.