Submitting Batch Jobs

A batch job is the typical way users run production applications on NERSC machines. The user submits a batch script to the batch system (Torque). This script specifies, at the very least, how many nodes and cores the job will use, how long the job will run, and the name of the application to run. The job will advance in the queue until it has reached the top, when it will then be launched on the compute nodes. The output of the job will be available when the job has completed.

Follow these steps to submit batch job on Dirac:

1. Copy the example batch script below into a file
2. Edit the batch script, modifying the Torque keywords for the desired run
3. Submit the job using the qsub command.

PBS (Torque) directive lines tell the batch system how to run a job and begin with #PBS. The following batch script requests 16 nodes (128 cores) with a wall clock limit of 10 minutes, in the regular queue, using Fermi nodes.

#PBS -q dirac_reg
#PBS -l nodes=16:ppn=8:fermi
#PBS -l walltime=00:10:00
#PBS -N my_job
#PBS -e my_job.$PBS_JOBID.err
#PBS -o my_job.$PBS_JOBID.out
#PBS -V

cd $PBS_O_WORKDIR
mpirun -np 128 ./my_executable

To request Tesla nodes:

#PBS -q dirac_reg
#PBS -l nodes=4:ppn=8:tesla
#PBS -l walltime=00:10:00
#PBS -N my_job
#PBS -e my_job.$PBS_JOBID.err
#PBS -o my_job.$PBS_JOBID.out
#PBS -V

cd $PBS_O_WORKDIR
mpirun -np 32 ./my_executable

In the sample scripts above, the line cd $PBS_O_WORKDIR changes the current working directory to the directory from which the script was submitted. NERSC recommends running jobs from $SCRATCH instead of $HOME. The easiest way to run a job from $SCRATCH is to submit the job from a $SCRATCH subdirectory. Alternatively, a user may replace cd $PBS_O_WORKDIR with a cd directly to the $SCRATCH subdirectory.