NERSCPowering Scientific Discovery Since 1974

Please use "gres" settings in your batch scripts

September 4, 2012 by Helen He

We would like to encourage you to use the generic resources ("gres") setting for various file systems that your batch jobs use. This feature is currently available on Hopper and Carver.  The advantage of this setting is that your jobs won't start (thus won't fail) during a scheduled file system maintenance.

The syntax for the "gres" setting is:
#PBS -l gres=filesystem1[%filesystem2%filesystem3...]            (new recommendation)
#PBS -l gres=filesystem1:1[%filesystem2:1%filesystem3:1...]        (as announced before)

Note that the "%" character means "and".  Therefore, if multiple file systems are specified, the job will not start if *any* of the specified file systems are unavailable.

The file systems available for "gres" setting are:
on Hopper: scratch, scratch2, gscratch, project, and projectb
on Carver: gscratch, project, and projectb
(note: home is not a defined resource, since it is used by every job anyway).

Below are some sample "gres" lines:
#PBS -l gres=project
#PBS -l gres=scratch
#PBS -l gres=gscratch2%project
#PBS -l gres=scratch%scratch2%projectb

To add this option to a queued job (not in "running" state), the following command can be used:
% qalter -l gres=... my_jobid

We will have a scheduled project file system maintenance on Sept 19, lasting for 2 to 3 days (details TBA).  If your current or future batch jobs depend on the project file system, please use the "gres=project" setting in your batch scripts (or use the qalter command for scripts already submitted). This will prevent your jobs from failing during this outage.

In general, we strongly encourage you *always* to include the "gres" directive in your batch scripts to specify the file systems needed to protect the jobs during future scheduled file system outages.