NERSCPowering Scientific Discovery Since 1974

Updates and Status

Batch System Changes

Beginning in Allocation Year 2018 on January 9, 2018 NERSC is introducing a simplified batch submission scheme. Users need only specify a QOS (quality of service) in their batch scripts. SLURM scripts that specify a partition are deprecated and those that do will fail upon submission starting February 9, 2018. Current scripts will work until that time.

The changes will make batch submission simpler for users and improve the reliability and availability of the batch system.

QOS specifications

Below are the new allowed QOS values Note that "-q xxx" is the equivalent of "--qos xxx" starting January 9, 2018.

Current Scheme New Scheme Max Wall Time Submit Limit Run Limit Systems
<nothing> -q debug 30 min -- 2 Edison, Cori
-p regular -q regular 48 hrs --  -- Edison, Cori
-p regular --qos premium -q premium 48 hrs 5 -- Edison, Cori
-p shared -q shared 48 hrs 5,000 -- Edison, Cori
--qos=special -q special  special          special special Edison, Cori
-p realtime -q realtime custom custom custom Cori
--qos=interactive -q interactive 4 hrs -- 1 (max nodes per job = 64) Cori
-p xfer -q xfer 12 hrs -- 15  Edison, Cori
-p bigmem -q bigmem 72 hrs 5 Cori

Please note:

  • There are no longer premium shared jobs. 
  • A low charge class is not available. After a repository runs out of time, its jobs will be automatically eligible to run in the low-priority scavenger QOS.

Reservation Required for Modes Other than Quad,Cache on Cori KNL

All KNL nodes in Cori will be provisioned in Quad,Cache mode beginning January 9, 2018. Users who want to run in other KNL modes must request a reservation in advance. NERSC is requiring this because most codes run well in Quad,Cache and changing KNL modes requires a lengthy reboot that makes less compute time available to users.

At the start of AY18 on January 9, batch jobs still in the queue requesting any modes other than quad,cache will be removed, and new jobs requesting such will fail at submission.

Frequency of Programming Environment Software Default Changes

In oder to provide a predictable, stable and consistent programming environment while still making necessary software updates, NERSC will be updating the Cray Programming Environment and Intel's compilers on a set cadence. 

  • New software will be installed every 3 months, usually in December, March, September, and June. The new versions will not be made the defaults when installed. 
  • New software defaults will be set twice a year: Once in January at the Allocation Year Rollover (the previous year's September release) and once in June (the March release). The actual versions may vary subject to software verification and security or urgent fixes.
  • Only three CDT versions, the current default, the previous default, and the newest, will be made available on the system at any given time. If you need a CDT version that we have removed from the system, please contact consult at Note: Cray CDT (Cray Developer Toolkit) contains Programming Environment Software release including Cray compiling, message passing interface, performance, debugging, third party libraries, etc.

Cray’s module snapshot feature can help you to save the programming environment of your custom modules loaded and restore to use it later easily. 

Recent Status Updates

Below are some recent Cori Status and Updates. Please also refer to Cori Timeline for more history information.

Jan 9, 2018.  New AY 2018 starts. QOS only submission scheme.  Intel compilers default version changed from to  Software default versions set to CDT/17.09.

Oct 6-9, 2017: Intel compilers default version changed from to

Aug 8-10, 2017: OS upgrade from CLE6.0UP03 to CLE6.0UP04.  Software default versions set to CDT/17.06.

July 1, 2017: Charging for time used on the Cori KNL nodes started. Large Jobs using 1,024 or more KNL nodes receive 20% charging discount.

May 30, 2017: All users are enabled on Cori KNL without restriction.

Running on the KNL Nodes

To use the KNL nodes, please compile (or recompile) your executables targeting the KNL architecture. The simplest way to do this is to do "module swap craype-haswell craype-mic-knl" before compiling in the usual way.

The KNL architecture is complex but affords a great deal of flexibility for experienced users. 

Example KNL batch scripts can be found here. We encourage you to use the Job Script Generator to aid you in the creation of job scripts for Cori's KNL nodes.