NERSCPowering Scientific Discovery Since 1974

Updates and Status

Software Default and Program Environment Change in July 2018

Following the scheduled maintenances on Cori (July 11) and Edison (July 25), the machines will return to service with a new default programmming environment. This is in compliance with the NERSC Programming Environment Update Policy  Please see detailed list of changes for more information.

Batch System Changes

Beginning in Allocation Year 2018 on January 9, 2018 NERSC is introducing a simplified batch submission scheme. Users need only specify a QOS (quality of service) in their batch scripts. SLURM scripts that specify a partition are deprecated and those that do will fail upon submission starting February 9, 2018. 

The changes will make batch submission simpler for users and improve the reliability and availability of the batch system.

QOS specifications

Below are the new allowed QOS values Note that "-q xxx" is the equivalent of "--qos xxx" starting January 9, 2018.

Current Scheme New Scheme Max Wall Time Submit Limit Run Limit Systems
<nothing> -q debug 30 min -- 2 Edison, Cori
-p regular -q regular 48 hrs --  -- Edison, Cori
-p regular --qos premium -q premium 48 hrs 5 -- Edison, Cori
-p shared -q shared 48 hrs 5,000 -- Edison, Cori
--qos=special -q special  special          special special Edison, Cori
-p realtime -q realtime custom custom custom Cori
--qos=interactive -q interactive 4 hrs -- 1 (max nodes per job = 64) Cori
-p xfer -q xfer 12 hrs -- 15  Edison, Cori
-p bigmem -q bigmem 72 hrs 5 Cori

Please note:

  • There are no longer premium shared jobs. 
  • A low charge class is not available. After a repository runs out of time, its jobs will be automatically eligible to run in the low-priority scavenger QOS.

Reservation Required for Modes Other than Quad,Cache on Cori KNL

All KNL nodes in Cori will be provisioned in Quad,Cache mode beginning January 9, 2018. Users who want to run in other KNL modes must request a reservation in advance. NERSC is requiring this because most codes run well in Quad,Cache and changing KNL modes requires a lengthy reboot that makes less compute time available to users.

At the start of AY18 on January 9, batch jobs still in the queue requesting any modes other than quad,cache will be removed, and new jobs requesting such will fail at submission.

Frequency of Programming Environment Software Default Changes

In oder to provide a predictable, stable and consistent programming environment while still making necessary software updates, NERSC will be updating the Cray Programming Environment and Intel's compilers on a set cadence. 

  • New software will be installed every 3 months, usually in December, March, September, and June. The new versions will not be made the defaults when installed. 
  • New software defaults will be set twice a year: Once in January at the Allocation Year Rollover (the previous year's September release) and once in July (the March release). The actual versions may vary subject to software verification and security or urgent fixes.
  • No more than four CDT versions, the December, March, September, and June releases, will be made available on the system at any given time. Same applies to the Intel compilers. If you need a CDT version that we have removed from the system, please contact consult at nersc.gov. Note: Cray CDT (Cray Developer Toolkit) contains Programming Environment Software release including Cray compiling, message passing interface, performance, debugging, third party libraries, etc.

Cray’s module snapshot feature can help you to save the programming environment of your custom modules loaded and restore to use it later easily. 

Recent Status Updates

Below are some recent Cori Status and Updates. Please also refer to Cori Timeline for more history information.

May 8-9, 2018. OS upgrade from CLE6.0UP04 to CLE6.0UP05. New VTune default 2018.up2. 

Jan 9, 2018.  New AY 2018 starts. QOS only submission scheme.  Intel compilers default version changed from 18.0.0.128 to 18.0.1.163.  Software default versions set to CDT/17.09.

Oct 6-9, 2017: Intel compilers default version changed from 17.0.2.174 to 18.0.0.128.

Aug 8-10, 2017: OS upgrade from CLE6.0UP03 to CLE6.0UP04.  Software default versions set to CDT/17.06.

July 1, 2017: Charging for time used on the Cori KNL nodes started. Large Jobs using 1,024 or more KNL nodes receive 20% charging discount.

May 30, 2017: All users are enabled on Cori KNL without restriction.

Running on the KNL Nodes

To use the KNL nodes, please compile (or recompile) your executables targeting the KNL architecture. The simplest way to do this is to do "module swap craype-haswell craype-mic-knl" before compiling in the usual way.

The KNL architecture is complex but affords a great deal of flexibility for experienced users. 

Example KNL batch scripts can be found here. We encourage you to use the Job Script Generator to aid you in the creation of job scripts for Cori's KNL nodes.