NERSCPowering Scientific Discovery Since 1974

Updates and Status

Cori KNL Low QOS

The "low" qos is added on February 13, 2019 on Cori KNL nodes to allow non-urgent jobs to run with a lower usage charge.  It is is charged 50% as compared to the "regular" QOS, but no extra large jobs discount. See Cori Queue Policy pages for detailed information.

Cori Allocation Year Rollover (January 8, 2019)

Cori will be scheduled for an Allocation Year (AY) Rollover maintenance from 7am to 5pm on January 8, 2019, we will set the Cray Programming Environment Software release CDT/18.09 as default,  install the new release CDT/18.12, and retire old CDT/18.06 on Cori. This is in compliance with the NERSC Programming Environment Update Policy.  There will be no Intel compiler version default change at this time.  Please see a detailed list of changes for more information. 

After the maintenance, Queued ("idle") jobs associated with non-continuing repositories will be deleted. Pending 2018 "scavenger" jobs (those in the queue that had no AY18 allocations left) will also be deleted from the queue.

Also, "premium" QOS will not be available from January 2 to January 7 to allow fair use from repos with remaining AY18 allocation hours.

New charge factors for Cori Haswell (90, increased from 80) and KNL(90, decreased from 96) will be in effect starting January 9, 2018. Also starting in AY19, jobs for which there is insufficient allocation will be rejected, and users will be required to explicitly submit to the "scavenger" or "shared_scavenger" qos.  And any held user jobs will be deleted from the queue 12 weeks after they were submitted. More details about the updated queue policy can be found here.

OS Upgrade and Data Warp Maintenance in September 2018

During the September 19, 2018 scheduled Cori maintenance, there was an OS upgrade from CLE6.0UP05 to CLE6.0UP07. Relinking your applications is recommended to incorporate the updated system libraries.

Also, there was a DataWarp maintenance during this maintenance. This requires a reformat of all DataWarp SSDs, therefore all Persistent Reservations were removed before the update. Users were asked to back up any data you wish to keep before September 19 and re-create the Persistent Reservation afterward if needed.

We also installed the new Cray Programming Environment Software release CDT/18.09, and retire old CDT/17.12. This is in compliance with the NERSC Programming Environment Update Policy.  New Intel compiler version 18.0.3.222 was installed. There was no software default change this time.   Please see a detailed list of changes for more information.

Software Default and Program Environment Change in July 2018

Following the scheduled maintenances on Cori (July 11) and Edison (July 25), the machines returned to service with a new default programming environment. This is in compliance with the NERSC Programming Environment Update Policy  Please see a detailed list of changes for more information.

Batch System Changes

Beginning in Allocation Year 2018 on January 9, 2018, NERSC introduced a simplified batch submission scheme. Users need only specify a QOS (quality of service) in their batch scripts. SLURM scripts that specify a partition are deprecated and those that do will fail upon submission starting February 9, 2018. 

The changes makes batch submission simpler for users and improve the reliability and availability of the batch system.

QOS specifications

Below are the new allowed QOS values Note that "-q xxx" is the equivalent of "--qos xxx" starting January 9, 2018.

Current Scheme New Scheme Max Wall Time Submit Limit Run Limit Systems
<nothing> -q debug 30 min -- 2 Edison, Cori
-p regular -q regular 48 hrs --  -- Edison, Cori
-p regular --qos premium -q premium 48 hrs 5 -- Edison, Cori
-p shared -q shared 48 hrs 5,000 -- Edison, Cori
--qos=special -q special  special          special special Edison, Cori
-p realtime -q realtime custom custom custom Cori
--qos=interactive -q interactive 4 hrs -- 1 (max nodes per job = 64) Cori
-p xfer -q xfer 12 hrs -- 15  Edison, Cori
-p bigmem -q bigmem 72 hrs 5 Cori

Please note:

  • There are no longer premium shared jobs. 
  • A low charge class is not available. After a repository runs out of time, its jobs will be automatically eligible to run in the low-priority scavenger QOS.

Reservation Required for Modes Other than Quad,Cache on Cori KNL

All KNL nodes in Cori are provisioned in quad,cache mode. Users who want to run in other KNL modes must request a reservation in advance. NERSC is requiring this because most codes run well in quad,cache and changing KNL modes requires a lengthy reboot that makes less compute time available to users.

Frequency of Programming Environment Software Default Changes

In order to provide a predictable, stable and consistent programming environment while still making necessary software updates, NERSC will be updating the Cray Programming Environment and Intel's compilers on a set cadence. 

  • New software will be installed every 3 months, usually in December, March, September, and June. The new versions will not be made the defaults when installed. 
  • New software defaults will be set twice a year: Once in January at the Allocation Year Rollover (the previous year's September release) and once in July (the March release). The actual versions may vary subject to software verification and security or urgent fixes.
  • No more than four CDT versions will be made available on the system at any given time, with the consideration of keeping past-default versions longer than non-past-default versions. Same applies to the Intel compilers. If you need a CDT version that we have removed from the system, please contact consult at nersc.gov. Note: Cray CDT (Cray Developer Toolkit) contains Programming Environment Software release including Cray compiling, message passing interface, performance, debugging, third party libraries, etc.

Cray’s module snapshot feature can help you to save the programming environment of your custom modules loaded and restore to use it later easily. 

Recent Status Updates

Please also refer to Cori Timeline for more history information.

Running on the KNL Nodes

To use the KNL nodes, please compile (or recompile) your executables targeting the KNL architecture. The simplest way to do this is to do "module swap craype-haswell craype-mic-knl" before compiling in the usual way.

The KNL architecture is complex but affords a great deal of flexibility for experienced users. 

Example KNL batch scripts can be found here. We encourage you to use the Job Script Generator to aid you in the creation of job scripts for Cori's KNL nodes.