Updates and Status
Batch System Changes
Beginning in Allocation Year 2018 on January 9, 2018 NERSC is introducing a simplified batch submission scheme. Users need only specify a QOS (quality of service) in their batch scripts. SLURM scripts that specify a partition are deprecated and those that do will fail upon submission starting February 9, 2018.
The changes will make batch submission simpler for users and improve the reliability and availability of the batch system.
Below are the new allowed QOS values Note that "-q xxx" is the equivalent of "--qos xxx" starting January 9, 2018.
|Current Scheme||New Scheme||Max Wall Time||Submit Limit||Run Limit||Systems|
|<nothing>||-q debug||30 min||--||2||Edison, Cori|
|-p regular||-q regular||48 hrs||--||--||Edison, Cori|
|-p regular --qos premium||-q premium||48 hrs||5||--||Edison, Cori|
|-p shared||-q shared||48 hrs||5,000||--||Edison, Cori|
|--qos=special||-q special||special||special||special||Edison, Cori|
|-p realtime||-q realtime||custom||custom||custom||Cori|
|--qos=interactive||-q interactive||4 hrs||--||1 (max nodes per job = 64)||Cori|
|-p xfer||-q xfer||12 hrs||--||15||Edison, Cori|
|-p bigmem||-q bigmem||72 hrs||5||1||Cori|
- There are no longer premium shared jobs.
- A low charge class is not available. After a repository runs out of time, its jobs will be automatically eligible to run in the low-priority scavenger QOS.
Reservation Required for Modes Other than Quad,Cache on Cori KNL
All KNL nodes in Cori will be provisioned in Quad,Cache mode beginning January 9, 2018. Users who want to run in other KNL modes must request a reservation in advance. NERSC is requiring this because most codes run well in Quad,Cache and changing KNL modes requires a lengthy reboot that makes less compute time available to users.
At the start of AY18 on January 9, batch jobs still in the queue requesting any modes other than quad,cache will be removed, and new jobs requesting such will fail at submission.
Frequency of Programming Environment Software Default Changes
In oder to provide a predictable, stable and consistent programming environment while still making necessary software updates, NERSC will be updating the Cray Programming Environment and Intel's compilers on a set cadence.
- New software will be installed every 3 months, usually in December, March, September, and June. The new versions will not be made the defaults when installed.
- New software defaults will be set twice a year: Once in January at the Allocation Year Rollover (the previous year's September release) and once in June (the March release). The actual versions may vary subject to software verification and security or urgent fixes.
- Only three CDT versions, the current default, the previous default, and the newest, will be made available on the system at any given time. If you need a CDT version that we have removed from the system, please contact consult at nersc.gov. Note: Cray CDT (Cray Developer Toolkit) contains Programming Environment Software release including Cray compiling, message passing interface, performance, debugging, third party libraries, etc.
Cray’s module snapshot feature can help you to save the programming environment of your custom modules loaded and restore to use it later easily.
Recent Status Updates
Below are some recent Cori Status and Updates. Please also refer to Cori Timeline for more history information.
May 8-9, 2018. OS upgrade from CLE6.0UP04 to CLE6.0UP05. New VTune default 2018.up2.
Jan 9, 2018. New AY 2018 starts. QOS only submission scheme. Intel compilers default version changed from 184.108.40.206 to 220.127.116.11. Software default versions set to CDT/17.09.
Oct 6-9, 2017: Intel compilers default version changed from 18.104.22.168 to 22.214.171.124.
Aug 8-10, 2017: OS upgrade from CLE6.0UP03 to CLE6.0UP04. Software default versions set to CDT/17.06.
July 1, 2017: Charging for time used on the Cori KNL nodes started. Large Jobs using 1,024 or more KNL nodes receive 20% charging discount.
May 30, 2017: All users are enabled on Cori KNL without restriction.
Running on the KNL Nodes
To use the KNL nodes, please compile (or recompile) your executables targeting the KNL architecture. The simplest way to do this is to do "module swap craype-haswell craype-mic-knl" before compiling in the usual way.
The KNL architecture is complex but affords a great deal of flexibility for experienced users.