NERSCPowering Scientific Discovery Since 1974

Updates and Status

The Cori Haswell nodes were returned to production for all NERSC users on Oct. 31, 2016. The system had been unavailable to users since Sep. 19 while 9,300 Xeon Phi nodes were added to the system.

Please note that the option passed to the job launcher (srun -c) for requesting optimal process and thread affinity on Haswell changed on Oct. 31, 2016.  You must also now use the -C option to request the node type.  Please see details of these changes here

Dec 21, 2016: All NERSC users are now eligible to use the KNL nodes of Cori via the debug partition, in which users can run jobs of up to 512 nodes for up to 30 minutes.

Jan 15, 2017: We have temporarily disabled KNL mode reboot into a different cluster and memory mode due to potential system crash it could cause.  Currently, there are 1200 quad flat nodes available, and the rest are quad cache nodes.  We are investigating this problem, and expect that the earliest time rebooting node may be reenabled is during the Feb 6 system maintenance.

Jan 23, 2017: All NERSC users are now also eligible to use the KNL nodes of Cori via the regular partition, in which users can run jobs of up to 512 nodes for up to 2 hours.

Running on the KNL Nodes

To use the KNL nodes, please compile (or recompile) your executables targeting the KNL architecture. The simplest way to do this is to do "module swap craype-haswell craype-mic-knl" before compiling in the usual way.

The KNL architecture is complex but affords a great deal of flexibility for experienced users. There are multiple memory modes, and changing from one to another requires a time-consuming reboot of the node. Therefore we recommend that users begin their experience on the KNL nodes in quad cache mode (NERSC's default memory mode). This mode will work well for the majority of use cases.

Example KNL batch scripts can be found here. We encourage you to use the Job Script Generator to aid you in the creation of job scripts for Cori's KNL nodes.

Gaining Full Access to the KNL System

Because the KNL architecture is so different from what most NERSC users have seen, it is important that users understand their codes' performance before gaining full access to the KNL nodes. NERSC has developed a gating and application procedure for allowing new users onto the KNL nodes.  Currently, NESAP teams have full access to the KNL nodes; we expect to begin allowing full access to all KNL nodes for non-NESAP teams no later than March 1.

Software Packages for KNL

Many NERSC-provided applications are not yet available for KNL. At this time we are supporting only a limited number of applications to run on KNL. Most Cray-provided libraries support the KNL architecture, and some NERSC-provided libraries have been built to support KNL as well. We do not plan to compile more libraries for KNL until next year.