NERSCPowering Scientific Discovery Since 1974

Transitioning Edison workloads to Cori KNL nodes

Edison has been retired as of May 13, 2019. If you have not moved your workloads to Cori, please refer to this presentation for a quick instruction. The following is a short summary.

Transit your workloads to Cori KNL

Cori consists of two types of compute nodes, Haswell and KNL. You may find your applications run faster on Haswell nodes if you haven't optimized your codes for KNL. However, we strongly recommend you to transit your workloads to Cori KNL instead of Haswell for a faster queue turnaround, as there are four times more KNL nodes (9688) than Haswell nodes (2388) on Cori.

How to compile

The compilation on Cori KNL is very similar to Edison. For most cases, you just need to swap a module before compiling:

module swap craype-haswell craype-mic-knl

provided you use the compiler wrappers (ftn for Fortran, cc for C and CC for C++ codes) provided by Cray. Please refer to our docs page (select Programming link in the left navigation bar) for more info. You can also get help from the NERSC consultants via phone calls, emails, and the KNL office hours (see the How to get help section below).

How to run jobs on Cori KNL

Please refer to the Queues and Policy page for the available QOS's on Cori. Notice that Cori has a QOS called "interactive" which is not available on Edison.

To run jobs interactively:

We recommend you to make use of the interactive QOS which is capable of allocating you the requested number of compute nodes (up to 64 nodes per repo) for up to 4 hours instantly most of the time. You must use the salloc command like to run jobs interactively under the interactive QOS:

salloc -N 2 –q interactive –t 4:00:00 -C knl

You will get two KNL nodes for 4 hours to run jobs interactively with the above salloc command. 

To submit batch jobs:

Here are sample job scripts to run a pure MPI and a hybrid MPI+OpenMP jobs on Cori KNL nodes. 

For pure MPI jobs:

#!/bin/bash -l
#SBATCH –N 1
#SBATCH -C knl
#SBATCH –q regular
#SBATCH –t 1:00:00
#SBATCH –L SCRATCH 
srun –n68 -c4 --cpu_bind=cores ./a.out

For MPI+OpenMP jobs:

#!/bin/bash -l
#SBATCH –N 1
#SBATCH –q regular
#SBATCH –t 1:00:00
#SBATCH -C knl

export OMP_PROC_BIND=true
export OMP_PLACES=threads
export OMP_NUM_THREADS=4

# launching 1 task every 2 cores/8 CPUs
srun –n32 –c8 --cpu_bind=cores ./a.out

How to get help

  1. To talk to a consultant, dial 1-800-666-3772 (or 1-510-486-8600) menu option 3
  2. To get help by emails, file a ticket at help.nersc.gov or  https://my.nersc.gov/tickets.php
  3. To join  the KNL Office Hours, join the Zoom Meeting at 9:00-15:00 PDT on Fridays (4/26, 5/3, 5/10, 5/17, 5/24, 5/31) at https://lbnl.zoom.us/j/214639988