NERSCPowering Scientific Discovery Since 1974

Cori Configuration

cori panorama 1997x723

 

NERSC's Cori supercomputer consists of two partitions, one with Intel Xeon "Haswell" processors  (Phase I) and another with Intel Xeon Phi "Knights Landing" (KNL) processors (Phase II), all on the same Cray "Aries" high speed inter-node network.  The system also has a large Lustre scratch file system and a first-of-its kind NVRAM "burst buffer" storage device.

System Details 

CategoryQuantityDescription
Haswell Cabinets 14 Each cabinet has 3 chassis; each chassis has 16 compute blades, each compute blade has 4 dual socket nodes
Haswell Compute nodes  2,388 Each node has two sockets, each socket is populated with a 16-core Intel® Xeon™ Processor E5-2698 v3 ("Haswell") at 2.3 GHz
    32 cores per node
    Each core supports 2 hyper-threads, and has 2 256-bit-wide vector units
    36.8 Gflops/core; 1.2 TFlops/node; 2.29 PFlops total (theoretical peak)
    Each node has 128 GB DDR4 2133 MHz memory (four 16 GB DIMMs per socket); 203 TB total aggregate memory.
    Each core has its own L1 and L2 caches, with 64 KB (32 KB instruction cache, 32 KB data) and 256 KB, respectively; there is also a 40-MB shared L3 cache per socket
KNL Cabinets 54 Each cabinet has 3 chassis; each chassis has 16 compute blades, each compute blade has 4 nodes
KNL Compute nodes 9,688 Each node is a single-socket Intel® Xeon Phi™ Processor 7250 ("Knights Landing") processor with 68 cores per node @ 1.4 GHz
    Each core has two 512-bit-wide vector processing units. Each core has 4 hardware threads (272 threads total). Two cores form a tile. 
    44 GFlops/core; 3 TFlops/node; 29.1 PFlops total (theoretical peak)
    Each node has 96 GB DDR4 2400 MHz memory, six 16 GB DIMMs (102 GB/s peak bandwidth). Total aggregate memory (combined with MCDRAM) is 1 PB.
    Each node has 16 GB MCDRAM (multi-channel DRAM), > 460 GB/s peak bandwidth
    Each core has its own L1 caches, with 64 KB (32 KB instruction cache, 32 KB data). Each tile (2 cores) shares a 1MB L2 cache. 
Interconnect   Cray Aries with Dragonfly topology with 5.625 TB/s global bandwidth (Phase I).  45.0 TB/s global peak bisection bandwidth (Phase II).
Login nodes  12 Dual socket (16 cores per socket, 32 total cores), 2.3 GHz Intel® Xeon™ Processor E5-2698 v3 ("Haswell") with 512 GB memory.
MOM nodes  -- There are no dedicated MOM nodes on Cori
Shared Root Server Nodes 16  
Lustre Router nodes 130  
DVS Server Nodes 32  
RSIP nodes 10  
Scratch storage system   Cray Sonexion 2000 Lustre appliance. Scratch storage maximum aggregate bandwidth: > 700 GB/sec
Burst Buffer   Cray DataWarp system consisting of 288 Burst Buffer nodes, with maximum aggregate I/O >1.7 TB/s, and >28M IOP/s

 

CategorySoftware NameDescription
Operating System CNL on compute nodes Compute nodes run a lightweight kernel and run-time environment based on the SuSE Linux Enterprise Server (SLES) Linux distribution.
 Full SUSE Linux on Login nodes External login nodes run a standard SLES distribution similar to the internal service nodes. 
Batch System SLURM  

Cori Intel Xeon Phi (KNL) Nodes

Cori Phase 2 has arrived in the summer of 2016! Cori Phase 2 features the Intel Knights Landing manycore architecture. Read More »

Cori Haswell Nodes

The Cori Phase 1 (also known as the "Cori Data Partition") system utilizes Intel Haswell processors and provides approximately the same sustained application performance as the Hopper system. Cori Phase 1 has a number of new features to support data intensive science. Read More »

KNL Processor Modes

The Xeon-Phi "Knights-Landing" 7250 processors in Cori have 68 CPU cores where are organized into 34 "tiles" (each tile comprising two CPU cores and a shared 1MB L2 cache) which are placed in a 2D mesh, connected via an on-chip interconnect as shown in the following figure: As shown in the figure, the KNL processor has 6 DDR channels, with controllers to the right and left of the mesh 8 MCDRAM channels, with controllers spread across 4 "corners" of the mesh.  NUMA on KNL NUMA stands for… Read More »

Interconnect

Cori employs the "Dragonfly" topology for the interconnection network. This topology is a group of interconnected local routers connected to other similar router groups by high speed global links. The groups are arranged such that data transfer from one group to another requires only one route through a global link. This topology is composed of circuit boards, copper and optical cables. Routers (represented by the Aries ASIC) are connected to other routers in the chassis via a backplane. Read More »

Compute Nodes Topology

This page provides compute nodes topology information for some of the Haswell and KNL node types.  You may need to zoom in to view details: Haswell Node KNL,Quad,Cache Node KNL,Quad,Flat Node KNL,SNC2,Cache Node KNL,SNC2,Flat Node KNL,SNC4,Cache Node KNL,SNC4,Flat Node  KNL Quad Cache Nodes:   KNL Quad Flat Nodes:  … Read More »

Burst Buffer Architecture and Software Roadmap

The Burst Buffer on Cori is a layer of non-volatile storage that sits between the a processors' memory and the parallel file system. The burst buffer will serve to accelerate I/O performance of application on Cori. Read More »