NERSCPowering Scientific Discovery Since 1974

New Features of the Edison XC30

While the Edison and Hopper systems have similar programming environments and software, there are some key architectural differences between the two systems. This page describes those differences.

Compute nodes 

Edison and Hopper both have a total of 24 cores on each compute node.  Edison, like Hopper, has two sockets on each compute node, but instead of four "NUMA" memory domains, Edison has only two.  Edison uses Intel processors, unlike Hopper which has processors from AMD. Edison's processors have Intel Hyper-Threading (HT) enabled, which means you can run with 48 logical cores per node. At run time you can decide to run with 24 cores per node (the default setting) or 48 logical cores per node. 

EdisonHopper

24 cores per node  (48 logical cores with Hyper-Threading). Dual-socket, 12-core processor node, Intel Xeon "Ivy Bridge" @ 2.4 GHz

24 cores per node. Dual-socket, 12-core processor node, AMD Opteron @2.1 GHz
64 GB memory per node (DDR3 memory at 1600 MHz);  2.67GB/core.
32 GB memory per node (DDR3 1333 MHz memory); 1.33 GB/core.
5192 nodes   6384 nodes

Interconnect

Edison uses Cray Aries interconnect for inter-node communication, while Hopper uses the Cray Gemini network.  Aries provides a higher bandwidth, lower latency interconnect than Gemini, and should exhibit reduced network congestion. Edison's Aries network is connected through a "Dragonfly" topology, compared to Hopper's torus network. See the Configuration section for more details.

External Login Nodes

Like Hopper, Edison has 12 login nodes that are "external" to the main compute portion of the system. Since the login nodes are external, you can login, access file systesms and submit jobs when the main compute protion of the sytem is down for maintenance. The login nodes on Edison have 512 GB of memory, compared to 128 GB on Hopper.

EdisonHopper
12 quad-socket, quad-core nodes (16 cores per node w/ Hyper-Threading 32 logical cores), Intel Xeon "Sandy Bridge" @ 2.0 GHz 12 quad-socket, quad-core nodes (16 cores per node), AMD Opteron @2.0 GHz
512 GB memory per login node  128 GB memory per login node

Scratch File Systems

Edison has 7.5 PB of scratch spread over three disk partitions. The aggregate I/O bandwidth is 165 GB/sec, while Hopper has 2 PB disk with 70 GB/s I/O bandwidth.  All users have a scratch area defined by the environment variable $SCRATCH on one of two smaller scratch partitions.  A third larger file system is dedicated to users who have a demonstrated need for large I/O bandwidth. Users can request access to /scratch3 via a web form. Edison uses Cray Sonexion lustre storage system. The default scratch quota is 10TB.

  File systems Size (PB) Aggregate Peak Performance (GB/s) # IO Servers (OSSs) OSTs File System Software Disk Array Vendor User quota(TB)
 Edison   /scratch1 2.1 48 24 96 Lustre Cray  10
/scratch2 2.1 48 24 96 Lustre Cray  10
/scratch3 3.2 72 36 144 Lustre Cray  --
Hopper  /scratch 1 36 26 156 Lustre  LSI  5