NERSCPowering Scientific Discovery Since 1974

Interconnect

jaguar xt4

Hopper's "Gemini" network is connected in a 3D torus.

 

Description

Hopper's compute nodes are connected via a custom high-bandwidth, low-latency network provided by Cray. The connectivity is in the form of a "mesh" in which each node is connected to other nearby nodes like strands in a fishing net, except that the mesh extends in three dimensions. Each network node handles not only data destined for itself, but also data to be relayed to other nodes. Nodes at the "edges" of the mesh network are connected to nodes at the other edge to form a 3-D torus.

The custom chips that route communication over the network are known as "Gemini" and the entire network is often referred to as the "Cray Gemini Network." Each network node is referred to as a "link."

The dimensions of Hopper's torus are 17 x 8 x 24. Although topologically a regular 3-D torus, the network is not necessarily wired so that consecutively named nodes and "nearby" on the network. In fact, the actual wiring is very complex and optimized for application performance for the system as a whole.

The entire map of the node IDs, cabniet coordinates and torus coordinates can be found in the spreadsheet, hoppertopo.xls, provided at the bottom of the web page. There is a simple utility available on Hopper that can show this information, too:

% topolcoords -n all
NID   TYPE   CABX CABY CAGE SLOT NODE    X    Y    Z
 -----------------------------------------------------
    0  service    0    0    0    0    0    0    0    0
    1  service    0    0    0    0    1    0    0    0
    2  service    0    0    0    1    0    0    0    1
3  service    0    0    0    1    1    0    0    1
...
6524  compute    1    1    2    1    2    1    7    1
6525  compute    1    1    2    1    3    1    7    1
 6526  compute    1    1    2    0    2    1    7    0
 6527  compute    1    1    2    0    3    1    7    0

where CABX and CABY are cabinet column and row coordinates; CAGE, SLOT, and NODE are for the cage, slot and node numbers within a cabinet. Hopper cabinets are arranged into 4 rows and 17 columns; there are 3 cages in a cabinet, 8 slots per cage, and 4 nodes per slot. X, Y and Z are torus coordinates. If you add or subtract 1 from any of X, Y or Z, there is a physical connection. For information about other display options, run the utility with no flag.

Each pair of Hopper nodes (containing 48 cores) is connected to one Gemini Application-Specific Integrated Circuit (ASIC). Each Gemini ASIC provides two network interface controllers (NICs), and a 48-port router. Each NIC has its own HyperTransport™ 3  interface for connection to the compute sockets.

Gemini Characteristics

  • The Gemini NIC can transfer 64 bytes of data in each direction every 5 cycles, which corresponds to a maximum bandwidth per direction of 64 x 650 / 5 = 8.3 GBytes/s.
  • Each Gemini NIC has 10 network connections, two each in +X, -X, +Z, -Z, and one +Y and one -Y
  • Each of the 10 Gemini torus connections is comprised of 12 lanes in each direction operating at 3.125 to 6.25 GHz, yielding link
    bandwidths of 4.68 to 9.375 GBytes/sec per direction, although MPI can see only 2.9 to 5.8 GB/sec per direction.
  • Link-level reliability and adaptive routing
  • Maximum injection bandwidth per node is 20 GB/s.
  • In addition to supporting MPI, the Gemini interconnect has hardware support for partitioned global address space (PGAS) programming models including Unified Parallel C (UPC), Co-array Fortran (CAF) and Chapel. Gemini allows remote-node references to be pipelined in these programming models, which can result in orders-of-magnitude performance improvement over library-based message passing models.
  • Supports millions of MPI messages per second
  • Internode latency is about 1.27 μs (nearest nodes pair) and 3.88 μs (farthest nodes pair) on a quiet network.  Latency between cores is <1 μs for two cores connected to the same Gemini chip.

 

Cray Gemini Interconnect with an "exploded" view of the ASIC

 

WIring

Wiring up a Cray XE6

Downloads