NERSCPowering Scientific Discovery Since 1974

Interconnect

Edison employs the "Dragonfly" topology for the interconnection network. This topology is a group of interconnected local routers connected to other similar router groups by high speed global links. The groups are arranged such that data transfer from one group to another requires only one route through a global link.

This topology is composed of circuit boards, copper and optical cables. Routers (represented by the Aries ASIC) are connected to other routers in the chassis via a backplane. Chassis are connected together to form a two-cabinet group (a total of six chassis) using copper cables. Network connections outside the two-cabinet group require a global link. The System uses optical cables for all global links. All two-cabinet groups are directly connected to each other with these cables. See Figure below.

In the Figure above, each router (Rx) is connected to four processors nodes (P). Sixteen blades, each with one router, are connected together at the chassis level by circuit board links (Rank-1 Subtree). Six chassis are connected together to form the two-cabinet group by using copper cabling at the cabinet level (Rank-2 Subtree). Finally, the two-cabinet groups are connected to each other by using optical cables for the global links (Rank-3 Subtree).  Rank 1 routing is characterized by one electrical link between routers. Rank 2 is characterized by three electrical links and Rank 3 is characterized by two optical links between routers.

Rank 1 Detail

Within a chassis, the internal wiring of the backplane connects every Aries ASIC in the chassis to each other. As many as 16 Aries reside in each chassis (one per base board); there is one link between each ASIC. The interconnections of the chassis level ASICs require no cables. This set of interconnections is called the intra-chassis network. See Figure below for the “Intra-chassis Connections (Rank-1)”. 

Rank-2 Details

Copper cables connect each chassis in the two-cabinet group. Each cable contains three links that comprise a total of 18 differential pair wires (36 total). Each cable connects a blade router to a blade router in another chassis, in the same slot location. For example, the router in Slot 1, Chassis 0, Cabinet 0 would be connected to the five blades in the Slot 1 position in the five other chassis (two in the same cabinet and three in the neighboring cabinet). Fully connecting a two-cabinet group requires 240 cables.

Rank-3 Details

The Rank-3 network is used to interconnect two-cabinet groups.  This level of the topology utilizes optical cables that are housed in integrated cable trays above the system cabinets.

The optical connection uses a 24-channel optical cable: 12 channels for transmit and 12 channels for receive. Each cable handles four links (six channels per link), two from each of two Aries ASICs.  There are up to five optical cables associated with every pair of Aries ASICs and a total of 40 optical connections possible for each chassis.  Thus a complete two-cabinet group has up to 240 optical connections.

The Rank-3 connections must form an all-to-all network between the two-cabinet groups.  The width of these connections is variable and can be as few as 1 optical cable between two-cabinet groups and as many as INT(240/(N-1)) where N is the number of two-cabinet groups.  Therefore Edison with 30 cabinets (or 15 two-cabinet groups) can utilize up to 17 optical cables (INT(240/(15-1)) between each pair of two-cabinet groups.