Updating the Testbed Configuration for FY 2004

The testbed system has provided us with a useful facility for developing the benchmark methodology and special benchmark codes for the GUPFS project. It has also been useful in helping to establish the credibility of GUPFS with technology vendors, and in building relationships with various technology vendors. However, with the inexorable progression of technical advancements, it became apparent that the testbed was inadequate in size for conducting the types and levels of technology evaluations needed for the GUPFS project in FY 2004

Technological advancements over the last year have outstripped the ability of the existing testbed to incorporate them. Experience with the testbed and attempts to integrate new storage and fabric technology into it demonstrated that more nodes were needed in the testbed to allow emerging advanced technologies to be integrated into it for evaluation, even in the near term. Given the technologies that the GUPFS project plans to begin evaluating in FY 2004, it was clear that the existing testbed could not accommodate their inclusion.

In addition to the testbed  being limited in accommodating new technologies, practical experience during FY 2003 indicated to us that the testbed could be improved in a number of ways that would increase the speed at which file system evaluations could be conducted, in conjunction with specific combinations of fabrics and high-performance storage. Foremost of these was that the testbed should be able to conduct multiple, simultaneous, independent evaluations with node sets of various sizes. Next was that the testbed be able to reconfigure the fabric and storage connections much more easily. Other important areas to improve were to greatly increase the Gigabit Ethernet connectivity for iSCSI and cross-fabric testing, and to only have a single node type used in testing. Based on these needs, we designed a testbed upgrade to facilitate more simultaneous evaluations and much more rapid and easy reconfiguration. The design considerations for the update to the testbed and discussed in the following section.

 

We completed the improved testbed system design at the end of the third quarter of FY 2003. We then identified, tested, and procured the components during the fourth quarter.  These were    assembled and integrated them into the existing testbed system at the end of the fourth quarter of FY 2003 in preparation for the planned FY 2004 activities. The configuration of the expanded testbed system is discussed in detail later in this section.

1.    Updated Testbed Design Considerations

The design of the upgrade to the GUPFS project testbed system was predicated on a number of factors. These included (1) the lessons learned and (2) limitations encountered from using the FY 2003 testbed to integrate and test new technologies, and (3) the new and emerging technologies expected to be investigated in the next year; these are discussed below, along with (4) a brief review of other factors affecting the upgrade.

Lessons Learned

The FY 2003 testbed included many design features that improved its usability and utility over the FY 2002 testbed. However, a number of lessons were learned through the use of the updated FY 2003 testbed and the evaluations conducted on it. These directly impacted the design of the testbed upgrade for FY 2004. The major lessons learned are:

·          Multiple evaluations need to be done simultaneously. An evaluation of a technology component is complex, and time consuming to both set up and conduct. The reality of the situation is that a technology component cannot be evaluated in isolation. In general, one of the GUPFS component technology types can only be evaluate in conjunction with one or more of the other component types. For example, to evaluate a file system, it is necessary to test it in conjunction with storage and some fabric or interconnect connecting the clients to the storage, In addition, each component usually has required versions of software, necessitating client systems being configured with specific OS and driver versions and specific fabric connectivity. Setting up the required configurations to conduct specific evaluations is time consuming, often in the order of several weeks or a month. Often, a single evaluation takes several months and requires dedicated resources. Because of the number of important component technology issues that need to be investigated, particularly in the file system arena and multi-cluster configurations, and because of the rapidly changing component technologies, it is vital to have enough resources to be able to have extended unrelated evaluations in progress simultaneously.

·          Only a single type of compute node should be used. The FY 2003 testbed contained two types of compute nodes — the 18 dual Pentium-4 compute nodes and the 5 legacy dual Pentium-3 compute nodes. Because of the limited number of available compute nodes and the high volume of items to evaluate or investigate, it was necessary to use both types of nodes for evaluations. This predictably caused problems in a number of areas. First, the different node architectures required distinct software configurations be built, installed, and tested for each type, thereby increasing the administrative burden for the testbed. Secondly, the hardware components and performance differences between the types of nodes made it very difficult to compare results obtained from evaluations, and made evaluations using both types of nodes at the same time too difficult to understand. The solution to this difficulty is to standardize on a single node type for use as compute nodes in evaluations, and to obtain adequate numbers of them to conduct the necessary number of evaluations simultaneously at adequate scale.

·          A separate interactive node is needed. The FY 2003 testbed used a single dual Pentium-3 node as both a management node for running and maintaining the testbed environment and as an interactive node for the project members to log into and from which to conduct tests and evaluations. With the increased size and complexity of the testbed and the increased number of evaluations in progress at one time, it became apparent to us that a single node of this type could not perform both functions without impacting one or the other. In addition to periodic failures and substantial delays caused by overloading the management node, the launching of long-running benchmarks frequently prevented timely maintenance activities, such as rebooting the management node to clear software problems or activate patches. Another reason to separate the management and interactive functionality was to reduce the possibility of the management node accidentally being destroyed by the setting up evaluations and running of benchmarks by the project members, both activities that frequently required running with elevated privileges, leading to a number of close calls. Management node functionality is much more difficult and time consuming to configure and install than interactive node functionality. The necessity of separating the management and interactive node functionality contributed to the decision to stop using any of the Pentium-3 nodes as compute nodes, and to assign them to testbed support roles.

 

Limitations

In addition to lessons learned, the design of the FY 2004 testbed was also influenced by certain limitations encountered while using the FY 2003 testbed. These limitations include:

·          The EMC CX600 did not meet expectations. The EMC CX 600 storage device did not live up to expectations regarding performance scalability. This was partly the result of unexpected architectural limitations and partly the result of configuration constraints. As a consequence, neither the expected aggregate bandwidth nor the desired scalability was achieved. This made the GUPFS project dependent on other scalable storage that was being evaluated, such as the Yotta Yotta NetStorager and the 3PARdata Inserv. With the completion of the 3PARdata evaluation and the Yotta Yotta extended beta test at the beginning of the 4th quarter of FY 2003, the GUPFS project faced the prospect of not having storage that was of high enough performance or sufficiently scalable to use for the file system evaluation planned for the end of FY 2003 and for all of FY 2004.

·          The number of Gigabit Ethernet switch ports was inadequate. The GUPFS testbed’s Gigabit Ethernet fabric quickly became limited by the 48 available switch ports. The available ports were quickly consumed by the testbed systems, inter-switch links between the two Gigabit switches, original iSCSI router and Intel iSCSI HBA connections, and the InfiniCon InfiniBand to Gigabit Ethernet bridges. With the introduction of additional equipment requiring Gigabit Ethernet connections, the number of available ports was oversubscribed by at least a factor of two. This resulted in the serialization of evaluations and made it necessary to disconnect various equipment in order to connect and test other equipment. The additional equipment included Topspin InfiniBand to Gigabit Ethernet fabric bridges, Panasas storage devices, Adaptec iSCSI HBAs and TOE cards, and the inter-switch links to the Alvarez management Ethernet switch. Based on the need to maintain adequate inter-switch bandwidth, a 3x expansion of the Gigabit switch ports was needed.

·          Reconfiguring the physical connections between storage, fabric elements, and client systems became extremely difficult. In order conduct evaluations of various file systems, fabric components and bridges, and storage combinations, and to conduct evaluations of various loaner equipment, it was necessary to change the physical fiber optical connections of the equipment connected to the three 16 port Fibre Channel switches in order to connect the correct set of components. This was made extremely difficult by the rigidity of the bundles of fiber cables and the fragility of the connectors. Moving a fiber between one switch and another often required major efforts to obtain adequate slack to permit the connector to plug into another switch. This was particularly a problem when we were evaluating new equipment such as Fibre Channel switches, which might be physically mounted in a different cabinet. Making such connections required that substantial time be devoted to rebundling fibers or stringing new ones. In addition, replugging the connectors exposed them to mechanical failure (the 2 Gb/s SFPs are especially fragile), and to contamination of the optics with dust. Another problem with making changes to the fiber configuration was that it soon be came very difficult to determine what was connected to what and which fibers were active. A fiber patch panel is needed to resolve these problems.

·          A single dedicated metadata server node is not enough. It became apparent that a single special-purpose node acting as a dedicated metadata server was inadequate. Nearly all of the shared file systems being tested required either a metadata server or a centralized lock manager. As a consequence, testing of these file systems became serialized because of the single metadata/lock server. Frequently, more than one of these file systems was in some stage of the installation and evaluation cycle; sometimes a file system would be undergoing several tests at once, each instance having different hardware and/or software configurations. This required either very careful alternating of test segments, or the suborning other nodes to become additional metadata servers. Using other testbed nodes as secondary metadata servers impacted other activities by limiting the number of nodes available to them. Similar constraints applied to testing configurations supporting metadata/lock server redundant operation and failover. At least one more special purpose Pentium-4 node with an identical configuration needs to be dedicated to the metadata/lock server role.

·          A full complement of eight 4U Pentium-4 nodes is needed. The GUPFS testbed only had six 4U Pentium-4 nodes. Most new technologies are initially implemented on standard height (4U) PCI/PCI-X cards, and only in the second or third generation of the technology do low-profile (2U) cards become available. The initial Gigabit Ethernet, 1x InfiniBand, 4x InfiniBand, Myrinet 2000, iSCSI HBA, Gigabit Ethernet TOE cards, and 1 Gb/s Fibre Channel cards were all standard height cards, requiring 4U cases. Most fabrics provide switching capabilities as powers of 2, and frequently with 8 ports as a minimum, leading to a standard purchase of an 8-port switch and 8-host interface cards, as 4-host cards provide little in the way of insights about scalability. Earlier constraints limited the testbed to six 4U Pentium-4 nodes, preventing eight-way evaluations when standard height interface cards were required. Adding two more 4U Pentium-4 nodes would enable eight-way evaluations requiring standard height interface cards to be conducted, permitting more direct comparisons with other evaluations.

New and Emerging Technologies

The design changes for the FY 2004 testbed were influenced by the new and emerging technologies impacting the GUPFS solution, which are expected to be available for evaluation during the coming year. In this regard, several important issues need to be investigated in the near term, including:

·          Conducting cross-platform file system tests. The GUPFS project plans to conduct cross-platform file system tests to explore functionality and deployment issues in a heterogeneous environment that involves multiple hardware and different OS architectures, which is designed to mimic the NERSC environment in which GUPFS will be deployed. These tests will require either incorporation of additional systems into the testbed, or opening up the testbed to other NERSC systems, both of which require additional fabric-switching capabilities.

·          Conducting multiple cluster file system tests. The GUPFS project plans to conduct file system tests involving multiple clusters accessing the same file system and storage simultaneously, as is expected in the NERSC environment at deployment. PDSF, Alvarez, and Dev2 are likely candidate peer systems. This will require opening up and connecting the testbed to these other NERSC systems.

·          Evaluating 4 Gb/s and 10 Gb/s Fibre Channel. Both 4 Gb/s and 10 Gb/s production quality Fibre Channel equipment will be becoming available in the time frame of the initial phase of GUPFS deployment. Because of the anticipated aggregate performance needs for production use — and as it is likely that the backend storage controllers, if not the storage itself, for most shared file system solutions will be Fibre Channel connected — these technologies are likely to be important to a successful GUPFS deployment. As such, they need to be evaluated and understood.

·          Evaluating 10 Gb/s Ethernet. The 10 Gb/s Ethernet technology is expected to be deployed during FY 2004. Because it is most likely that PDSF will be accessing the GUPFS file system over the Ethernet, 10 Gb/s Ethernet is a likely component of the deployed GUPFS solution and needs to be understood in a storage fabric context.

·          Evaluating Panasas file system and storage. The Panasas ActiveScale File System is a very interesting object-based file system implemented over the Ethernet. Architecturally, it is quite similar to Lustre, but is more standards based, being implemented with a variant of iSCSI. The Panasas file system offering is integrated with Ethernet-attached storage devices specific to the file system, and can be accessed either through integrated NFS and CIFS gateways, or as part of a shared file system through the DirectFlow client software. The Panasas file system should be accessible over any IP-based fabric that can bridge to the Ethernet. This is a promising candidate file system and needs to be evaluated for GUPFS.

·          Evaluating the IBRIX file system. The IBRIX file system is an interesting potential GUPFS file system solution that is based on federating the individual file systems of storage engines (SEs). The IBRIX file system is distributed over IP networks. It utilizes back-end SAN based storage. The IBRIX file system was originally scheduled to be available in preproduction versions for evaluation in FY 2003. This schedule has slipped into FY 2004.

·          Evaluating the IBM TotalStorage SANFS (StorageTank) file system. The IBM StorageTank file system was renamed the TotalStorage SANFS file system at the end of FY 2003. SANFS is expected to become available as a product in the first half of FY 2004. It targets very large numbers of client systems, and supports multiple hardware architectures and operating systems. It uses metadata servers in conjunction with block storage accessed via iSCSI over IP networks  (making it largely fabric agnostic), or accessed directly by Fibre Channel. The ability to access remote tanks over the WAN is being developed. SANFS is an extremely promising GUPFS candidate, although quite young, and needs to be investigated thoroughly.

·          Further iSCSI investigations. The iSCSI protocol is making an appearance in several promising shared file systems. It also provides a cheap mechanism for accessing block storage over inexpensive fabrics, although at the expense of the higher processor overhead. With the ability of most fabrics and interconnects to perform IP transfers, and with the ability of most fabrics to bridge to the Ethernet, iSCSI may facilitate the implementation of heterogeneous fabrics directly tied into cluster interconnects. However, it needs much more investigation as its availability and use expand.

Additional Considerations

Other considerations that affected the design of the FY 2004 testbed include:

·          InfiniBand technology refresh needed.  The testbed InfiniBand technology needs to be refreshed. Current second-generation 4x InfiniBand equipment, particularly HCAs and Fibre Channel and Gigabit gateways need to be acquired. The original 1x IB equipment and the loaner first generation 4x IB equipment are no longer supported.

·          Multiple management nodes needed. The testbed needs at least two management nodes. The management node is currently a single central point of failure in the testbed, and is extremely difficult and complex to configure. A second management node is needed to ensure the testbed and GUPFS project evaluations and investigations can continue if the existing management node fails. In addition, the availability of a second management node would allow the management node software versions and configurations to be upgraded one at a time without disruption.

·          Gigabit Ethernet emerging as the standard fabric-to-others bridge. Gigabit Ethernet is emerging as the common fabric to which all other fabrics and interconnects bridge. Because of this, a large number of Gigabit Ethernet switch ports are needed in the testbed, particularly in conjunction with the iSCSI, cross-platform, and multi-cluster file system testing planned for FY 2004.

·          The Myrinet to Gigabit Ethernet bridge would expand the file systems that can be tested with Alvarez. Myricom announced a Gigabit Ethernet bridge blade for their Myrinet switches, with 8 Gigabit Ethernet ports. With such a fabric bridge, the number of shared file systems that could be tested on and in conjunction with the LBNL Alvarez Linux cluster increases substantially. IP-based file systems such as Panasas and IBRIX could be evaluated for scalability. Block-based file systems, such as StorNext, could be tested for scalability using iSCSI bridged to storage in the GUPFS testbed. In addition, a Myrinet upgrade to the Rev D card in late FY 2003 allowed low-profile PCI-X Myrinet 2000 cards to be installed, enabling the Myrinet network to be installed in 2U nodes, thus freeing up the 4U nodes for other uses and enabling the full use of the Myrinet switch with 8 hosts.

2.    Updated Testbed Configuration for FY 2004

Design of the updated testbed configuration was completed at the end of the third quarter of FY 2003. This design was based on all of the considerations presented in the previous section. The central tenet of the updated configuration was to increase the number of simultaneous evaluations that could be conducted, increase the maximum scale of these evaluations, and simplify the process of physically reconfiguring the connectivity from testbed nodes to storage devices through various fabric components.

The updated configuration expanded the total number of Pentium-4 nodes from 22 to a total of 36. As in FY 2003, four of these Pentium-4 nodes were retained as dedicated special purpose nodes, although there were some changes in assigned functions. The remaining 32 Pentium-4 nodes were assigned as compute nodes dedicated to running benchmarks and conducting other investigations. To facilitate conducting multiple simultaneous independent evaluations, the 32 compute nodes were logically partitioned into four sets of eight. This logical partitioning allows up to four independent 8-way investigations to be conducted simultaneously, or in various size combinations, such as one 32-way, two 16-way, or one 16-way and two 8-way tests.

While the partitioning of the compute nodes into groups was at a logical level, there were some physical characteristics related to their partitioning. Each group of eight compute nodes was connected to a separate Dell Power Connect Gigabit Ethernet switch, which allowed the nodes in the group maximum communication performance among themselves. The Dell switches for each of the groups were then connected by four-way trunks to a central Extreme 7i switch. This allowed nodes in any of the groups to communicate with each other, but reduced aggregate bandwidth and increased latency. Another physical characteristic related to the partitioning of the nodes was the additional PCI-X fabric interface cards each node had. For a variety of reasons, the GUPFS project conducts evaluations of fabric interfaces/interconnects with a minimum of eight hosts for each fabric. In addition to Fibre Channel interfaces, present on all nodes except the management nodes, the GUPFS testbed contains three other sets of high-performance fabrics, each of which is connected to eight compute nodes. These additional fabrics are a Myrinet 2000 interconnect, and after the testbed upgrade, two 4x InfiniBand fabrics from different vendors. The three sets of nodes with extra fabric connections are each put into logically separate groups to facilitate the independent testing of these extra fabrics.

An additional element of the updated configuration was the dispensation of the original six Pentium-3 nodes. One of these has always been used as the testbed management node. The remaining five were used as compute nodes in FY 2002 and 2003, although in an auxiliary role during FY 2003. With the addition of more Pentium-4 nodes in the updated configuration, it was possible to stop using the Pentium-3 nodes as compute nodes and assign them to other supporting duties. One was to become a second testbed management node for redundancy and simplifying upgrades. Another was to become a dedicated interactive node, offloading this function from the management nodes for the reasons discussed earlier. The remaining three Pentium-3 nodes became dedicated development nodes for benchmark and analysis code development, and possible auxiliary HPSS integration investigation roles.

Another part of the upgraded configuration included the installation of a fiber-optic patch panel, allowing all fiber-optic ports to be centrally connected in a static configuration and then cross connected as necessary using easily movable fiber-optic patch cords. All fiber-optical Gigabit Ethernet, Myrinet, and Fibre Channel host adapters, switch ports, and device connections were hardwired into the central patch panel to simplify physical reconfiguration of the fabrics and connections.

The other major element of the updated configuration included purchasing the Yotta Yotta NetStorager as the standard high-performance storage to be used in evaluations in lieu of the disappointing CX 600, the previously mentioned Gigabit Ethernet switching capacity, additional Fibre Channel switching capacity, and 4x InfiniBand technology refresh from two vendors. A front view of the updated testbed for FY 2004 appears as Figure 1. A rear view of the testbed, showing the nodes and cable connections, appears in Figure 2. The updated FY 2004 testbed configuration is shown in Figure 3. The Port Fibre optical patch panel is shown in Figure 4.

 

 

Figure 1. The FY 2004 testbed, with the Refurbished NetStorager in front.

The following major components were added to the testbed as part of its technology upgrade for FY 2004:

·       Fourteen additional dual Pentium-4 nodes: 12 in 2U cases and 2 in 4U cases (these nodes were identical to those already in the testbed.

·       One 64-port 2 Gb/s Fibre Channel Qlogic SANbox2-64 switch with 48 ports

·       Five 24-port Dell Power Connect 5224 Gigabit Ethernet switches

·       One Yotta Yotta NetStorager GSX 2400 Disk Storage Subsystem

·       InfiniCon ISIS InfinIO 7000 switch and fabric bridge 4x InfiniBand upgrade

·       One Topspin TS90 4x InfiniBand switch and Fibre Channel gateway

·       One Myrinet 2000 MS-SW16-8E switch line card with 8 Gigabit Ethernet ports

·       A Fiber Optic patch panel and cables for Gigabit Ethernet, Myrinet, and Fibre Channel

 

Figure 2. Rear view of the FY2004 testbed.

The new Pentium-4 nodes obtained as part of the upgrade were as identical as possible to those obtained the previous year. They were configured with the same motherboard, the same quantity and performance grade of memory, the same 2.2 GHz Xeon CPU, and the same Intel Gigabit Ethernet and Qlogic Fibre Channel PCI-X cards. All of these components differed from the originals only in revision numbers. The nodes were all equipped with the same speed (10,000 RPM) and capacity (36 GB), and U160 SCSI disks, but from a different manufacturer.

Special efforts were made to configure the new Pentium-4 nodes to be identical to those obtained earlier. This was done to ensure uniformity of performance so that results obtained from both sets would be comparable and they could be intermixed without affecting evaluation results. The motherboard proved to be the most difficult to obtain as it was being phased out. However, a newer revision of the motherboard was acquired that showed nearly identical performance, which allowed the new and existing Pentium-4 nodes to be intermixed with negligible impact on the benchmark results.

 

 

Figure 3. Updated GUPFS testbed configuration for FY 2004.

The major components of the updated GUPFS testbed for FY 2004 included:

System nodes

·       36 dual Pentium-4 nodes: 28 in 2U cases and six in 4U cases; 32 for use as compute nodes and 4 for use as special-purpose nodes

·       6 dual Pentium-3 nodes in 4U cases, used as management, interactive, and auxiliary testbed support nodes

Fabric

·       One 240 connector Fiber Optical patch panel with SC connectors and patch cables (see Figure 4)

·       Ethernet

o       One 32-port Extreme 7i Gigabit Ethernet switch

o       One 16-port Extreme 5i Gigabit Ethernet switch

o       Five 24-port Dell Power Connect 5224 Gigabit Ethernet switches

o       Two 10/100 Ethernet switches for system management

·       Fibre Channel

o       One 48-port 2 Gb/s Fibre Channel Switch (SanBox2-64)

o       One 16-port 2 Gb/s Fibre Channel Switch (Brocade 3800)

o       One 16-port 1 Gb/s Fibre Channel Switch (Brocade 2800)

o       One Cisco SN5428 iSCSI Router fabric bridge to Ethernet

·       Myrinet

o       One Myrinet 2000 8-port switch with 8 Revision D host interface cards

o       One Myrinet 2000 MS-SW16-8E switch card with 8 Gigabit Ethernet ports for bridging between Myrinet and Gigabit Ethernet

·       InfiniBand

o       One InfiniCon ISIS InfinIO 7000 4x InfiniBand switch, 8 4x HCA host adapters, and single fabric bridge modules for Fibre Channel and Gigabit Ethernet

o       One Topspin TS90 4x InfiniBand switch, 8 4x HCA host adapters, and single fabric bridge modules for Fibre Channel and Gigabit Ethernet

Storage

·       Yotta Yotta NetStorager GSX 2400

·       EMC CLARiiON CX600 disk subsystem

·       Dot Hill 7124 RAID disk subsystem

·       Silicon Gear Mercury II RAID subsystem

·       Chaparral A8526 RAID subsystem with attached storage

The expanded testbed, with its increased scale, updated and new technologies, and features to support easier reconfiguration, will facilitate evaluations to be conducted during FY 2004. The increased scale of the testbed will enable both more simultaneous small-scale initial evaluations and single larger scale evaluations. The testbed’s multiple fabrics and the bridges between them will allow the issues involving heterogeneous fabric environments, such as those expected at NERSC, to be investigated and understood.

 

 

Figure 4. 240-Port Fibre optical patch panel.

The increased number of testbed nodes will also facilitate the conducting of cross-platform and multiple OS tests of promising file systems supporting such capabilities. A great deal of important information and experience stands to be gained through these tests, which will explore the issues associated with deployment in a heterogeneous environment, as is expected at NERSC. The increased Gigabit Ethernet fabric switching capacity and additional improved fabric bridging capabilities will further facilitate this testing and will enable multiple-cluster testing to be conducted with both the Alvarez cluster and PDSF systems. This will allow phased deployment to be simulated, and we can then begin addressing the networking issues associated with deployment.