table of contents advances in computational science research news the NERSC center

the nersc center

advanced development

Architecture Benchmarking and Evaluation

With the re-emergence of viable vector computing systems such as the Earth Simulator and the Cray X1, and with IBM and DOE’s BlueGene/L taking the number one spot on the TOP500 list of the world’s fastest computers, there is renewed debate about which architecture is best suited for running large-scale scientific applications.

In order to cut through conflicting claims, a team of researchers from Berkeley Lab’s Computational Research (CRD) and NERSC Center divisions have been putting various architectures through their paces, running benchmarks as well as scientific applications key to DOE research programs. The team (Figure 10) includes Lenny Oliker, Julian Borrill, Andrew Canning, and John Shalf of CRD; Jonathan Carter and David Skinner of NERSC; and Stephane Ethier of Princeton Plasma Physics Laboratory.

       
Julian Borrill Andrew Canning Jonathan Carter Stephane
Ethier (PPPL)
       
Lenny Oliker John Shalf David Skinner  
Figure 10. NERSC’s architecture benchmarking and evaluation team analyzes the complex interplay of scientific codes and system designs.


In their first study of 2004,5 the team put five different systems through their paces, running four different scientific applications key to DOE research programs. As part of the effort, the group became the first international team to conduct a performance evaluation study of the 5,120-processor Earth Simulator. The team also assessed the performance of:

“This effort relates to the fact that the gap between peak and actual performance for scientific codes keeps growing,” said team leader Lenny Oliker. “Because of the increasing cost and complexity of HPC systems, it is critical to determine which classes of applications are best suited for a given architecture.”

The four applications and research areas selected by the team for the evaluation were:

“The four applications successfully ran on the Earth Simulator with high parallel efficiency,” Oliker said. “And they ran faster than on any other measured architecture — generally by a large margin.” However, Oliker added, only codes that scale well and are suited to the vector architecture may be run on the Earth Simulator. “Vector architectures are extremely powerful for the set of applications that map well to those architectures,” Oliker said. “But if even a small part of the code is not vectorized, overall performance degrades rapidly.”

As with most scientific inquiries, the ultimate solution to the problem is neither simple nor straightforward.

“We’re at a point where no single architecture is well suited to the full spectrum of scientific applications,” Oliker said. “One size does not fit all, so we need a range of systems. It’s conceivable that future supercomputers would have heterogeneous architectures within a single system, with different sections of a code running on different components.”

One of the codes the group intended to run in this study — MADCAP, the Microwave Anisotropy Dataset Computational Analysis Package — did not scale well enough to be used on the Earth Simulator. MADCAP, developed by Julian Borrill, is a parallel implementation of cosmic microwave background map-making and power spectrum estimation algorithms. Since MADCAP has high I/O requirements, its performance was hampered by the lack of a fast global file system on the Earth Simulator.

Undeterred, the team re-tuned MADCAP and returned to Japan to try again. The resulting paper6 found that the Cray X1 had the best runtimes but suffered the lowest parallel efficiency. The Earth Simulator and IBM Power3 demonstrated the best scalability, and the code achieved the highest percentage of peak on the Power3. The paper concluded, “Our results highlight the complex interplay between the problem size, architectural paradigm, interconnect, and vendor-supplied numerical libraries, while isolating the I/O filesystem as the key bottleneck across all the platforms.”

As for BlueGene/L, currently the world’s fastest supercomputer, David Skinner is serving as Berkeley Lab’s representative to a new BlueGene/L consortium led by Argonne National Laboratory. (The first Blue Gene system is being installed at Lawrence Livermore National Laboratory.) The consortium aims to pull together a group of institutions active in HPC research, collectively building a community focused on the BlueGene family as a next step towards petascale computing. This consortium will work together to develop or port BlueGene applications and system software, conduct detailed performance analysis on applications, develop mutual training and support mechanisms, and contribute to future platform directions.

Science-Driven Architecture Ideas Go into Production

Two years after NERSC led a team from seven national laboratories to propose a new HPC architecture designed specifically to meet the needs of computational scientists, such an architecture is being used for the ASC Purple supercomputer being built by IBM for Lawrence Livermore National Laboratory (LLNL).

The 2003 white paper, entitled “Creating Science-Driven Computer Architecture: A New Path to Scientific Leadership,”7 was submitted to the High-End Computing Revitalization Task Force (HECRTF), whose goal was to re-establish the United States as the clear leader in high-performance computing. The white paper called for moving away from dependence on hardware that is designed and optimized for commercial applications and to create a new class of computational capability in the United States that is optimal for science — a strategy whose broad outline was adopted in HECRTF’s final report.8

This comprehensive strategy includes development partnerships with multiple vendors, in which teams of scientific applications specialists and computer scientists will work with computer architects from major U.S. vendors to create hardware and software environments that will allow scientists to extract the maximum performance and capability from the hardware. The white paper recommended pursuing three different options to determine the best price-performance in a scientific computing environment.

The second option called for using commercial microprocessors in a new architecture known as ViVA or Virtual Vector Architecture. ViVA was based on enhancing IBM’s existing Power5 plan in order to deploy a system with high sustained performance (up to one-third of peak) running real scientific codes. Each node would consist of eight single-core CPUs that would provide double the memory bandwidth of the standard Power5 CPUs and would have their own dedicated L1, L2, and L3 caches. The CPUs would have a peak performance of roughly 8–10 Gflop/s.

This architecture caught the attention of ASC (Advanced Simulation and Computing) managers at LLNL, who were working with IBM to design and build ASC Purple. When the final architecture of ASC Purple was agreed upon, it was designed around the eight-way node. As a result, the system is expected to be less expensive than originally expected, as well as more effective for carrying out its intended mission.

“We owe you a debt of gratitude for conceiving the eight-way node with IBM and pushing so strongly for it with IBM,” wrote Michael McCoy of LLNL when informing NERSC Center General Manager Bill Kramer of the ASC Purple decision. “This is a clear example of ASC benefiting from Office of Science — and NERSC in particular — working the system for better solutions for science. Also, the cost of the system is much lower than the original thanks in part to the new node. We all owe NERSC thanks for vision and energy in this singular effort.”

The 12,544-processor ASC Purple system is scheduled to be installed at Lawrence Livermore in July 2005.

Streamlining File Storage

In 2001, NERSC began a project to deploy a system to streamline its file storage system using existing and emerging technologies. The Global Unified Parallel File System (GUPFS) project aims to provide a scalable, high-performance, high-bandwidth, shared file system for use by all of NERSC’s high-performance production computational systems. GUPFS will provide unified file namespace for these systems and will be integrated with HPSS. Storage servers, accessing the consolidated storage through the GUPFS shared-disk file systems, will provide hierarchical storage management, backup, and archival services. An additional goal is to distribute GUPFS-based file systems to geographically remote facilities as native file systems over the DOE Science Grid.

When fully implemented, GUPFS will eliminate unnecessary data replication, simplify the user environment, provide better distribution of storage resources, and permit the management of storage as a separate entity while minimizing impacts on the NERSC computational systems.

The major enabling components of this envisioned environment are a high-performance shared file system and cost-effective, high performance storage-area networks (SANs) and emerging alternative fabrics. These technologies, while evolving rapidly, are generally not targeted towards the needs of high-performance scientific computing. The GUPFS project is working with vendors to incorporate HPC requirements into their products.

During the first two years of the project, the GUPFS team, led by Greg Butler of the Advanced Systems Group, tested and investigated shared file systems, SAN technologies, and other components of the GUPFS environment. The team has successfully assembled a complex testbed simulating the envisioned NERSC storage environment, with which they have developed the knowledge base needed to select and deploy production-quality solutions.

Now, during the third year, their focus has shifted from component evaluation to deployment planning — evaluating solutions, systems, and deployment scenarios with an eye toward issuing formal requests for information and proposals. The team is also narrowing its focus to candidate file systems.

“Fabric and storage technologies have been advancing and are not major concerns,” Butler said. “But file systems, while they are improving, still need work to meet the needs of parallel production environments, especially in terms of stability, parallel I/O performance and functionality, and scalability. File systems also need to support more platforms and operating systems. And vendors should consider open-source software if they want their products to be widely used.”

_______________

5 L. Oliker, A. Canning, J. Carter, J. Shalf, and S. Ethier, “Scientific computations on modern parallel vector systems,” presented at SC2004, Pittsburgh, Pa., Nov. 6-12, 2004; http://crd.lbl.gov/~oliker/papers/SC04.pdf.

6 J. Carter, J. Borrill, and L. Oliker, “Performance characteristics of a cosmology package on leading HPC architectures,” presented at HiPC 2004, The Eleventh International Conference on HPC, Bangalore, India, December 2004; http://crd.lbl.gov/~oliker/papers/HIPC04.pdf.

7 H. D. Simon, C. W. McCurdy, W. T. C. Kramer, R. Stevens, M. McCoy, M. Seager, T. Zacharia, J. Nichols, R. Bair, S. Studham, W. Camp, R. Leland, J. Morrison, and B. Feiereisen, “Creating Science-Driven Computer Architecture: A New Path to Scientific Leadership,” May 2003; http://www.nersc.gov/news/reports/HECRTF-V4-2003.pdf.

8 Federal Plan for High-End Computing: Report of the High-End Computing Revitalization Task Force (HECRTF), (Arlington, VA: National Coordination Office for Information Technology Research and Development, May 10, 2004); http://www.house.gov/science/hearings/full04/may13/hecrtf.pdf.

Next Section: Appendices >

Top