For more than 45 years the National Energy Research Scientific Computing Center (NERSC) has been providing leading edge computational and data systems to support scientific discovery for the DOE Office of Science. Launched in 1974 as a computing resource for Fusion Energy Research at Lawrence Livermore National Laboratory, NERSC quickly expanded its role to include users from all SC program offices, and in 1996 moved to Lawrence Berkeley National Laboratory in 1996. Today its 8,000 users make NERSC perhaps the most scientifically productive high performance computing (HPC) center in the world, with NERSC users producing more than 2,500 referred publications annually. In addition, over the years the center has been associated with six Nobel Prize winning scientists or their teams.
NERSC places a premium on delivering cutting-edge technology at scale and making it highly usable and productive in its role as the mission HPC center for the DOE Office of Science. As a unique resource, the center has been a leader in fielding next-generation supercomputing systems like the Cray T3E-900, the IBM SP POWER 3, IBM POWER 5+, and the Cray XT4. NERSC’s recent supercomputers were the first of their kind: Edison, a Cray XC30, included the first two cabinets of its line developed by Cray and DARPA; and Cori, a Cray XC40, is the first and largest system with Intel Xeon Phi “Knights Landing” processors, debuting as the 5th most powerful supercomputer in the world in 2016. In addition to its computing capabilities, Cori was the first large system to deploy an all-FLASH “burst buffer” that provided a world-best 1.7 TB/sec file system bandwidth when it was introduced. The first Cray/HPE Shasta system will arrive in late 2020 as the NERSC Perlmutter supercomputer.
NERSC is also a large-scale data analysis and storage center: there is a net flow of data into the center. Its tape archive, currently based on the High Performance Storage System (HPSS) technology, holds 200 PBs of data, some of which dates back to the first days of the center in the 1980s. NERSC is also unique among DOE National Laboratory computing centers in that it has never deleted scientific data from its archive. The center currently hosts datasets of more than 5PB each in climate science, genomics, nuclear physics and from light source facilities, cosmic microwave background studies, and a number of neutrino and high-energy physics experiments. To help scientists address their growing data needs, NERSC created a Data Department in 2015 and has developed a vigorous program supporting deep learning for science. The center maintains state-of-the art external networking capabilities – provided by ESnet – to help scientists move data to the center for analysis and archiving. NERSC has a long history of supporting data-driven science; many high-energy and nuclear physics teams have used NERSC data systems and the PDSF cluster and for their analyses. NERSC also maintains a close collaboration with DOE’s Joint Genome Institute, which conducts high-throughput DNA sequencing, synthesis and analysis in support of BER’s bioenergy and environmental missions.
To ensure that scientists, whose expertise in HPC range from world-class to first-year graduate students, are as productive as possible NERSC puts a lot of effort into creating systems that are highly available while offering expert consulting and performance optimization support from a team largely drawn from the scientific community itself. Among its innovations were using web technologies to expose job-level detail to users in the early 2000s, pioneering science gateways, developing the “Shifter” container technology for HPC, recently enabling machine learning, data analytics software, and workflow frameworks at scale. In 1978 NERSC developed CTSS, the Cray Time Sharing System, to allow a remote user interface to its Cray 1 supercomputer, the center was the first to checkpoint a full distributed-memory supercomputer (its Cray T3E) in 1997, and launched the DOE INCITE program in 2003.