NERSC marks fourth year of INCITE applications

When DOE Under Secretary of Science Dr. Raymond Orbach launched the Innovative and Novel Impact on Theory and Experiment (INCITE) program in 2003, NERSC was the only DOE HPC center able to support the high-impact computational science projects selected to participate.

The INCITE program seeks computationally intensive, large-scale research projects, with no requirement of current DOE sponsorship, that can make high-impact scientific advances through the use of a substantial allocation of computer time and data storage resources. From the three projects selected for NERSC allocations in each of the years 2004 and 2005, INCITE in 2007 comprises 45 projects with a total allocation of 95 million processor-hours at four Office of Science computing centers, including NERSC.

Under the 2007 INCITE awards, seven research projects will receive nearly 9 million processor hours at NERSC. The projects range from studying the behavior of a supernova to designing more energy-efficient cars. Application areas include turbulence, combustion, fusion energy, accelerator design, climate change, chemistry, and physics.

One of the projects receiving a 2007 NERSC allocation was a renewal of the 2006 INCITE-supported investigation into the chemistry of organolithium compounds led by Professor Lawrence Pratt of Fisk University in Tennessee (see this page). According to Pratt, before receiving his INCITE award, most of his research was carried out using PCs.

“This severely limited the job throughput and in some cases even limited the kinds of calculations that could be performed,” Pratt said. “The award has more than doubled my research productivity, resulting in two publications for 2006 and three papers accepted for publication in 2007 to date. Several more projects are nearing completion in 2007.”

Indeed, INCITE recipients say the allocations, coupled with effective services from NERSC staff, have enabled them to accelerate their work. Donald Lamb, from the University of Chicago, for example, was able to announce a breakthrough in March 2007 thanks partly to the NERSC staffers’ effort to make it possible to run large computing jobs under a tight deadline in January. Lamb’s research demonstrated for the first time the ability to naturally detonate the explosion of a white dwarf star in a three-dimensional simulation. The modeling of this Type Ia supernova also confirmed previous beliefs that the white dwarf star detonates in a supersonic process resembling diesel-engine combustion.

Other INCITE researchers also expressed their gratitude.

“NERSC staff worked with us extensively to get our runs done,” said Berkeley Lab fusion researcher Cameron Geddes, whose 2006 INCITE project created detailed three-dimensional models of laser-driven wakefield particle accelerators (see this page). “Consultants and managers met with us to provide needed disk space and queue organization and allocations to get large runs done, and this was critical to the results.”

Robert Harkness, a scientist at the San Diego Supercomputer Center and a co-investigator for the 2006 INCITE project on characterizing the shape, matter-energy contents, and expansion history of the universe, said he was impressed with Seaborg and HPSS. “Seaborg and HPSS have been quite simply outstanding in the high degree of reliability,” Harkness said. “I am particularly impressed with the HPSS—I’ve written 80 to 100 terabytes without any trouble.”

Making shared resources easy to use

Staff in the Open Software and Programming Group are bringing new services to NERSC users through grid technologies provided by the Open Science Grid.

The Open Science Grid (OSG), funded by the DOE Office of Science and the National Science Foundation, is a distributed computing infrastructure for scientific research built by a nationwide community of universities and national laboratories. In January 2006, NERSC General Manager Bill Kramer was elected the first chair of the OSG Council, made up of representatives from each member institution or partner organization.

The OSG software stack provides distributed mechanisms for moving data and submitting jobs to computing resources on the OSG grid. Several of the frameworks have long been in use in the high energy physics community and are seeing increased use in new science areas.

David Skinner

“The standard model for computing at most parallel computing centers is to provide batch queues where users can log in and submit specific jobs,” said David Skinner, who became leader of the Open Software and Programming Group in May 2006. “More and more science teams expect to be able to streamline their computing needs into a workflow that can be accessed and operated in a distributed way.”

The OSG received an allocation of time at NERSC for the purpose of extending its user base and the range of research areas and applications using grid technologies on parallel systems. Discussions with the DOE Joint Genome Institute and with nanoHUB, a web-based resource for nanoscience and technology, have extended the OSG’s application base to bioinformatics and materials science.

Providing excellent services to NERSC’s research customers depends on both having useful capabilities and also configuring and organizing them in ways that make them simple for researchers to access. Grid technologies can be daunting in their complexity, and easing the adoption of grid services into new science areas is an active area of work in the Open Software and Programming Group.

So far, OSG grid services have been deployed on PDSF, DaVinci, Jacquard, and HPSS. These NERSC resources provide OSG with new capabilities in parallelism and filesystem performance. “Not all data endpoints are equal in terms of performance,” Skinner said. “We stood up these services first on DaVinci because it provides a much faster connection to the NERSC Global Filesystem. DaVinci is also a natural location for data-intensive analytics activities. By providing the full suite of Open Science Grid software and services, we hope to make DaVinci much more useful to our users.”  The OSG software stack will be deployed on other NERSC systems in 2007.

Depositing a slice of the Universe

Data from the world’s largest radio telescope, which scans the sky for signs of pulsars, black holes, and extraterrestrial civilizations, has found a new home in NERSC’s vast storage system.

Led by Dan Werthimer, a scientist at the University of California-Berkeley’s Space Sciences Laboratory, the “Berkeley High Resolution Neutral Hydrogen Sky Survey” recently began feeding hundreds of gigabytes of data daily to NERSC’s HPSS.

Figure 2. Serving up the cosmos, the Arecibo dish in Puerto Rico detects radio signals emitted from space and helps researchers understand stars, dark energy, and everything in between. (Image courtesy of the NAIC Arecibo Observatory, a facility of the NSF.)

Collected by the Arecibo Observatory in Puerto Rico (Figure 2), the radio signal data will provide valuable information to astrophysicists, cosmologists, and other researchers who aim to solve some of the most intriguing mysteries about the universe. The Arecibo Observatory, a 305-meter dish, is operated by Cornell University under an agreement with the National Science Foundation.

“Before we received assistance from NERSC, our sky survey data was stored on over 1,000 tapes. So it was very difficult to access,” Werthimer said. “NERSC will allow us to make our data accessible to the scientific community for a variety of research projects.”

For Werthimer and his research team, the data will help advance three projects: SETI@home, AstroPulse, and hydrogen mapping. SETI@home, launched in 1995, seeks evidence of extraterrestrial civilizations by searching for certain types of radio signals that, because of their narrow bandwidths, do not occur naturally.

The SETI (Search for Extraterrestrial Intelligence) project has created a global distributed computing network spanning 226 countries that invites anyone with an Internet-connected computer to join. So far, it has attracted more than 5.2 million participants, whose personal computers help to crunch the data from Arecibo.

AstroPulse, on the other hand, will look for dispersed radio pulses that are shorter than the ones sought by SETI@home. The microsecond pulses could come from rapidly spinning pulsars, black holes, or extraterrestrial intelligence.

Both SETI@home and AstroPulse run on BOINC, an open-source software. BOINC, developed by David Anderson and his team at the Space Sciences Lab, makes it easy to develop and manage distributed computing projects. Projects that take advantage of BOINC include Einstein@home at the University of Wisconsin and LHC@home at CERN.

The third project maps the galactic distribution of hydrogen. The high-resolution map will provide a wealth of three-dimensional information such as the density and temperature of interstellar objects, enabling researchers to better understand the structure and dynamics of our galaxy.

“We recognize the importance of Dan’s projects to advance a wide-range of space research,” said Francesca Verdier, Associate General Manager for Science Driven Services at NERSC. “The data from Arecibo will be put to good use by the scientific community worldwide.”

Werthimer plans to store 200 terabytes of data over two years at HPSS. HPSS, which was put in place in 1998, currently has a theoretical capacity of 30 petabytes. It is capable of accepting data at 450 megabytes per second.

Survey gauges user satisfaction

NERSC users gave kudos to HPSS, Jacquard uptime, and Bassi Fortran compilers in the results of a survey to solicit feedback about their experiences using NERSC resources in 2006.

The annual survey enables NERSC staff to gauge their successes and make improvements. The survey asked researchers to rate their overall satisfaction with the hardware, software, and services. It also posed more specific questions about these tools and services, and invited users to share their thoughts. On a scale of 1 to 7, with 7 being “very satisfied,” survey respondents gave an average of 6.3 for the question about their overall satisfaction with NERSC, an improvement from 2005 and 2004.

“We take feedback from our users very seriously, whether it is in the form of day-to-day interactions, through the NERSC User Group meetings, or via our annual survey,” said Bill Kramer, General Manager of NERSC. “We work very hard to bring any low scores up and to keep the overall satisfaction high in all areas. I want to thank all the users who took the time to provide this valuable feedback.”

About 13 percent of the active NERSC users, or 256, took the online survey. The respondents represented all six programs within the DOE Office of Science.

Areas with the highest user satisfaction in 2006 included account and consulting services, DaVinci C/C++ compilers, and network performance within the NERSC center. The largest increases in satisfaction over the 2005 survey were for the Jacquard Linux cluster, Seaborg batch wait times and queue structure, NERSC’s available computing hardware, and the NERSC Information Management (NIM) system.

Despite the improvements, however, Seaborg batch wait times received a low average score compared with assessments about other systems and services. Other areas that received lower scores included PDSF disk storage, interactive services and performance tools, analytics facilities, and Bassi and Seaborg visualization software.

The survey gave users an opportunity to pen their own thoughts about using NERSC. The question “What does NERSC do well?” drew answers from 113 users. Among them, 87 stated that NERSC gave them access to powerful computing resources without which they could not do their science; 47 mentioned excellent support services and NERSC’s responsive staff; 27 highlighted good software support or an easy-to-use user environment; and 24 pointed to hardware stability and reliability.

“NERSC runs a reliable computing service with good documentation of resources,” wrote one user. “I especially like the way they have been able to strike a good balance between the sometimes conflicting goals of being at the ‘cutting edge’ while maintaining a high degree of uptime and reliable access to their computers.”

In previous years, queue turnaround and job scheduling issues were areas of the greatest concerns. NERSC has made many efforts to acquire new hardware, to implement equitable queuing policies across the NERSC machines, and to address queue turnaround times by adjusting the duty cycle of the systems.

These efforts have clearly paid off. In 2004, 45 users reported dissatisfaction with queue turnaround times. In 2005 this number dropped to 24. In 2006 only five users made such comments.

Improvements to Jacquard’s computing infrastructure and the deployment of the NERSC Global File System also addressed shortcomings voiced by users in previous years. Although job scheduling remained a concern in 2006, NERSC users pointed to new challenges for NERSC staff. Researchers said they would like more compute cycles and software fixes.

The complete survey results can be found at http://www.nersc.gov/news/survey/2006/.