National Energy Research Scientific Computing Center 2004 Annual Report
Navigation
the nersc center
secure connectivity
All NERSC Production Systems Now Grid-Accessible
Early in 2004 came good news for NERSC users — all of NERSC’s production computing and storage resources are now Grid-enabled and can be accessed by users who have Grid applications using the Globus Toolkit. Grid users now have access to Seaborg, HPSS storage, the PDSF cluster, and the visualization and math servers. Users can also get support for Grid-related issues from NERSC’s User Services Group.
“Now that the core functionalities have been addressed, our next push is to make the Grid easier to use and manage, for both end users and the administrators who handle applications that span multiple sites,” said Steve Chan, who coordinated the Grid project at NERSC.
One of the major challenges faced by NERSC’s Grid team was installing the necessary software on Seaborg, which operates using a complicated software stack. Additionally, the system needed to be configured and tested without interfering with its heavy scientific computing workload.
“With Steve Chan’s leadership and the technical expertise of the NERSC staff, we crafted a Grid infrastructure that scaled to all of our systems,” said Bill Kramer, general manager of the NERSC Center. “As the driving force, he figured out what needed to be done, then pulled together the resources to get it done.”
To help prepare users, Center staff presented tutorials at both Globus World and the Global Grid Forum, with training specifically designed for NERSC users.
When authenticated users log into NIM (NERSC Information Management), they are now able to enter their certificate information into the account database and have this information propagated out to all the Grid-enabled systems they normally access, superseding the need to have separate passwords for each system.
Because the Grid opens up a wider range of access paths to NERSC systems, the security system was enhanced. Bro,4 the LBNL-developed intrusion detection system which provides one level of security for NERSC, was adapted to monitor Grid traffic at NERSC for an additional layer of security.
Full Grid access is the culmination of a two-year effort by the staff members pictured in Figure 8.
| Mikhail Avrekh | Clayton Bagwell | Mark Heer | Nick Balthaser | Scott Campbell |
 |
 |
 |
 |
 |
| Shane Canon | Steve chan | Steve Lau | Ken Okikawa | |
 |
 |
 |
 |
|
| R. K. Owen | David Paul | Iwona Sakrejda | ||
 |
 |
 |
Figure 8. The NERSC Grid Team configured and tested Grid access without interfering with NERSC’s scientific computing workload. | |
| Jay Srinivasan | David Turner | |||
 |
 |
Keeping Out Cyber Intruders
NERSC, which has shared its cyber security expertise with the HPC community for the past three years, unintentionally emerged as a true leader in the field in March and April 2004, when a notorious hacker broke into many of the nation’s leading supercomputing centers and forced them to take their systems offline. Thanks to its Bro intrusion detection system, NERSC withstood the hacker’s attacks and remained in service to users.
The attacks prompted the organization of “Cybersecurity Summit 2004,” an invitation-only workshop held in Arlington, Virginia on September 27–28. This meeting brought together cyber security experts from some of the nation’s top research institutions to better prepare for future cyber attacks.
Sharing their experience and expertise at the workshop were Howard Walter and Scott Campbell of NERSC’s Networking, Security, Servers and Workstations Group, and Berkeley Lab’s Computer Protection Program Manager Dwayne Ramsey, who was an invited speaker.
“The biggest benefit to holding a meeting like this is building the trust relationships that are critical when it comes to developing a coordinated strategy against cyber attacks,” said Walter, a member of the summit program committee. “When an attack is detected by multiples sites at the same time, knowing who to talk to — and who to trust — can make all the difference.”
While the attacks in spring raised the visibility of cyber security, the issue is a constant threat. The NERSC Center is scanned for vulnerabilities about 5,000 times a month, while another 70 worms and viruses attempt to penetrate the systems each month.
Providing an invisible shield around NERSC is the Bro intrusion detection system developed at Berkeley Lab. Bro constantly monitors network links, searching for traffic that potentially violates a site’s access and usage policies. Bro automatically blocks incoming malicious traffic and notifies cyber security staff.
Steve Lau of NERSC’s Networking, Security, Servers and Workstations Group has also developed a visualization tool for Bro to graphically show how dangerous the Internet has become. First deployed at the SC2003 conference and then again at SC2004, the “Spinning Cube of Potential Doom” (Figure 9) attempts to display the overall level of malicious traffic in a fashion that can be easily understood by those without a computer security or networking background.
The Cube leverages Bro’s capability to log all instances of completed
or attempted TCP connections, displaying this information within a three-dimensional
cube. Each axis represents a different component of a TCP connection: the
X axis represents the local IP address space, the Z axis represents the
global IP address space, and the Y axis represents the port number. Port
numbers are used in connections to locate services and coordinate communication
(e.g., 22 for ssh and 80 for http).
Figure 9. The Spinning Cube of Potential Doom reveals surprising patterns
in malicious Internet traffic.
Each TCP connection, whether attempted or successful, is displayed as a single point in the Cube. Successful connections are shown as white dots, while incomplete connections are colored using a rainbow color map, with the color varying by port number. This color mapping was used to assist the viewer in locating the point in 3D space. The vast majority of colored dots can be considered malicious traffic searching for potentially vulnerable systems.
One of the more interesting findings, even surprising to those with backgrounds in computer security, is that visual patterns emerge from the data. Various distinct patterns are easily discernable. Port scans appear as lines, with vertical lines representing scans directed at a particular host searching for any listening port, and horizontal lines being scans directed at the entire local address space on a particular port.
Another unexpected pattern emerging from the data was dubbed “barber pole” because it looks like the striping on barber poles. These scans vary their port numbers and IP addresses in an attempt to elude detectors. Although they may be capable of evading detectors, they stand out when visualized by the Cube of Doom.
Another type of scan detected was dubbed a “lawnmower” scan. These scans are quite noisy, in that they scan a wide range of contiguous ports while simultaneously marching across the entire local address space. Some of these scans are quite rapid, occurring within a few seconds, while others are more leisurely, ranging on the order of minutes.
The Cube’s visual display received much interest from SC conference attendees, many of whom were surprised at the amount of potentially malicious traffic on the Internet. Many attendees were curious as to what portion of the data represented attempts against their systems and expressed surprise that they had not noticed it themselves. Many of them said that the data had inspired them to make sure their systems are up to date.
Making Access More Secure and Efficient
One of the most popular displays at the past few SC conferences has been a monitor displaying unprotected passwords as they were sent over the network. The data, gathered by Berkeley Lab’s Bro intrusion detection system, was of great interest because is showed both how little effort some people put into choosing their passwords, and how easily those passwords could be intercepted.
To minimize the risk of this happening to both NERSC users and staff, members of the Networking, Security, Servers and Workstations Group have been assessing the use of one-time passwords. Not only was the group interested in the effectiveness of the offerings from the three leading vendors in the field, but staff also wanted to develop an approach that was relatively easy to use.
A major obstacle with implementing one-time passwords is that they are literally just that, said Steve Chan. “The difficulty is that every time you log into a system, you need to use a new password. In the course of a day, you could find yourself having to generate and use one password for your desktop machine, another one to log into Seaborg, yet another one if you use PDSF, then another one if you want to open a new window on Seaborg. And if you were running batch jobs and wanted to copy files from HPSS or using Grid FTP, you’d need even more passwords,” Chan explained.
Such a system would make it difficult to carry out automated processes, such as backups, which are often done late at night.
So, the group looked into how one-time passwords could be integrated with a single sign-on, allowing the user to use one secure password across all NERSC systems. After investigating numerous options, the group developed a plan to integrate the password system with Kerberos, which would generate a single “ticket” and be recognized across most platforms. For password generation, the group is planning to build an initial system using CryptoCard, a one-time password vendor that provides and an open-solution toolkit.
The group, working with ESnet and other labs, also tested a federated system to allow single sign-on across other DOE laboratories. In the spring and summer of 2004, a test program was conducted across ESnet to evaluate the technical feasibility of using a remote laboratory’s one-time password system for accessing local systems at Lawrence Berkeley, Argonne, Oak Ridge, and Pacific Northwest national laboratories. The successful tests led ESnet to propose building out an “authentication fabric” linking the individual authentication systems at each lab together, while NERSC sought funding for a local one-time password system that ties the one-time passwords into Kerberos. While still awaiting final word on the funding proposal, Chan said NERSC will begin testing a system in 2005.
“In addition to testing how the system will be used, we also want to make sure that communication across the channels is secure and that nothing sensitive goes across the network without being protected,” Chan said.
Ultimately, just as Berkeley Lab and NERSC have done with Bro, the goal is to take a lead role in developing and deploying an effective and affordable system that can be used, added to and extended by other DOE sites — and that does not hinder scientific productivity.
_______________