Simulating Ultra-Relativistic Heavy-Ion Collisions for the STAR Experiment at RHIC

P. Jacobs, D. Olson, I. Sakrejda, R. Snellings, F. Wang, D. Zimmerman, M. Durst, and C. Tull, Lawrence Berkeley National Laboratory
Y. Fisyak, P. Nevski, and T. Wenaus, Brookhaven National Laboratory
D. Russ and M. Kaplan, Carnegie Mellon University
M. Madrid, M. Mathis, N. Nystrom, R. Reddy, and S. Sanielivici, Pittsburgh Supercomputer Center

Research Objective

The Relativistic Heavy Ion Collider (RHIC) at Brookhaven National Laboratory will begin operation in fall 1999, colliding beams of gold nuclei at a total center of mass energy of 39 TeV. RHIC will open a new era in the study of strongly interacting matter under extreme conditions of high temperature and pressure, hopefully leading to the discovery and study of the deconfined state of strongly interacting matter known as the quark-gluon plasma. The complexity of these heavy-ion collisions will be unprecedented in accelerator-based experimentation, with over 10,000 secondary particles generated in the most violent central interactions. The Solenoidal Tracker at RHIC (STAR) is one of two major experimental facilities being constructed to observe these collisions. STAR will generate over 300 TB of data per year.

The massive data handling needs of the STAR experiment are typical of the next generation of large high-energy and nuclear physics (HENP) experiments nearing completion in the U.S. and around the world. Numerous data intensive computing issues must be addressed for STAR and other experiments to meet their scientific goals. These include storage issues resulting from the total volume of data, efficiency issues related to moving data from tertiary storage to program memory, and access pattern issues resulting from the scientific algorithms and content of the data being analyzed. Simulating and analyzing large volumes of realistic data at NERSC is helping physicists clarify and resolve these issues.

STAR is carrying out two "Mock Data Challenges" (MDCs) prior to the start of the experiment in order to gain experience in the handling and analysis of huge datasets. The generation of simulated data on the Cray T3Es at NERSC (part of the HENP Grand Challenge on Data Access) is targeted primarily at supplying realistic input data for use in the MDCs. But more generally, the data will serve the STAR collaboration as an important resource for understanding the experimental environment and preparing for real data analysis.

Computational Approach

Generation of realistic simulated data for heavy ion collisions requires two components: (1) an event generator, which simulates the dynamics of the collision on the nuclear scale within a particular theoretical framework, and whose output consists of 4-vectors of the particles produced in the collision, and (2) a model of the response of the detector to these particles.

From the variety of event generators for heavy ion collisions, we have chosen to focus on the Hijing event generator (X. N. Wang and M. Gyulassy), which contains physics of special interest to the STAR collaboration.

Modeling the response of the detectors in HENP experiments is done with the GEANT detector description and simulation tool, which was developed principally at CERN in Geneva over the last two decades and represents the research community's accumulated knowledge of the interaction of particles with matter. Using GEANT to generate data is extremely compute-intensive. Depending on the run configuration, generation of a single event (~20 MB data volume) can consume up to 2 PE-hours on a 450 MHz T3E processor.

Subsequent stages of the STAR simulations are performed at Brookhaven's RHIC Computing Facility (RCF). The data files were transferred from the NERSC HPSS to the RCF using the DPSS tool developed at Berkeley Lab.

Accomplishments

All of the code needed for this project was developed on non-T3E architectures. Porting the GEANT code and associated software (called CERNLIB) to the T3E was a large and complex task which had never before been successfully completed, though there had been several attempts. We successfully ported GEANT (in its STAR-specific manifestation, called GSTAR) and the event generators Hijing and RQMD.

The simulated event production was managed using a set of PERL scripts based on the SimProd package for the BABAR experiment. This package set up the input and output files for the parallel jobs and maintained a database of information about the experimental conditions and physics assumptions used for each simulation run.

Following development and debugging of the code and management scripts, Hijing/GSTAR was run intensively over a period of six weeks during August and September, using about 90k PE-hours on NERSC's T3E and generating over 1 TB of simulated data (~50k events). The cross-country data transfer to the RCF at Brookhaven achieved transfer rates of 800-900 kB/sec over brief periods and an average 200 kB/sec sustained over days.

Significance

Handling and accessing huge quantities of data is the critical computing problem to be overcome in order to achieve the scientific goals of the new large-scale HENP experiments. Developing these capabilities is an essential component of the STAR collaboration's (and RHIC's) preparations for real data acquisition. The simulated data serve as the testbed for developing data analysis tools and algorithms. Use of the Cray T3Es for this purpose was essential: there is no other computing resource available to STAR that can supply the processing power needed for this large project.

Publications

http://www-rnc.lbl.gov/GC/

http://www.rhic.bnl.gov/STAR/star.html