Craig Tull

HEP Case Study Worksheet

1.1. Project Information - High Energy Physics on PDSF

2. Project Summary & Scientific Objectives for the Next 5 Years

Please give a brief description of your project - highlighting its computational aspect - and outline its scientific objectives for the next 3-5 years. Please list one or two specific goals you hope to reach in 5 years.

Presented here is a case study for Detector Simulation and Data Analysis for experimental High Energy Physics at NERSC. This covers several distinct experiments. Specifically ATLAS, Daya Bay, and CDF.

The ATLAS experiment is being constructed by 1800 collaborators in 150 institutes around the world. It will study proton-proton interactions at the Large Hadron Collider (LHC) at the European Laboratory for Particle 
Physics (CERN). The detector is due to begin operation in the November 2009. The primary purpose of the detector will be studies of the origin of mass at the electroweak scale, therefore the detector has been designed for sensitivity to the largest possible Higgs mass range. The detector will also be used for studies of top quark decays and supersymmetry searches. Currently, ATLAS uses PDSF for physics and detector simulations to understand detector behavior and to test physics hypotheses. This will continue. After turn-on, ATLAS will use PDSF for data analysis to understand detector behavior and to pursue physics topics such as the search for the Higgs Boson. ATLAS uses HPSS for backup, and online storage (eg GPFS) for active data.

The Daya Bay Neutrino Experiment is a neutrino-oscillation experiment designed to measure the mixing angle theta13 using anti-neutrinos produced by the reactors of the Daya Bay and Ling Ao nuclear power plants. The experiment is being build by blasting 3 kilometers of tunnel through the granite rock under the mountains of the power plants. Data taking is scheduled to start in Winter 2010, and reach full configuration in Winter 2011. On PDSF, Data Bay performs simulations of the detectors, reactors, and surrounding mountains to help design and anticipate detector properties and behavior. This will continue for at least 5 years. Once real data is available, Daya Bay will be using PDSF to analyze data from commissioning of, and operation of the detectors. Daya Bay uses HPSS as the central US repository for all data, information, and backups. US collaborators from 15 institutions will access data stored at NERSC. 
 
CDF is a legacy project completing its physics analysis of Tevatron data from the Collider Detector at Fermilab (CDF). The work involves both the analysis of experimental data collected at Fermilab and also the generation, simulation and analysis of Monte Carlo data samples used to calculate detector acceptance. The analysis concentrates on the search for the Standard Model Higgs boson, study of properties of the top quark (mass and production cross section), and precision measurements of B hadron decays. The analysis performed during this project should result in several publication papers in referreed journals (Phys Rev Lett and/or Phys Rev D). 
 
Other HEP projects, large and small, use the facilities at NERSC and PDSF to do eg. feasibility study simulation and/or analysis, or to explore advanced techniques associated with simulation and analysis of HEP data.  

3. Current HPC Usage and Methods

3a. Please list your current primary codes and their main mathematical methods and/or algorithms. Include quantities that characterize the size or scale of your simulations or numerical experiments; e.g., size of grid, number of particles, basis sets, etc. Also indicate how parallelism is expressed (e.g., MPI, OpenMP, MPI/OpenMP hybrid)

There are too many codes to list exhaustively, but of special note are: - GEANT4: The object-oriented simulation framework in use for almost all HEP experiments. GEANT4 models extremely complex detector geometries, material composition, and physics processes including radiation, particle capture, spallation, Cherenkov radiation, optical photons, electromagetics, etc. In essence, almost any interaction of particles and matter. 
- ROOT: ROOT is an object-oriented toolkit and framework in use by most HEP experiments. ROOT forms the basis of many analysis and visualization systems in HEP. 
- Gaudi: Gaudi is a general purpose simulation and analysis framework in use by many HEP experiments including ATLAS and Daya Bay. 
- Python and PyROOT: Python is very actively used by HEP and especially ATLAS and Daya Bay. PyROOT is an introspection-driven interface between Python and ROOT which presents a fully-featured python interface to any ROOT C++ class described in the ROOT dictionary. 
- CERNVM: CERN Virtual Machine is a virtualization project being developed at CERN and by LBNL scientists to  
- PAW and GEANT3 are legacy FORTRAN programs which predate ROOT and GEANT4. 
 
 

3b. Please list known limitations, obstacles, and/or bottlenecks that currently limit your ability to perform simulations you would like to run. Is there anything specific to NERSC?

3c. Please fill out the following table to the best of your ability. This table provides baseline data to help extrapolate to requirements for future years. If you are uncertain about any item, please use your best estimate to use as a starting point for discussions.

Please list any required or important software, services, or infrastructure (beyond supercomputing and standard storage infrastructure) provided by HPC centers or system vendors.

4. HPC Requirements in 5 Years

4a. We are formulating the requirements for NERSC that will enable you to meet the goals you outlined in Section 2 above. Please fill out the following table to the best of your ability. If you are uncertain about any item, please use your best estimate to use as a starting point for discussions at the workshop.

4b. What changes to codes, mathematical methods and/or algorithms do you anticipate will be needed to achieve this project's scientific objectives over the next 5 years.

4c. Please list any known or anticipated architectural requirements (e.g., 2 GB memory/core, interconnect latency < 3 #s).

4d. Please list any new software, services, or infrastructure support you will need over the next 5 years.

4e. It is believed that the dominant HPC architecture in the next 3-5 years will incorporate processing elements composed of 10s-1,000s of individual cores, perhaps GPUs or other accelerators. It is unlikely that a programming model based solely on MPI will be effective, or even supported, on these machines. Do you have a strategy for computing in such an environment? If so, please briefly describe it.

New Science With New Resources

To help us get a better understanding of the quantitative requirements we've asked for above, please tell us: What significant scientific progress could you achieve over the next 5 years with access to 50X the HPC resources you currently have access to at NERSC? What would be the benefits to your research field if you were given access to these kinds of resources?

Please explain what aspects of "expanded HPC resources" are important for your project (e.g., more CPU hours, more memory, more storage, more throughput for small jobs, ability to handle very large jobs).