Julian Borrill

HEP Case Study Worksheet

1.1. Project Information - Cosmic Microwave Background Data Analysis For The Planck Satellite Mission

2. Project Summary & Scientific Objectives for the Next 5 Years

Please give a brief description of your project - highlighting its computational aspect - and outline its scientific objectives for the next 3-5 years. Please list one or two specific goals you hope to reach in 5 years.

The Planck satellite, launched in May 2009, will gather the most detailed Cosmic Microwave Background (CMB) data set to date. With 74 detectors at 9 frequencies it will measure the temperature and polarization of the entire microwave sky for at least one - and likely two or more - years at up to 5 arcminute resolution. This will constitute the definitive CMB data set for at least the next decade. 
 
The next 5 years covers the proprietary period of Planck data analysis; by the end of this time, we will have to have completed the analysis and released the final data products to the community. 
 
The dominant computational challenge during this period will be the need to generate tens of thousands of Monte Carlo realizations of the entire Planck mission - including all know systematic effects - to quantify the uncertainties on the analysis of the real data. 
 
We will also use the Planck data analysis as a starting point for the next generation of suborbital and ultimately satellite experiments which will gather 10's to 1000's of times the data gathered by Planck.

3. Current HPC Usage and Methods

3a. Please list your current primary codes and their main mathematical methods and/or algorithms. Include quantities that characterize the size or scale of your simulations or numerical experiments; e.g., size of grid, number of particles, basis sets, etc. Also indicate how parallelism is expressed (e.g., MPI, OpenMP, MPI/OpenMP hybrid)

The dominant CMB simulation and analysis algorithms loop (by iteration and/or realization) over FFT/SHT kernels that scale linearly or log-linearly with the number of samples in the data set - of the order of 8x10^11 for Planck, 3x10^13 for PolarBear, 10^14 for QUIET-II, and 10^15 for CMBpol. 
 
The core codes for generating and analyzing Monte Carlo realizations are: 
 
The Planck LevelS simulation software - serial tools. 
M3 data abstraction layer, including on-the-fly simulation (OTFS) capability - MPI parallel. 
Springtide destriping map-maker - MPI-parallel. 
MADAM destriping map-maker - MPI parallel. 
MADmap maximum likelihood map-maker - MPI parallel. 
 
Each of these is also often used with several instances running simultaneously in a single job submission script. 

3b. Please list known limitations, obstacles, and/or bottlenecks that currently limit your ability to perform simulations you would like to run. Is there anything specific to NERSC?

IO has been the limiting factor in many of our largest runs. Much of the problem has been solved by changing our approach (OTFS) but this is not always applicable. Franklin NGF performance is now a limiting factor. 

3c. Please fill out the following table to the best of your ability. This table provides baseline data to help extrapolate to requirements for future years. If you are uncertain about any item, please use your best estimate to use as a starting point for discussions.

Please list any required or important software, services, or infrastructure (beyond supercomputing and standard storage infrastructure) provided by HPC centers or system vendors.

4. HPC Requirements in 5 Years

4a. We are formulating the requirements for NERSC that will enable you to meet the goals you outlined in Section 2 above. Please fill out the following table to the best of your ability. If you are uncertain about any item, please use your best estimate to use as a starting point for discussions at the workshop.

4b. What changes to codes, mathematical methods and/or algorithms do you anticipate will be needed to achieve this project's scientific objectives over the next 5 years.

Scaling to next-generation systems (especially IO & communication) 
Taking advantage of next-generation architectures (scaling to very high concurrency, tuning for heterogeneous systems, GPUs etc)

4c. Please list any known or anticipated architectural requirements (e.g., 2 GB memory/core, interconnect latency < 3 #s).

More more more! Our driving data volume is growing with Moore's Law.

4d. Please list any new software, services, or infrastructure support you will need over the next 5 years.

4e. It is believed that the dominant HPC architecture in the next 3-5 years will incorporate processing elements composed of 10s-1,000s of individual cores, perhaps GPUs or other accelerators. It is unlikely that a programming model based solely on MPI will be effective, or even supported, on these machines. Do you have a strategy for computing in such an environment? If so, please briefly describe it.

We are just beginning to work on this with an NSF PetaApps grant. 

New Science With New Resources

To help us get a better understanding of the quantitative requirements we've asked for above, please tell us: What significant scientific progress could you achieve over the next 5 years with access to 50X the HPC resources you currently have access to at NERSC? What would be the benefits to your research field if you were given access to these kinds of resources?

Please explain what aspects of "expanded HPC resources" are important for your project (e.g., more CPU hours, more memory, more storage, more throughput for small jobs, ability to handle very large jobs).

Next-generation suborbital CMB polarization experiments will gather orders of magnitude more data than Planck (to achieve the required increased sensitivity) while experiencing significantly worse systematic effects (absent the advantages of a satellite mission). Simply due to data volumes we will need to follow Moore's Law, but once they greater challenge of these analyses is taken into account we will likely need an *additional* 10x increase in computational capability to realize the scientific protential of these data sets.