NERSCPowering Scientific Discovery Since 1974

Martin Greenwald

FES Requirements Worksheet

1.1. Project Information - Fusion Simulation Program

Document Prepared By

Martin Greenwald

Project Title

Fusion Simulation Program

Principal Investigator

Bill Tang

Participating Organizations

PPPL, MIT, ORNL, TechX, GA, UCSD, LBNL, LLNL, ANL, U. Texas, Lehigh, NYU, LANL (currently, and likely to broaden in the future)

Funding Agencies


2. Project Summary & Scientific Objectives for the Next 5 Years

Please give a brief description of your project - highlighting its computational aspect - and outline its scientific objectives for the next 3-5 years. Please list one or two specific goals you hope to reach in 5 years.

The Fusion Simulation Program (FSP) mission is to provide predictive capability for the behavior of magnetic confinement devices via science-based simulations of nonlinear, coupled phenomena on time and space scales required for fusion energy production. This will require multi-scale, multi-physics integration well beyond current capabilities. The mission will be accomplished through improvements and innovation in physics formulation, numerics and algorithms along with the use of increasingly powerful computer architectures. A rigorous verification and validation program will be an integral part of the FSP, requiring significant computational resources on its own. Productions services, with a large user base are also planned. The FSP is currently in the middle of a 2 year planning exercise whose main goal is to define a compelling and detailed program plan. Significant funding for the program is included in the FY11 Presidential budget request. Project execution is expected to begin in FY12 (October, 2011). If fully funded, the FSP will roughly double the scope of computing in the MFE program.

3. Current HPC Usage and Methods

3a. Please list your current primary codes and their main mathematical methods and/or algorithms. Include quantities that characterize the size or scale of your simulations or numerical experiments; e.g., size of grid, number of particles, basis sets, etc. Also indicate how parallelism is expressed (e.g., MPI, OpenMP, MPI/OpenMP hybrid)

The FSP is currently in a planning stage. 

3b. Please list known limitations, obstacles, and/or bottlenecks that currently limit your ability to perform simulations you would like to run. Is there anything specific to NERSC?


3c. Please fill out the following table to the best of your ability. This table provides baseline data to help extrapolate to requirements for future years. If you are uncertain about any item, please use your best estimate to use as a starting point for discussions.

Facilities Used or Using

 NERSC  OLCF  ACLF  NSF Centers  Other:  

Architectures Used

 Cray XT  IBM Power  BlueGene  Linux Cluster  Other:  

Total Computational Hours Used per Year


NERSC Hours Used in 2009


Number of Cores Used in Typical Production Run


Wallclock Hours of Single Typical Production Run


Total Memory Used per Run


Minimum Memory Required per Core


Total Data Read & Written per Run


Size of Checkpoint File(s)


Amount of Data Moved In/Out of NERSC

 GB per  

On-Line File Storage Required (For I/O from a Running Job)

 TB and  Files

Off-Line Archival Storage Required

 TB and  Files

Please list any required or important software, services, or infrastructure (beyond supercomputing and standard storage infrastructure) provided by HPC centers or system vendors.


4. HPC Requirements in 5 Years

4a. We are formulating the requirements for NERSC that will enable you to meet the goals you outlined in Section 2 above. Please fill out the following table to the best of your ability. If you are uncertain about any item, please use your best estimate to use as a starting point for discussions at the workshop.

Computational Hours Required per Year

 100 - 1500

Anticipated Number of Cores to be Used in a Typical Production Run

40k - 1M

Anticipated Wallclock to be Used in a Typical Production Run Using the Number of Cores Given Above


Anticipated Total Memory Used per Run

 40 - 100,000 GB

Anticipated Minimum Memory Required per Core

 0.1 - 2 GB

Anticipated total data read & written per run

 1 - 25,000 GB

Anticipated size of checkpoint file(s)

 1 - 5,000 GB

Anticipated Amount of Data Moved In/Out of NERSC

 100 - 200 GB per  day

Anticipated On-Line File Storage Required (For I/O from a Running Job)

 5 - 1,000 TB and 100 - 3,000 Files

Anticipated Off-Line Archival Storage Required

 1,000 - 3,000 TB and 3,000 Files

4b. What changes to codes, mathematical methods and/or algorithms do you anticipate will be needed to achieve this project's scientific objectives over the next 5 years.

4c. Please list any known or anticipated architectural requirements (e.g., 2 GB memory/core, interconnect latency < 1 μs).

4d. Please list any new software, services, or infrastructure support you will need over the next 5 years.

4e. It is believed that the dominant HPC architecture in the next 3-5 years will incorporate processing elements composed of 10s-1,000s of individual cores, perhaps GPUs or other accelerators. It is unlikely that a programming model based solely on MPI will be effective, or even supported, on these machines. Do you have a strategy for computing in such an environment? If so, please briefly describe it.


New Science With New Resources

To help us get a better understanding of the quantitative requirements we've asked for above, please tell us: What significant scientific progress could you achieve over the next 5 years with access to 50X the HPC resources you currently have access to at NERSC? What would be the benefits to your research field if you were given access to these kinds of resources?

Please explain what aspects of "expanded HPC resources" are important for your project (e.g., more CPU hours, more memory, more storage, more throughput for small jobs, ability to handle very large jobs).

The size and scope of the FSP will drive significant computational and storage requirements. 
1. More CPU hours (Roughly doubling size of MFE computing with a focus on some of the largest, most demanding computational problems.)  
2. Fast turn-around for smaller jobs, especially in support of code development, verification and validation. 
3. Support for production computing including a "Simulation as a Service" model. Requires some level of federation for authentication and authorization. 
4. Integrated data management, long term storage and advanced cataloging of modeling and experimental data.