John Drake
Case Study Worksheet
Project Information - A Scalable and Extensible Earth System Model for Climate Change Science
| Document Prepared By | John Drake |
|---|---|
| Project Title | A Scalable and Extensible Earth System Model for Climate Change Science |
| Principal Investigator | John Drake |
| Participating Organizations | Argonne National Laboratory - Robert Jacob, Ray Loy Brookhaven National Laboratory - Robert McGraw Lawrence Berkeley National Laboratory - Michael Wehner Lawrence Livermore National Laboratory - Phillip Cameron-Smith, Arthur Mirin, Cathy Chuang, Cyndi Atherton, and Peter Connell Los Alamos National Laboratory - Philip Jones (Co-PI), Scott Elliot, William Lipscomb, and Matt Maltrud National Center for Atmospheric Research - William Collins, Tony Craig, Peter Gent, Jean Francois Lamarque, Mariana Vertenstein, and Warren Washington Oak Ridge National Laboratory - John B. Drake (PI), David Erickson, Forest Hoffman, and Patrick Worley Pacific Northwest National Laboratory - Steven Ghan Sandia National Laboratories - Mark Taylor State University of New York at Stony Brook - Wei Zhu |
| Science Category | Climate Environmental Science Biological Sciences |
| Funding Agencies | DOE SC DOE NSA NSF NOAA NIH Other: |
Project Summary (Scientific Objectives)
Please give a brief description of your project and its scientific objectives for the next 3-5 years.
This project is also called the "CCSM Consortium project" and represents DOE's partnership with NSF and NCAR to simulate the earth's climate and climate change based on first principles, physical modeling techniques using parallel, scalable computers. Based on this physical climate system model we also are involved in extending the process models to a comprehensive earth system model that balances the global carbon and sulfur cycles. The immediate goals of the project involve development and release of the CCSM4 for use in upcoming IPCC Assessement and CMIP 5 studies. These will involve a coarse resolution 2 degree model for earth system modeling and mitigation studies and a high resolution 0.5 degree model for regional projection of climate change.
The scientific objectives of the work are to assemble a first-generation Earth system model that allows us to understand the coupling between the physical, chemical, and biogeochemical processes in the climate system. We hope to predict future climates based on the specification of greenhouse gas emissions rather than specification of atmospheric concentrations, as is done in present models, which make assumptions about the carbon cycle that are likely not valid. From a more comprehensive treatments of the processes governing well-mixed greenhouse gases, natural and anthropogenic aerosols, the aerosol indirect effect and tropospheric ozone we seek to reduce key uncertainties in climate change studies.
Current HPC Usage and Methods
| Facilities Used |
|
|
ACLF |
|
|
|---|---|---|---|---|---|
| Architectures Used |
|
|
|
|
Other: |
| Total Computational Hours Used per Year | Core-Hours | NERSC Hours Used per Year | 0 Core-Hours | ||
| Number of Cores Used in Typical Production Run | Wallclock Hours of Single Typical Production Run | ||||
| Total Memory Used per Run | GB | Minimum Memory Required per Core | GB | ||
| Total Data Read & Written per Run | GB | Size of Checkpoint File(s) | GB | ||
| Amount of Data Moved In/Out of NERSC | GB | How Often | |||
| On-Line File Storage Required (Directly Accesible from a Running Job) | GB | Files | |||
| Off-Line Archival Storage Required | GB | Files | |||
Please list any required or important software, services, or infrastructure (beyond supercomputing and standard storage infrastructure) provided by HPC centers or system vendors.
The usage stats are currently weighted by the development stage of the project. The INCITE Climate End Station is the umbrella project that provides the project researchers with access and time on DOE computers.
In the production phase with assessment runs, also part of the End Station allocation proposal, the usage stats will change.
Please list your current primary codes and their main mathematical methods and/or algorithms. Include quantities that characterize the size or scale of your simulations or numerical experiments; e.g., size of grid, number of particles, basis sets, etc. Also indicate how parallelism is expressed (e.g., MPI, OpenMP, MPI/OpenMP hybrid)
Community Atmospheric Model (CAM3.5) - finite volume, lagrangian vertical with explicit sub-cycling and polar FFT filters. 180x90x26 x 3.5million time steps.
Parallel Ocean Program (POP) - finite volume with semi-implicit elliptic solve for barotropic modes. 360x180 x 300,000 time steps.
Community Land Model (CLM 4) - land surface, hydrology and terrestrial ecology.
Community Ice Model (CICE) - sea ice melting and dynamics with finite volume, incremental remapping advection
Please list the known limitations/obstacles/bottleneck of resources currently available HPC systems, and in particular, those at NERSC.
No bottlenecks. Speed of the simulation is limited by strong scalability. NERSC system is well suited for long term climate simulations.
HPC Usage and Methods for the Next 3-5 Years
Anticipated changes to codes, mathematical methods and/or algorithms needed to achieve this project's scientific objectives.
better scaling of the atmospheric model will allow higher resolution simulations to be performed. This will result from a change in the dynamical methods. We expect continued use of finite volume dynamical cores but with a change to the cubed sphere grid and the introduction of higher order spectral element methods.
| Computational Hours Required per Year | ||
|---|---|---|
| Anticipated Number of Cores to be Used in a Typical Production Run | ||
| Anticipated Wallclock to be Used in a Typical Production Run Using the Number of Cores Given Above | ||
| Anticipated Total Memory Used per Run | GB | |
| Anticipated Minimum Memory Required per Core | GB | |
| Anticipated total data read & written per run | GB | |
| Anticipated size of checkpoint file(s) | GB | |
| Anticipated On-Line File Storage Required (Directly Accesible from a Running Job) | GB | Files |
| Anticipated Off-Line Archival Storage Required | GB | Files |
Known or Anticipated architectural requirements (e.g., 2 GB memory/core).
Please list any additional required or important software, services, or infrastructure beyond those listed in the previous section.
It is believed that the dominant HPC architecture in the next 3-5 years will incorporate processing elements composed of 10s-1,000s of individual cores. It is unlikely that a programming model based solely on MPI will be effective, or even supported, on these machines. Do you have a strategy for computing in such an environment? If so, please briefly describe it.
For effective use of the multi-core architecture, we have implemented a hybrid MPI-OpenMP specification of parallelism that allows each component model to find a sweet spot on current HPC architectures. To further take advantage of specialized processors or additional structure in the memory hierarchy is a challenge we are investigating but find ourselves blocked by lack of portable programming standards and techniques.
What Do You Need from NERSC?
Please tell us what you need from NERSC to meet your project's computing needs over the next 3-5 years. Also please feel free to make any general comments.
We need stability and reliable computing over the next three years to support the goals of the assessment work and ongoing development of scalable and extensible earth system models. Some dedicated, high processor count jobs will be required to provide high resolution simulation experiments. A large number of low resolution, ensemble simulation studies will also be required. The subsequent data management problem and the integration of simulation results into the Earth System Grid for analysis and distribution will be a critical element of NERSCs support and our requirements.
