NERSCPowering Scientific Discovery Since 1974

Mark Jarrell

BES Requirements Worksheet

1.1. Project Information - SciDAC: Next Generation Multi-Scale Quantum Simulation Software for Strongly Correlated Materials

Document Prepared By

Mark Jarrell

Project Title

SciDAC: Next Generation Multi-Scale Quantum Simulation Software for Strongly Correlated Materials

Principal Investigator

Mark Jarrell

Participating Organizations

Louisiana State University 
Ohio SuperComputer Center 
UC Davis 
ORNL 

Funding Agencies

 DOE SC  DOE NSA  NSF  NOAA  NIH  Other:

2. Project Summary & Scientific Objectives for the Next 5 Years

Please give a brief description of your project - highlighting its computational aspect - and outline its scientific objectives for the next 3-5 years. Please list one or two specific goals you hope to reach in 5 years.

Our project involves the development and application of the multi-scale many body (MSMB) approach for the study of strongly correlated electronic systems. MSMB is a green function approach. It uses QMC to treat correlations at short length scales explicitely. Intermediate length scales are treated with a two-particle diagrammatic parquet method, and the longest length scales are treated with a dynamical mean field. 
 
The parquet approach requires 100's of GB of memory for simple problems and far more for the study of realistic model systems parameterized by first principles LDA. 
 
In the next 2 years, we will develop efficient methods to solve and iterate two-particle equations and a stable method for solving the parquet equations.

3. Current HPC Usage and Methods

3a. Please list your current primary codes and their main mathematical methods and/or algorithms. Include quantities that characterize the size or scale of your simulations or numerical experiments; e.g., size of grid, number of particles, basis sets, etc. Also indicate how parallelism is expressed (e.g., MPI, OpenMP, MPI/OpenMP hybrid)

QMC codes include a continuous time, Hirsch-Fye and linear in beta Determinantal QMC solvers. 
 
There are two parquet codes. One distributes the rank-three tensors for the vertices and susceptibilities according to one external label q per core. The second is a hybrid parallel code which distributes them according to one q per shared memory node. 
 
The QMC codes scale efficiently to several thousand cores and the first parquet code scales efficiently to a few tens of thousands of cores. The second parquet code has not yet been tested exhaustively. 

3b. Please list known limitations, obstacles, and/or bottlenecks that currently limit your ability to perform simulations you would like to run. Is there anything specific to NERSC?

The main limitation of QMC is the minus sign problem, which cause it to scale expoentially in the space-time volume of the cluster (rather than its cube). This limits studyies using this code to about 30 correlated orbitals.  
 
The "parquet" code requires the self consistent solution of the parquet, Schwinger-Dyson, and Bethe-Salpeter equations. The main limitation of the parquet code is an instability of the parquet equations at low T and large values of the correlation scale. Additional limitations involve the latency of rank-three tensor rotations that are involved in the self consistency step between each Bethe-Salpeter and parquet solution. 

3c. Please fill out the following table to the best of your ability. This table provides baseline data to help extrapolate to requirements for future years. If you are uncertain about any item, please use your best estimate to use as a starting point for discussions.

Facilities Used or Using

 NERSC  OLCF  ACLF  NSF Centers  Other:  jaguar and karaken XT5

Architectures Used

 Cray XT  IBM Power  BlueGene  Linux Cluster  Other:  

Total Computational Hours Used per Year

 25000000 Core-Hours

NERSC Hours Used in 2009

 0 Core-Hours

Number of Cores Used in Typical Production Run

 1000(QMC) 10000(parquet)

Wallclock Hours of Single Typical Production Run

 4

Total Memory Used per Run

 200--1000 GB

Minimum Memory Required per Core

 2G GB

Total Data Read & Written per Run

 GB

Size of Checkpoint File(s)

1G for QMC GB

Amount of Data Moved In/Out of NERSC

 GB per  

On-Line File Storage Required (For I/O from a Running Job)

 GB and  Files

Off-Line Archival Storage Required

 GB and  Files

Please list any required or important software, services, or infrastructure (beyond supercomputing and standard storage infrastructure) provided by HPC centers or system vendors.

 

4. HPC Requirements in 5 Years

4a. We are formulating the requirements for NERSC that will enable you to meet the goals you outlined in Section 2 above. Please fill out the following table to the best of your ability. If you are uncertain about any item, please use your best estimate to use as a starting point for discussions at the workshop.

Computational Hours Required per Year

 

Anticipated Number of Cores to be Used in a Typical Production Run

 

Anticipated Wallclock to be Used in a Typical Production Run Using the Number of Cores Given Above

 

Anticipated Total Memory Used per Run

 GB

Anticipated Minimum Memory Required per Core

 GB

Anticipated total data read & written per run

 GB

 

 

Anticipated size of checkpoint file(s)

 GB

Anticipated On-Line File Storage Required (For I/O from a Running Job)

 GB and  Files

Anticipated Amount of Data Moved In/Out of NERSC

 GB per  

Anticipated Off-Line Archival Storage Required

 GB and  Files

4b. What changes to codes, mathematical methods and/or algorithms do you anticipate will be needed to achieve this project's scientific objectives over the next 5 years.

4c. Please list any known or anticipated architectural requirements (e.g., 2 GB memory/core, interconnect latency < 3 #s).

4d. Please list any new software, services, or infrastructure support you will need over the next 5 years.

4e. It is believed that the dominant HPC architecture in the next 3-5 years will incorporate processing elements composed of 10s-1,000s of individual cores, perhaps GPUs or other accelerators. It is unlikely that a programming model based solely on MPI will be effective, or even supported, on these machines. Do you have a strategy for computing in such an environment? If so, please briefly describe it.

 

New Science With New Resources

To help us get a better understanding of the quantitative requirements we've asked for above, please tell us: What significant scientific progress could you achieve over the next 5 years with access to 50X the HPC resources you currently have access to at NERSC? What would be the benefits to your research field if you were given access to these kinds of resources?

Please explain what aspects of "expanded HPC resources" are important for your project (e.g., more CPU hours, more memory, more storage, more throughput for small jobs, ability to handle very large jobs).