Greg Newman

BES Requirements Worksheet

1.1. Project Information - Large Scale Geophysical Imaging

2. Project Summary & Scientific Objectives for the Next 5 Years

Please give a brief description of your project - highlighting its computational aspect - and outline its scientific objectives for the next 3-5 years. Please list one or two specific goals you hope to reach in 5 years.

To image/invert geophysical data obtained from electromagnetic (EM), seismic and gravity surveys to obtain images of how geophysical attributes, such as electrical conductivity, seismic velocity and density, are distributed in the Earth in 3D. The main thrust of this effort will be development of novel method to jointly invert both large scale seismic and EM data sets. To insure both data sets has similar spatial resolution, sesimic data will be Laplace-Fourier transformed before before inverison with the EM data. Results from this endevor will then be useful in constructing starting velocity models critical for sucessfuly reverse time migration and full wave form sesimic imaging schemes for high resolution imaging of subsurface reflectivity and subtle velocity variations.  
 
We envisage imaging data sets involving 1000's of shots where each shot involves the solution of wave equations in acoustic and electromagnetic wave propagation in 3 dimensions.  
 
Another goal would be to reduce the time to solution investigation the use of GPU's of FPGA's. GPU's and FPGA's offer a potential 10 fold speed up over exisiting and established CPU technologies.

3. Current HPC Usage and Methods

3a. Please list your current primary codes and their main mathematical methods and/or algorithms. Include quantities that characterize the size or scale of your simulations or numerical experiments; e.g., size of grid, number of particles, basis sets, etc. Also indicate how parallelism is expressed (e.g., MPI, OpenMP, MPI/OpenMP hybrid)

EMGeo is the primary code. It uses finite differences to the 3D wave equation. The wave propagation problems are solved using iterative Krylov methods. The imaging component of the algorithm is based on non-linear conjugate gradient and steepest decent optimization scheme. A line search is also involved over the parameter space to find the optimal model step size to reduce the data misfit. Multiple levels of parallelization are exploited, both over the data and model space (domain decomposition)using MPI. Excellent scaling of the algoritm has been oberved up to 32,000 cores.  
 
The computational workhourse in the algorithm is the forward/adjoint solve of the wave equaiton needed to compute predicted data, the gradient of the objective function that is being minimized, and the line search, which insures an acceptable model step in the update. 
 
Each solve is expected to be on the order of 25 million field unknowns, each requiring close 1/2 Gbyte of memory. To be sucessful, algorithm will have to scale up to ten's of thousands of cores, with solver for each shot/source distributed across subsets of these cores. 100s of cores.  

3b. Please list known limitations, obstacles, and/or bottlenecks that currently limit your ability to perform simulations you would like to run. Is there anything specific to NERSC?

The bottleneck is in the krylov solver, and the completion of a matrix-vector multiply. Time to access the cache memory to complete the multiply is limiting. Nothing specific to NERSC with this problem; it is seen acroos multiple platforms outside of NERSC. New technolgies are needed to achieve 10 fold speed up. Perhaps clustered GPU's or FPGA's have potential.  
 
Note algorithm is CPU constrainted not IO constrained 

3c. Please fill out the following table to the best of your ability. This table provides baseline data to help extrapolate to requirements for future years. If you are uncertain about any item, please use your best estimate to use as a starting point for discussions.

Please list any required or important software, services, or infrastructure (beyond supercomputing and standard storage infrastructure) provided by HPC centers or system vendors.

None at this time 

4. HPC Requirements in 5 Years

4a. We are formulating the requirements for NERSC that will enable you to meet the goals you outlined in Section 2 above. Please fill out the following table to the best of your ability. If you are uncertain about any item, please use your best estimate to use as a starting point for discussions at the workshop.

4b. What changes to codes, mathematical methods and/or algorithms do you anticipate will be needed to achieve this project's scientific objectives over the next 5 years.

Need to see a speed up of 10x per shot/source, therefore access time to cache must be reduced accordingly. GPU's or FPGA's technolgy looks promising. 
Other possibilities?

4c. Please list any known or anticipated architectural requirements (e.g., 2 GB memory/core, interconnect latency < 3 #s).

More memory per core; 4 Gbyte, faster interconnect/backbone.

4d. Please list any new software, services, or infrastructure support you will need over the next 5 years.

GPUs with douple precision arithmetic, FPGAs if possible, corresponding support software (Cuda or similar). 

4e. It is believed that the dominant HPC architecture in the next 3-5 years will incorporate processing elements composed of 10s-1,000s of individual cores, perhaps GPUs or other accelerators. It is unlikely that a programming model based solely on MPI will be effective, or even supported, on these machines. Do you have a strategy for computing in such an environment? If so, please briefly describe it.

Our imaging methodology has a hierarchical parallel framework, using several levels of parallelism where both model space and data space is distributed over an arbitrarily large number of cores. This can be adapted to different environments such as GPU clusters.  

New Science With New Resources

To help us get a better understanding of the quantitative requirements we've asked for above, please tell us: What significant scientific progress could you achieve over the next 5 years with access to 50X the HPC resources you currently have access to at NERSC? What would be the benefits to your research field if you were given access to these kinds of resources?

Please explain what aspects of "expanded HPC resources" are important for your project (e.g., more CPU hours, more memory, more storage, more throughput for small jobs, ability to handle very large jobs).

A 50x increase in HPC resources would provides a roadmap to tackle the hardest part of the 3D joint geophysical inverse problem - imaging problems involving different geophysical attributes and data types, where there is no established rock physics model to couple the different attributes and data; i.e. EM, gravity and seismic. It would enable a critical step forward in solving the joint imaging problem and would provide a means to produce a joint image from EM, gravity and seismic data in a self consistent manner. It improves the likelihood of success of full waveform imaging and migration of the seismic trace at its greatest resolution and detail. Under these considerations a 50x increase in computing power opens the possibility to image across multiple scale lengths, incorporating different types of geophysical data and attributes in the process.  
 
Uses of this technology that are of specific interest to the DOE include mapping subsurface hydrological properties, imaging hazardous waste sites, and potential for monitoring sequestration of CO2. It also has important applications in oil and gas and geothermal exploration and reservoir management. A new 3D joint imaging technology offers substantial benefit by combining electrical conductivity and seismic velocity images in such a way that could provide a new powerful approach in mapping offshore hydrocarbons. All of this is of significant relevance to the Department of Energy missions in energy security and environmental stewardship of its facilities.