Timothy Scheibe

Case Study Worksheet

Project Information - Hybrid Numerical Methods for Multiscale Simulations of Subsurface Biogeochemical Processes

Project Summary (Scientific Objectives)

Please give a brief description of your project and its scientific objectives for the next 3-5 years.

In this SciDAC Science Application, we are developing an integrated multiscale modeling framework with the capability of directly linking different subsurface flow, transport, and reaction process models at continuum, pore, and sub-pore scales. These codes will be modified and/or developed using advanced high-performance component architectures and efficient parallel solvers, and will be integrated into a component-based workflow environment to facilitate seamless integration of codes operating at multiple scales with different physical, biological, and chemical conceptualizations appropriate to the needs of specific simulation problems.

Current HPC Usage and Methods

Please list any required or important software, services, or infrastructure (beyond supercomputing and standard storage infrastructure) provided by HPC centers or system vendors.

PetSC 
Gnuplot, 
ParaView, 
VisIt 
Global Arrays 
CCA

Please list your current primary codes and their main mathematical methods and/or algorithms. Include quantities that characterize the size or scale of your simulations or numerical experiments; e.g., size of grid, number of particles, basis sets, etc. Also indicate how parallelism is expressed (e.g., MPI, OpenMP, MPI/OpenMP hybrid)

PARASIM incorporates three popular particle simulation methods: molecular dynamics (MD) with embedded-atom and Lennard-Jones potentials, dissipative particle dynamics (DPD), and smoothed particle hydrodynamics (SPH). 
 
PARASIM is based on LAMMPS (Large-scale Atomic/Molecular Massively Parallel Simulator) and the Parallel DYNAMO code developed at Sandia National Lab (SNL). 
 
PARASIM was written in Fortran 90 and C. The integration of the particle equation of motion is performed using the fifth order Gear predictor-corrector algorithm or the velocity-Verlet algorithm. PARASIM makes efficient use of memory via dynamic memory allocation. The parallelism is implemented using MPI (message passing interface). There are two method are implemented to achieve parallelism: domain decomposition [1] and force decomposition [2]. 
 
 
SPH_CCA performs smooth particle hydrodynamic simulations (SPH) on systems containing millions of SPH particles and is being used to do pore scale simulation of fluid flow. The code is currently incorporated into the CCA framework. 
 
The code is based on the SPH formulation of the hydrodynamic equations. These represent a random sampling of the continuum hydrodynamic fields by by discrete points (the SPH particles). 
 
The SPH algorithm is a particle-based simulation technique. It uses simple explicit time integration methods to generate trajectories of the SPH particles. 
 
STOMP is a computer model designed to be a general-purpose tool for simulating subsurface flow and reactive transport. STOMP's target capabilities were guided by proposed or applied remediation activities at sites contaminated with volatile organic compounds and/or radioactive material. The simulator's capabilities address a variety of subsurface environments, including nonisothermal conditions, fractured media, multiple-phase systems, nonwetting fluid entrapment, soil freezing conditions, nonaqueous phase liquids, first-order chemical reactions, radioactive decay, solute transport, dense brines, nonequilibrium dissolution, and surfactant-enhanced dissolution and mobilization of organics. It is currently being used to simulate contaminant transport at several sites at the DOE Hanford reservation and other DOE sites. 
 
The STOMP simulator solves the partial-differential equations that describe the conservation of mass or energy quantities in porous media. The simulator has been written with a variable source code that allows the user to choose the solved governing equations (e.g., water mass, air mass, dissolved-oil mass, oil mass, salt mass, thermal energy). Depending on the chosen operational mode, the governing transport equations will be written over one to four phases (e.g., aqueous phase, gas phase, (nonaqueous phase liquid) NAPL phase, ice phase, solid phase). 
 
STOPM applies integrated-volume finite-difference discretization to the physical domain and backward Euler discretization to the time domain. The resulting equations are nonlinear coupled algebraic equations, which are solved using Newton-Raphson iteration. Solute transport, radioactive decay, and first-order chemical reactions are solved using a direct solution technique (e.g., Patankar's power-law formulation, (total variation diminishing) TVD scheme) following the solution of the coupled flow equations. 
 

Please list the known limitations/obstacles/bottleneck of resources currently available HPC systems, and in particular, those at NERSC.

HPC Usage and Methods for the Next 3-5 Years

Anticipated changes to codes, mathematical methods and/or algorithms needed to achieve this project's scientific objectives.

Parallel I/O 
Linear scaling solvers 
Adaptive algorithms (hybrid multiscale code coupling) and adaptive mesh refinement 
Parallel visualization (ray tracing)

Known or Anticipated architectural requirements (e.g., 2 GB memory/core).

Please list any additional required or important software, services, or infrastructure beyond those listed in the previous section.

It is believed that the dominant HPC architecture in the next 3-5 years will incorporate processing elements composed of 10s-1,000s of individual cores. It is unlikely that a programming model based solely on MPI will be effective, or even supported, on these machines. Do you have a strategy for computing in such an environment? If so, please briefly describe it.

We do not have a comprehensive strategy for dealing with multicore architectures at this time. However, we are in close contact with the Global Arrays development team, which is looking at the issue of multicore programming models and will be following developments closely. Also, the component strategy that we have been following for developing our codes will provide a lot of flexibility for incorporating new communication paradigms into our software as they are developed.

What Do You Need from NERSC?

Please tell us what you need from NERSC to meet your project's computing needs over the next 3-5 years. Also please feel free to make any general comments.

Reliable parallel IO libraries 
Archival storage for large datasets 
Linear or near-linear scaling solvers 
Remote parallel visualization 
High bandwidth communication for moving data from NERSC to other sites