NERSCPowering Scientific Discovery Since 1974

James Muckerman

BES Requirements Worksheet

1.1. Project Information - Computational Studies at BNL of the Chemistry of Energy Production

Document Prepared By

James Muckerman

Project Title

Computational Studies at BNL of the Chemistry of Energy Production

Principal Investigator

James Muckerman

Participating Organizations

Brookhaven National Laboratory

Funding Agencies


2. Project Summary & Scientific Objectives for the Next 5 Years

Please give a brief description of your project - highlighting its computational aspect - and outline its scientific objectives for the next 3-5 years. Please list one or two specific goals you hope to reach in 5 years.

We plan to run NWChem, Gaussian03, GAMESS and Orca on a more than 120-atom system, [(M)2(H2O)2(Q)2(btpyan)]2+, where btpyan = bis-terpyradineanthracene, Q = ortho-quinone or 2-imino-benzoquinone, and M = Ru or Fe, based on the Tanaka catalyst to determine the structures and energetics of intermediates along the catalytic pathway for water oxidation to O2. One of the motivations is the in silico design of an effective water oxidation catalyst based on an earth-abundant metal. Numerous geometry optimizations will be carried out at the density functional level of theory using about 1400 basis functions characterize the various redox and protonated states of intermediates of the catalytic cycle in the gas phase and in aqueous solution. TDDFT calculations requiring at least 40 excited states will be used to predict the UV-Vis spectra to compare with spectroscopic experiments. CASSCF calculations will be carried out to elucidate the multi-configurational (GVB-CI) effects in view of experimental evidence that "anti-ferromeganetic coupling" is a qualitatively inadequate description of the electronic structure of several key intermediates. 
A new thrust in our water oxidation work is to elucidate the mechanism of water oxidation at the surface of a semicomductor photocatalyst or photoanode such as GaN or GaN/ZnO solid solutions in aqueous solution. 
We propose to calculate the thermodynamic hydricities (the tendency to donate a hydride ion, analogous to acidity being the tendency to donate a proton) of various species using the Gaussian03 and NWChem programs. The computational screening is aimed at indentifying appropriate candidate hydride-transfer agents to CO2- or CO-containing hydride-acceptor molecules, and greatly reduce the number of candidate molecules that need to be synthesized in the laboratory. Specifically, the proposed computational work involves the B3LYP 
and/or MP2 calculation of the optimized geometry and free energy of a series of the hydride "acceptor" and "donor" forms of candidate transition metal complexes in acetonitrile or water solution. These data can then be combined with similar calculations of the photogenerated NADPH-like donor molecules 
and their transition state for reaction to predict the reaction thermodynamics and kinetics. The average size of molecule we are screening requires about 
1200 basis functions. We also need to compute the UV-Vis spectra to predict the ability of hydride donors to be created by visible-light MLCT transitions followed by reductive quenching, and to guide and compare with the experimental 
spectroscopic detection of products. 
We propose to extend the atomistic mechanisms of AlH, AlH2, and AlH3 formation on Al(100) surfaces with using Density Functional Theory based kinetic Monte Carlo (KMC) simulations. The DFT calculations will be carried out with VASP and 
Quantum Espresso; the KMC program is our own. In addition, we propose to explore the formation of larger alane species, e.g., Al2H6 of AlnH3n, where n 
is the number of Al atoms in the species, on Al(111) in the presence of steps and edges. The inclusion of larger alane clusters will allow us to carry out a more realistic kinetic study to compare with our current results. The most stable size of cluster could play an important role in the regeneration of the metal hydride for hydrogen storage under ambient conditions.

3. Current HPC Usage and Methods

3a. Please list your current primary codes and their main mathematical methods and/or algorithms. Include quantities that characterize the size or scale of your simulations or numerical experiments; e.g., size of grid, number of particles, basis sets, etc. Also indicate how parallelism is expressed (e.g., MPI, OpenMP, MPI/OpenMP hybrid)

The typical size of our molecular catalyst calculations is about 160 atoms and 1200 basis functions. The main theory we use is Density Functional Theory. The code is usually Gaussian 03 and the parallel scheme is OpenMP. 

3b. Please list known limitations, obstacles, and/or bottlenecks that currently limit your ability to perform simulations you would like to run. Is there anything specific to NERSC?

(1) The major obstacle is the time limit of queue system. This is generally 48 hours or less for any queue implemented at NERSC. Our vibrational frequency calculations (required to obtain the free energy by including zero-point and thermal energy, and entropy corrections to the electronic energy) are often not able to finish in the 48-hour time frame, and we know of no way to checkpoint intermediate results. A special arrangement for a longer queue is required; 
(2) the number of jobs allowed to run simultaneously is also a limit for researchers who run less massively parallel calculations; and  
(3) the size of available SMP machines (number of cpus per node) also limits the size of our simulation model design. If a larger SMP node were available, we could expand the size of our model catalytic systems from ten angstroms to ten nanometers. 

3c. Please fill out the following table to the best of your ability. This table provides baseline data to help extrapolate to requirements for future years. If you are uncertain about any item, please use your best estimate to use as a starting point for discussions.

Facilities Used or Using

 NERSC  OLCF  ACLF  NSF Centers  Other:  

Architectures Used

 Cray XT  IBM Power  BlueGene  Linux Cluster  Other:  

Total Computational Hours Used per Year

 2,000,000 Core-Hours

NERSC Hours Used in 2009

 1,500,000 Core-Hours

Number of Cores Used in Typical Production Run


Wallclock Hours of Single Typical Production Run


Total Memory Used per Run

24 GB

Minimum Memory Required per Core

 1 (perfer 4) GB

Total Data Read & Written per Run

 150 GB

Size of Checkpoint File(s)

 100 GB

Amount of Data Moved In/Out of NERSC

 10 GB per  week

On-Line File Storage Required (For I/O from a Running Job)

 0.004 GB and 30 Files

Off-Line Archival Storage Required

0.060 GB and  400 Files

Please list any required or important software, services, or infrastructure (beyond supercomputing and standard storage infrastructure) provided by HPC centers or system vendors.

Required software: Gaussian 

4. HPC Requirements in 5 Years

4a. We are formulating the requirements for NERSC that will enable you to meet the goals you outlined in Section 2 above. Please fill out the following table to the best of your ability. If you are uncertain about any item, please use your best estimate to use as a starting point for discussions at the workshop.

Computational Hours Required per Year


Anticipated Number of Cores to be Used in a Typical Production Run


Anticipated Wallclock to be Used in a Typical Production Run Using the Number of Cores Given Above


Anticipated Total Memory Used per Run

64 GB

Anticipated Minimum Memory Required per Core

 4 GB

Anticipated total data read & written per run


Anticipated size of checkpoint file(s)

 1,000 GB

Anticipated On-Line File Storage Required (For I/O from a Running Job)

 2 GB and  400 Files

Anticipated Amount of Data Moved In/Out of NERSC

100 GB per  year

Anticipated Off-Line Archival Storage Required

 GB and  Files

4b. What changes to codes, mathematical methods and/or algorithms do you anticipate will be needed to achieve this project's scientific objectives over the next 5 years.

Better parallization of Gaussian on OpenMP

4c. Please list any known or anticipated architectural requirements (e.g., 2 GB memory/core, interconnect latency < 3 #s).

Enable larger SMP calculations (more cores); 8GB of memory per core would be helpful.

4d. Please list any new software, services, or infrastructure support you will need over the next 5 years.

More cores per node, large memory per core, more jobs per user per queue, longer time limit on special queues. 

4e. It is believed that the dominant HPC architecture in the next 3-5 years will incorporate processing elements composed of 10s-1,000s of individual cores, perhaps GPUs or other accelerators. It is unlikely that a programming model based solely on MPI will be effective, or even supported, on these machines. Do you have a strategy for computing in such an environment? If so, please briefly describe it.

Not familiar with the GPU and accelerators. Need high numberical precision; the chemistry of the systems being studied is generally in the 6th or 7th significant figure of the total electronic energy. 

New Science With New Resources

To help us get a better understanding of the quantitative requirements we've asked for above, please tell us: What significant scientific progress could you achieve over the next 5 years with access to 50X the HPC resources you currently have access to at NERSC? What would be the benefits to your research field if you were given access to these kinds of resources?

Please explain what aspects of "expanded HPC resources" are important for your project (e.g., more CPU hours, more memory, more storage, more throughput for small jobs, ability to handle very large jobs).

With new resources, we would be able to treat solvent molecules quantum mechanically. This is significant for the treatment of water quantum mechanically, especially for water splitting catalysis. With a larger SMP computing infrastructure, we could explore even larger catalysts, e.g. molecular catalyst attached to electrodes, semiconductor photocatalysts, or self-assembled polymers. More throughput for small jobs is a key element for studying catalytic redox systems. Different oxidation states can not be combined into one calculation. Electronic structure calculations are defined by the number of nuclei and electrons. That represents one oxidation state. For studying electrochemical catalysis, it is necessary to investigate many possible oxidation states. Therefore, enhancing the throughput for calculating these small-to-medium sized jobs would be very beneficial to the progress of computational catalytic chemistry.