AI/ML

AI and Machine Learning

NERSC engages in research and development for scalable, powerful, and easy-to-use artificial intelligence and machine learning (AI/ML) in scientific computing. These activities include benchmarking both existing and novel approaches on NERSC and other HPC systems, as well as deep engagements with domain scientists applying AI/ML to problems across the DOE Science portfolio.

NERSC engagement with cosmologists applying self-supervised learning (SSL) to cosmological sky surveys

2024 GenAI Call for Proposals

In early 2024, NERSC invited proposals for projects that will leverage Perlmutter to push the state of the art in Generative AI (GenAI) and deep learning for science, as well as produce novel science outcomes. The call was focused on teams with expertise in the use of deep learning for science, a thorough understanding of the scientific domain, and demonstrated proofs of concept. With over 50 proposals, 34 teams were awarded a total of O(300K) GPU node hours based on project scope, relevance to the NERSC mission areas, and readiness to use large computational resources at scale to advance their scientific outcome. Application areas include the use of novel GenAI models and techniques in material sciences, biological sciences, high-energy physics, weather (and climate) sciences, and applied mathematics.

MLPerf HPC

With the explosion of interest in and exploration of AI in the DOE science communities, HPC centers are preparing for a shift toward new AI-enhanced computational workflows. It is imperative that the scientific HPC community be ready to support this emerging workload with representative and robust benchmarks that allow characterization of the computational workload and drive innovation in system and software design.

MLPerf benchmarks from the MLCommons organization are the industry standard benchmark for AI performance—that is, current drivers of innovation in systems and software. NERSC co-founded the HPC working group within MLCommons, which developed the MLPerf HPC benchmark suite to address HPC and scientific AI workloads, and has continued to participate in planning and organization through the MLPerf HPC v3.0 submission round.

During this submission round, the group added the new OpenFold benchmark, a protein-folding application based on DeepMind’s famous AlphaFold2 model, and laid the groundwork for a robust power measurement methodology in coordination with the MLCommons Training group. The MLPerf HPC v3.0 results were published on November 8, 2023 and featured 30 performance results (a 50% increase over the previous year), new submitters, and impressive speed-ups over previous submission rounds (e.g., the latest DeepCAM results are 14X faster than when the benchmark debuted).

NERSC partnered with HPE to submit results to MLPerf HPC v3.0 using Perlmutter in its final configuration with upgraded Slingshot 11 network. The Perlmutter system achieved excellent results across all workloads and factors of node scaling with impressive speedups over the previous v1.0 results, including 5.3X faster results on OpenCatalyst, 2X faster results on CosmoFlow, 1.4X faster results on DeepCAM, and highly competitive results on the new OpenFold benchmark.

Deep Learning Applications

NERSC is involved in several projects to push the state-of-the-art in deep learning for science. Recent publications include:

Jacobus et al. 2023. "Reconstructing Lyα Fields from Low-resolution Hydrodynamical Simulations with Deep Learning" Published in The Astrophysical Journal 10.3847/1538-4357/acfcb5
Brenowitz et al., 2024. "A Practical Probabilistic Benchmark for AI Weather Models" https://arxiv.org/abs/2401.15305
McCabe et al., 2023. "Towards stability of autoregressive neural operators" Published in Transactions on Machine Learning Research. h ttps://arxiv.org/abs/2306.10619
Kurth et al., 2023. "Fourcastnet: Accelerating global high-resolution weather forecasting using adaptive fourier neural operators" PASC Best Paper. doi:10.1145/3592979.3593412
Chen et al., 2024, "Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning". https://arxiv.org/abs/2402.15734
Subramanian et al., 2023, "Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior." Accepted to NeurIPS 2023. https://arxiv.org/abs/2306.00258
Pathak et al., 2022, “FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators.” https://arxiv.org/abs/2202.11214
Stein et al., 2021, “Mining for strong gravitational lenses with self-supervised learning" - submitted to The Astrophysical Journal and under review. https://arxiv.org/abs/2110.00023
Hayat et al., 2021, “Self-supervised Representation Learning for Astronomical Images” - published in The Astrophysical Journal, vol. 911, doi:10.3847/2041-8213/abf2c7
Harrington et al., 2021, “Fast, high-fidelity Lyman α forests with convolutional neural networks” - submitted to The Astrophysical Journal and under review. https://arxiv.org/abs/2106.12662
Horowitz et al. 2021, “HyPhy: Deep Generative Conditional Posterior Mapping of Hydrodynamical Physics” - submitted to The Astrophysical Journal and under review. https://arxiv.org/abs/2106.12675
Jiang et al., 2020, “MeshfreeFlowNet: A Physics-Constrained Deep Continuous Space-Time Super-Resolution Framework” - https://arxiv.org/abs/2005.01463, accepted for publication at SC20 and Best Student Paper Finalist
Chattopadhyay et al., 2020, “Deep spatial transformers for autoregressive data-driven forecasting of geophysical turbulence” - https://eartharxiv.org/cqmb2/, accepted for publication at Climate Informatics 2020
Mayur Mudigonda et al., 2020, “Climatenet: Bringing The Power Of Deep Learning To Weather And Climate Sciences Via Open Datasets And Architectures,” ICLR 2020
Karthik Kashinath, Mayur Mudigonda, Kevin Yang, Jiayi Chen, Annette Greiner, and Prabhat Prabhat, 2019, “ClimateNet: Bringing the power of Deep Learning to weather and climate sciences via open datasets and architectures,” Proceedings of the 9th International Workshop on Climate Informatics: CI 2019 (No. NCAR/TN-561+PROC). doi:10.5065/y82j-f154
Wang et al., 2020, “Towards Physics-informed Deep Learning for Turbulent Flow Prediction,,” KDD, https://doi.org/10.1145/3394486.3403198
Jiang et al., 2020, “Enforcing Physical Constraints in CNNs through Differentiable PDE Layer,” ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations
Muszynski et al., 2020, “Atmospheric Blocking Pattern Recognition in Global Climate Model Simulation Data,” ICPR200
Wu et al., 2020, "Enforcing statistical constraints in generative adversarial networks for modeling chaotic dynamical systems,” JCP, Volume 406, 109209 https://doi.org/10.1016/j.jcp.2019.109209
Toms et al., 2020, Testing the Reliability of Interpretable Neural Networks in Geoscience Using the Madden-Julian Oscillation, Geosci. Model Dev
“ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning” - https://arxiv.org/abs/2006.00719, submitted to triple AAAI
Prabhat et al., "ClimateNet: an expert-labeled open dataset and Deep Learning architecture for enabling high-precision analyses of extreme weather,” Geoscientific Model Development.
Kashinath et al., 2020, “Physics-informed knowledge-guided Machine Learning for weather and climate modeling: progress and challenges”, Proceedings of the Royal Society, https://royalsocietypublishing.org/doi/full/10.1098/rsta.2020.0093
Pathak et al., 2020, “Using Machine Learning to Augment Coarse-Grid Computational Fluid Dynamics Simulations,” https://arxiv.org/abs/2010.00072
Pathak et al., 2020, “ML-PDE: A Framework for a Machine Learning Enhanced PDE Solver,” NeurIPS ML4PS
Hayat et al. “Estimating Galactic Distances From Images UsingSelf-supervised Representation Learning,” NeurIPS ML4PS