AI/ML

AI and Machine Learning

NERSC engages in research and development for scalable, powerful, and easy-to-use artificial intelligence and machine learning (AI/ML) in scientific computing. These activities include benchmarking both existing and novel approaches on NERSC and other HPC systems, as well as deep engagements with domain scientists applying AI/ML to problems across the DOE Science portfolio.

NERSC engagement with cosmologists applying self-supervised learning (SSL) to cosmological sky surveys

MLPerf HPC

MLPerf, from the MLCommons organization, is the current industry standard in AI benchmarking, driving innovation in AI performance for model training and inference workloads. We co-lead the HPC working group in MLCommons, which developed the MLPerf HPC benchmark suite and brought large-scale scientific HPC AI workloads to MLPerf.

MLPerf HPC had its second submission round (v1.0) in 2021, which featured a new Graph Neural Network benchmark for catalyst atomic systems and a new throughput measurement that tests full-system-scale performance by training many AI models concurrently (https://mlcommons.org/en/news/mlperf-hpc-v1/). This round featured over 30 results from 8 leading supercomputing organizations in the U.S. and abroad. NERSC submitted results using the Perlmutter Phase 1 system.

The MLPerf HPC paper (https://arxiv.org/abs/2110.11466) was presented at the Workshop on Machine Learning in HPC Environments, SC21.

Deep Learning Applications

NERSC is involved in several projects to push the state-of-the-art in deep learning for science. Recent publications include:

Jacobus et al. 2023. "Reconstructing Lyα Fields from Low-resolution Hydrodynamical Simulations with Deep Learning" Published in The Astrophysical Journal 10.3847/1538-4357/acfcb5
Brenowitz et al., 2024. "A Practical Probabilistic Benchmark for AI Weather Models" https://arxiv.org/abs/2401.15305
McCabe et al., 2023. "Towards stability of autoregressive neural operators" Published in Transactions on Machine Learning Research. h ttps://arxiv.org/abs/2306.10619
Kurth et al., 2023. "Fourcastnet: Accelerating global high-resolution weather forecasting using adaptive fourier neural operators" PASC Best Paper. doi:10.1145/3592979.3593412
Chen et al., 2024, "Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning". https://arxiv.org/abs/2402.15734
Subramanian et al., 2023, "Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior." Accepted to NeurIPS 2023. https://arxiv.org/abs/2306.00258
Pathak et al., 2022, “FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators.” https://arxiv.org/abs/2202.11214
Stein et al., 2021, “Mining for strong gravitational lenses with self-supervised learning" - submitted to The Astrophysical Journal and under review. https://arxiv.org/abs/2110.00023
Hayat et al., 2021, “Self-supervised Representation Learning for Astronomical Images” - published in The Astrophysical Journal, vol. 911, doi:10.3847/2041-8213/abf2c7
Harrington et al., 2021, “Fast, high-fidelity Lyman α forests with convolutional neural networks” - submitted to The Astrophysical Journal and under review. https://arxiv.org/abs/2106.12662
Horowitz et al. 2021, “HyPhy: Deep Generative Conditional Posterior Mapping of Hydrodynamical Physics” - submitted to The Astrophysical Journal and under review. https://arxiv.org/abs/2106.12675
Jiang et al., 2020, “MeshfreeFlowNet: A Physics-Constrained Deep Continuous Space-Time Super-Resolution Framework” - https://arxiv.org/abs/2005.01463, accepted for publication at SC20 and Best Student Paper Finalist
Chattopadhyay et al., 2020, “Deep spatial transformers for autoregressive data-driven forecasting of geophysical turbulence” - https://eartharxiv.org/cqmb2/, accepted for publication at Climate Informatics 2020
Mayur Mudigonda et al., 2020, “Climatenet: Bringing The Power Of Deep Learning To Weather And Climate Sciences Via Open Datasets And Architectures,” ICLR 2020
Karthik Kashinath, Mayur Mudigonda, Kevin Yang, Jiayi Chen, Annette Greiner, and Prabhat Prabhat, 2019, “ClimateNet: Bringing the power of Deep Learning to weather and climate sciences via open datasets and architectures,” Proceedings of the 9th International Workshop on Climate Informatics: CI 2019 (No. NCAR/TN-561+PROC). doi:10.5065/y82j-f154
Wang et al., 2020, “Towards Physics-informed Deep Learning for Turbulent Flow Prediction,,” KDD, https://doi.org/10.1145/3394486.3403198
Jiang et al., 2020, “Enforcing Physical Constraints in CNNs through Differentiable PDE Layer,” ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations
Muszynski et al., 2020, “Atmospheric Blocking Pattern Recognition in Global Climate Model Simulation Data,” ICPR200
Wu et al., 2020, "Enforcing statistical constraints in generative adversarial networks for modeling chaotic dynamical systems,” JCP, Volume 406, 109209 https://doi.org/10.1016/j.jcp.2019.109209
Toms et al., 2020, Testing the Reliability of Interpretable Neural Networks in Geoscience Using the Madden-Julian Oscillation, Geosci. Model Dev
“ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning” - https://arxiv.org/abs/2006.00719, submitted to triple AAAI
Prabhat et al., "ClimateNet: an expert-labeled open dataset and Deep Learning architecture for enabling high-precision analyses of extreme weather,” Geoscientific Model Development.
Kashinath et al., 2020, “Physics-informed knowledge-guided Machine Learning for weather and climate modeling: progress and challenges”, Proceedings of the Royal Society, https://royalsocietypublishing.org/doi/full/10.1098/rsta.2020.0093
Pathak et al., 2020, “Using Machine Learning to Augment Coarse-Grid Computational Fluid Dynamics Simulations,” https://arxiv.org/abs/2010.00072
Pathak et al., 2020, “ML-PDE: A Framework for a Machine Learning Enhanced PDE Solver,” NeurIPS ML4PS
Hayat et al. “Estimating Galactic Distances From Images UsingSelf-supervised Representation Learning,” NeurIPS ML4PS