NERSC engages in research and development to ensure highly-scalable, productive data, AI and analytics tools are available and deployed at NERSC; as well as to promote the use of cutting-edge data and analytics approaches and technologies in science. Recent activity is outlined below.
Python and Jupyter
Python continues to be popular with users: We measured more than 2000 unique users in 2020. We have also deployed new NERSC monitoring that captures Python library imports for each user. We are working to build infrastructure to analyze this large amount of data and also gain insights about how people are using Python at NERSC. An additional R&D thrust is Python GPU preparedness. We are working to understand and test various Python GPU frameworks to help our users transition their Python code to Perlmutter. Through this work, we lead the Python HPC community - e.g., through co-chairing the HPC track at SciPy
- Rollin Thomas, Laurie Stephey, Annette Greiner, and Brandon Cook, 2021, “Monitoring Scientific Python Usage on a Supercomputer,” Proceedings of the 20th Python in Science Conference (SciPy 2021), (https://doi.org/10.25080/majora-1b6fd038-010).
- Daniel Margala, Laurie Stephey, Rollin Thomas, and Stephen Bailey, 2021, “Accelerating Spectroscopic Data Processing Using Python and GPUs on NERSC Supercomputers,” Proceedings of the 20th Python in Science Conference (SciPy 2021), (https://doi.org/10.25080/majora-1b6fd038-004).
The Jupyter interactive environment is enabling a new mode of computing for scientists at NERSC. Scientists love Jupyter because it combines documentation, visualization, data analytics, and code into a document they can share, modify, and even publish.
DAS and the Usable Software Systems group in CRD are partnering to enhance the Jupyter framework for high performance scientific computing environments and to develop new capabilities in Jupyter for the next generation of scientific analysis. NERSC and CRD have taken a collaborative approach to our development of Jupyter-centric tools, where we collaborate with scientific partners to develop tools that are useful to the broader community and deploy these for all users at NERSC.
Use cases from scientific collaborations such as NCEM, LCLS, and others provide us with insights into how Jupyter Notebooks can help make HPC more accessible to scientists. These collaborations also identify general patterns for enhancements and tools that the entirety of the NERSC user base can capitalize on. Other topics of the research collaboration include finding ways to leverage Jupyter notebooks, containers, and best practices for software development to address reproducibility in science and HPC. Related publications:
- Rollin Thomas and Shreyas Cholia, 2021, “Interactive Supercomputing with Jupyter,” Computing in Science and Engineering,
- Shreyas Cholia, Lindsey Heagy, Matthew Henderson, Drew Paine, Jonathan Hays, Ludovico Bianchi, Devarshi Ghoshal, Fernando Pérez, Lavanya Ramakrishnan, 2020, “Towards Interactive, Reproducible Analytics at Scale on HPC Systems.” IEEE/ACM HPC for Urgent Decision Making, UrgentHPC (https://doi.ieeecomputersociety.org/10.1109/UrgentHPC51945.2020.00011).
- Matthew Henderson, William Krinsman, Shreyas Cholia, Rollin Thomas, Trevor Slaton, 2020. “Accelerating Experimental Science Using Jupyter and NERSC HPC.” In Communications in Computer and Information Science. (https://dx.doi.org/10.1007/978-3-030-44728-1_9).
Dilworth Parkinson, Harinarayan Krishnan, Daniela Ushizima, Matthew Henderson, Shreyas Cholia, 2020. “Interactive Parallel Workflows for Synchrotron Tomography.” XLOOP 2020, (https://doi.ieeecomputersociety.org/10.1109/XLOOP51963.2020.00010).
AI and Machine Learning
MLPerf, from the MLCommons organization, is the current industry standard in AI benchmarking, driving innovation in AI performance for model training and inference workloads. We co-lead the HPC working group in MLCommons, which developed the MLPerf HPC benchmark suite and brought large-scale scientific HPC AI workloads to MLPerf.
MLPerf HPC had its second submission round (v1.0) in 2021, which featured a new Graph Neural Network benchmark for catalyst atomic systems and a new throughput measurement that tests full-system-scale performance by training many AI models concurrently (https://mlcommons.org/en/news/mlperf-hpc-v1/). This round featured over 30 results from 8 leading supercomputing organizations in the U.S. and abroad. NERSC submitted results using the Perlmutter Phase 1 system.
The MLPerf HPC paper (https://arxiv.org/abs/2110.11466) was presented at the Workshop on Machine Learning in HPC Environments, SC21.
Deep Learning Applications
NERSC is involved in several projects to push the state-of-the-art in deep learning for science. Recent publications include:
- Pathak et al., 2022, “FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators.” https://arxiv.org/abs/2202.11214
- Stein et al., 2021, “Mining for strong gravitational lenses with self-supervised learning" - submitted to The Astrophysical Journal and under review. https://arxiv.org/abs/2110.00023
- Hayat et al., 2021, “Self-supervised Representation Learning for Astronomical Images” - published in The Astrophysical Journal, vol. 911, doi:10.3847/2041-8213/abf2c7
- Harrington et al., 2021, “Fast, high-fidelity Lyman α forests with convolutional neural networks” - submitted to The Astrophysical Journal and under review. https://arxiv.org/abs/2106.12662
- Horowitz et al. 2021, “HyPhy: Deep Generative Conditional Posterior Mapping of Hydrodynamical Physics” - submitted to The Astrophysical Journal and under review. https://arxiv.org/abs/2106.12675
- Jiang et al., 2020, “MeshfreeFlowNet: A Physics-Constrained Deep Continuous Space-Time Super-Resolution Framework” - https://arxiv.org/abs/2005.01463, accepted for publication at SC20 and Best Student Paper Finalist
- Chattopadhyay et al., 2020, “Deep spatial transformers for autoregressive data-driven forecasting of geophysical turbulence” - https://eartharxiv.org/cqmb2/, accepted for publication at Climate Informatics 2020
- Mayur Mudigonda et al., 2020, “Climatenet: Bringing The Power Of Deep Learning To Weather And Climate Sciences Via Open Datasets And Architectures,” ICLR 2020
- Karthik Kashinath, Mayur Mudigonda, Kevin Yang, Jiayi Chen, Annette Greiner, and Prabhat Prabhat, 2019, “ClimateNet: Bringing the power of Deep Learning to weather and climate sciences via open datasets and architectures,” Proceedings of the 9th International Workshop on Climate Informatics: CI 2019 (No. NCAR/TN-561+PROC). doi:10.5065/y82j-f154
- Wang et al., 2020, “Towards Physics-informed Deep Learning for Turbulent Flow Prediction,,” KDD, https://doi.org/10.1145/3394486.3403198
- Jiang et al., 2020, “Enforcing Physical Constraints in CNNs through Differentiable PDE Layer,” ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations
- Muszynski et al., 2020, “Atmospheric Blocking Pattern Recognition in Global Climate Model Simulation Data,” ICPR200
- Wu et al., 2020, "Enforcing statistical constraints in generative adversarial networks for modeling chaotic dynamical systems,” JCP, Volume 406, 109209 https://doi.org/10.1016/j.jcp.2019.109209
- Toms et al., 2020, Testing the Reliability of Interpretable Neural Networks in Geoscience Using the Madden-Julian Oscillation, Geosci. Model Dev
- “ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning” - https://arxiv.org/abs/2006.00719, submitted to triple AAAI
- Prabhat et al., "ClimateNet: an expert-labeled open dataset and Deep Learning architecture for enabling high-precision analyses of extreme weather,” Geoscientific Model Development.
- Kashinath et al., 2020, “Physics-informed knowledge-guided Machine Learning for weather and climate modeling: progress and challenges”, Proceedings of the Royal Society, https://royalsocietypublishing.org/doi/full/10.1098/rsta.2020.0093
- Pathak et al., 2020, “Using Machine Learning to Augment Coarse-Grid Computational Fluid Dynamics Simulations,” https://arxiv.org/abs/2010.00072
- Pathak et al., 2020, “ML-PDE: A Framework for a Machine Learning Enhanced PDE Solver,” NeurIPS ML4PS
- Hayat et al. “Estimating Galactic Distances From Images UsingSelf-supervised Representation Learning,” NEeurIPS ML4PS
NERSC has been involved in several Climate Analytics applications, including “Exascale Deep Learning for Climate Analytics,” which was the first deep learning application to achieve 1 Exaflop (FP16) with Tensorflow and Horovod and won the Gordon Bell Prize (Kurth et al. SC18)
Recently, ClimateNet was launched to bring the power of Deep Learning to the climate community by creating community-sourced open-access expert-labeled datasets and architectures for improved accuracy and performance, details of which were presented at ILCR and other venues.
I/O and Data Management
The ExaHDF5 project will productize features and techniques prototyped in earlier projects, explore optimization strategies on upcoming architectures, maintain and optimize existing HDF5 features for ECP applications, and release these new features in HDF5 for broad deployment on HPC systems. Focusing on the challenges of exascale I/O, we will develop technologies based on the massively parallel storage hierarchies that are being built into pre-exascale systems. We will enhance HDF5 software to achieve efficient parallel I/O on exascale systems in ways that will impact a large number of DOE science applications.
Proactive Data Containers (PDC)
Moving toward new paradigms for SSIO in the extreme-scale era, the Proactive Data Container (PDC) project proposes to investigate novel object-based data abstractions and storage mechanisms that take advantage of the deep storage hierarchy and enable proactive, automated performance tuning. In order to achieve these overarching goals, we propose a fundamental new data abstraction called Proactive Data Containers (PDC). A PDC is a container within a locus of storage (memory, NVRAM, disk, etc.) that stores science data in an object-oriented manner. Managing data as objects enables powerful optimization opportunities for data movement and transformations. In this project, we will research: 1) formulation of object-oriented PDCs and their mapping in different levels of the exascale storage hierarchy; 2) efficient strategies for moving data in deep storage hierarchies using PDCs; 3) techniques for transforming and reorganizing data based on application requirements; and 4) novel analysis paradigms for enabling data transformations and user-defined analysis on data in PDCs. The intent of our research is to move the field of HPC SSIO in a direction where it may ultimately be possible to develop scientific applications without the need to perform cumbersome and inefficient tuning to optimize data movement on every system the application runs on.