Kevin Gott

The AMReX software framework for massively parallel block-structured AMR applications has undergone extensive improvements to run efficiently on GPU supercomputers, especially the DOE exascale systems Perlmutter, Frontier and Aurora. The latest generation of computing technologies has led to additional studies in performance and algorithmic design with a focus on usability and scientific achievement. These advancements have demonstrated substantial gains across the AMReX suite of applications, including WarpX, Nyx, Castro, Pele, MFix-Exa and AMR-Wind. 

This talk will give an overview of recent AMReX advancements in GPU design and implementation. Topics include advancements in porting to AMD and Intel software frameworks, advances and remaining deficiencies in GPU performance and new technologies explored to enhance AMReX’s capabilities and prepare for the next-generation of scientific research.

AMReX, the block-structured AMR ECP Co-Design Center, is currently developing its software framework in preparation for the upcoming exascale systems, Frontier and Aurora. AMReX is targeting performance portable strategies that can be implemented natively in C++, require no additional dependencies, and can yield runtime improvements in CUDA, HIP and DPC++. The goal is to make AMReX-based applications as performant as possible on the next-generation exascale systems as soon as they are available. 

This talk will be an overview of some of AMReX’s advancements for targeting these supercomputers, focusing on general purpose algorithms that can be useful to the broader computational community. Discussed features will include asynchronous I/O, automated fused GPU kernel launches and other recent additions that are shaping AMReX’s workflow. An overview of the status of AMReX’s ECP applications will also be presented, highlighting how these algorithms are already impacting the scientific community.

WarpX is a highly scalable, electromagnetic particle-in-cell code developed as part of the Exascale Computing Project. While its primary purpose is to simulate plasma-based particle accelerators, its core PIC routines and advanced algorithms to mitigate numerical artifacts in mesh-refinement simulations can also be leveraged to study particle acceleration in the astrophysical context. In this presentation, we report on the use of WarpX to model pulsar magnetospheres and on the main challenge in using a fully-kinetic approach to model pulsar magnetospheres: the disparate length-scales that span the simulation domain. Indeed, the smallest skin-depth in the critical current-sheet region is six orders of magnitude smaller than the size of the domain required to model the pulsar magnetosphere. Resolving these small length-scales with a uniform grid is intractable even on large supercomputers. As a work-around, existing PIC simulations decrease the scale-difference by reducing the magnetic-field strength of the pulsar. We will present preliminary work on extending WarpX to model pulsar magnetospheres and study the effect of scaling-down the magnetic field-strength on the predictions of Poynting vector and braking-index of the pulsar. We will also explore the use of mesh-refinement for modeling current-sheet regions, which will enable us to extend the current state-of-the-art by enabling simulations with stronger magnetic fields.

The National Energy Research Scientific Computing Center (NERSC) is the mission HPC center for the U.S. Department of Energy Office of Science and supports the needs of 800+ projects and 7,000+ scientists with advanced HPC and data capabilities. NERSC’s newest system, Perlmutter, is an upcoming Cray system with heterogeneous nodes including AMD CPUs and NVIDIA Volta-Next GPUs. It will be the first NERSC flagship system with GPUs. Preparing our diverse user base for the new system is a critical part of making the system successful in enabling science at scale. The NERSC Exascale Science Application Program is responsible for preparing the simulation, data, and machine learning workloads to take advantage of the new architecture. We'll outline our strategy to enable our users to take advantage of the new architecture in a performance-portable way and discuss early outcomes. We'll highlight our use of tools and performance models to evaluate application readiness for Perlmutter and how we effectively frame the conversation about GPU optimization with our wide user base. In addition, we'll highlight a number of activities we are undertaking in order to make Perlmutter a more productive system when it arrives through compiler, library, and tool development. We'll also cover outcomes from a series of case studies that demonstrate our strategy to enable users to take advantage of the new architecture. We'll discuss the programming model used to port codes to GPUs, the strategy used to optimize code bottlenecks, and the GPU vs. CPU speedup achieved so far. The codes will include Tomopy (tomographic reconstruction), Exabiome (genomics de novo assembly), and AMReX (Adaptive Mesh Refinement software framework).

AMReX is a software framework for the development of block-structured AMR algorithms on current and future architectures. AMR reduces the computational cost and memory footprint compared to a uniform mesh while preserving the essentially local descriptions of different physical processes in complex multiphysics algorithms. AMReX supports a number of different time-stepping strategies and spatial discretizations, and incorporates data containers and iterators for mesh-based fields, particle data and irregular embedded boundary (cut cell) representations of complex geometries. Current AMReX applications include accelerator design, additive manufacturing, astrophysics, combustion, cosmology, microfluidics, materials science and multiphase flow. In this talk I will focus on AMReX's strategy for balancing readability, usability, maintainability and performance across multiple applications and architectures.

AMReX is a software framework for building massively parallel block-structured AMR applications using mesh operations, particles, linear solvers and/or complex geometry. AMReX was originally designed to use MPI + OpenMP on multicore systems and recently has ported the majority of its features to GPU accelerators. AMReX’s porting strategy has been designed to allow code teams without a heavy computer science background to port their codes efficiently and quickly with the software framework of their choosing, while minimizing impact to CPU performance or the scientific readability of the code. Further elements of this strategy include providing a clear and concise recommended strategy to application teams, supporting features that allow porting to GPUs in a piece-meal fashion as well as creating sufficiently general interfaces to facilitate adaptation to future changes without user intervention. This talk will give an overview of AMReX's GPU porting strategy to date. This includes a general overview of the porting philosophy and some specific examples that generated noteworthy lessons about porting a large-scale scientific framework. The discussion will also include the current status of AMReX applications that have begun to migrate to hybrid CPU/GPU systems, detail into GPU specific features that have given substantial performance gains, issues with porting a hybrid C++/Fortran code to GPUs and an overview of the limitations of the strategy.

The ability to compute atomic forces through quantum contributions rather than through simple pairwise potentials is one of the most compelling reasons materials scientists use Kohn-Sham pseudopotential density functional theory (DFT). PARSEC is an actively developed real space pseudopotential DFT solver that uses Fortran MPI+OpenMP parallelization. PARSEC provides atomic forces by self-consistently solving for the electronic structure and then summing local and nonlocal contributions. Through experimentation with PARSEC, we present why increasingly bulk synchronous processing and vectorization of the contributions is not enough to fully utilize current HPC hardware. We address this limitation through a demonstration of multithreaded communication approaches for local and nonlocal force computations on Intel Knights Landing supercomputers that yield feasible calculation times for systems of over 20,000 atoms.