SuperLU Solver Supercharges NIMROD
Developers of the NIMROD code, which is used to simulate fusion reactor plasmas, collaborated with members of the SciDAC Terascale Optimal PDE Simulations Center to implement the SuperLU linear solver software within NIMROD. As a result, NIMROD runs four to five times faster for cutting-edge simulations of nonlinear macroscopic electromagnetic dynamics—with a corresponding increase in scientific productivity.
The NIMROD project, funded by the DOE Office of Fusion Energy Sciences and the SciDAC Center for Extended Magnetohydrodynamic Modeling (CEMM), is developing a modern computer code suitable for the study of long-wavelength, low-frequency, nonlinear phenomena in fusion reactor plasmas. These phenomena involve large-scale changes in the shape and motion of the plasma and severely constrain the operation of fusion experiments. CEMM also supports a complementary plasma simulation code, AMRMHD, which uses different mathematical formulations (see below).
By applying modern computational techniques to the solution of extended magnetohydrodynamics (MHD) equations, the NIMROD team is providing the fusion community with a flexible, sophisticated tool which can lead to improved understanding of these phenomena, ultimately leading to a better approach to harnessing fusion energy. Since the beginning of the project, the NIMROD code has been developed for massively parallel computation, enabling it to take full advantage of the most powerful computers to solve some of the largest problems in fusion. The team’s primary high-end computing resource is Seaborg at NERSC.
NIMROD’s algorithm requires solution of several large sparse matrices in parallel at every time step in a simulation. The stiffness inherent in the physical system leads to matrices that are ill-conditioned, since rapid wave-like responses provide global communication within a single time-step. The preconditioned conjugate gradient (CG) solver that has been used was the most computationally demanding part of the algorithm, so Carl Sovinec of the NIMROD team consulted with the SciDAC Terascale Optimal PDE Simulations (TOPS) Center to find a replacement.
Conversations with David Keyes of Old Dominion University, Dinesh Kaushik of Argonne National Laboratory, and Sherry Li and Esmond Ng of Lawrence Berkeley National Laboratory pointed to SuperLU as a possibly more efficient matrix solver for NIMROD. SuperLU is a library of software for solving nonsymmetric sparse linear systems in parallel using direct methods.
It took less than a month to implement, test, and release SuperLU into the full NIMROD production code, and the performance improvements were dramatic. For two-dimensional linear calculations of MHD instabilities, NIMROD runs 100 times faster with SuperLU than it does with the CG solver. While linear calculations do not require supercomputer resources, they are extremely useful for preliminary explorations that help determine which cases require in-depth nonlinear simulations. For linear problems, this performance improvement makes NIMROD competitive with special-purpose linear codes.
For cutting-edge three-dimensional, nonlinear tokamak simulations (Figure 2), which require supercomputing resources, NIMROD with SuperLU runs four to five times faster than before. This improved efficiency yields four to five times more physics results for the same amount of time on the computer—a major improvement in scientific productivity.
 |
| Figure 2 Full 3D numerical simulation of plasma particle drift orbits in a tokamak. From Charlson C. Kim, Scott E. Parker, and the NIMROD team, “Kinetic particles in the NIMROD fluid code,” 2003 Sherwood Fusion Theory Conference, Corpus Christi, Texas. |
The nonlinear simulations accumulate relatively small changes in the matrix elements at each time step, but there are no changes to the sparsity pattern. The NIMROD implementation allows SuperLU to reuse its factors repeatedly until the matrix elements accumulate a significant change and refactoring becomes necessary. Refactoring makes the performance improvement less dramatic in nonlinear simulations than in linear calculations, but it is still very significant.
The net result is that computationally demanding simulations of macroscopic MHD instabilities in tokamak plasmas, which until now were considered too difficult, have become routine.
Research funding: FES, SciDAC