Cubed-sphere spectral element grid.

Stephen Thomas, Richard Loft, and John Dennis, National Center for Atmospheric Research

Research Objectives
The purpose of this project is to develop a scalable dynamical core for atmospheric general circulation models (GCMs).

Computational Approach
We are building a 3D primitive equations dynamical core for an atmospheric GCM. The time-discretization employs either a fully explicit or semi-implicit scheme. We hope to also implement a semi-Lagrangian advection scheme for tracer transport and possibly the full dynamics. The horizontal discretization is based on spectral elements, and the vertical uses a hybrid pressure coordinate with an energy and angular momentum conserving finite difference scheme

Accomplishments
Traditionally, climate model dynamical cores have been based on the spectral transform method because the global spherical harmonic basis functions provide an isotropic representation on the sphere. In addition, it is trivial to implement semi-implicit time-stepping schemes, as the spherical harmonics are eigenfunctions of the Laplacian on the sphere, and the resulting Helmholtz problem is embarrassingly parallel in spectral space. Despite the lack of exploitable parallelism at relatively low climate resolutions, spectral models have exhibited high performance on parallel vector architectures. Achieving high simulation rates on microprocessor clusters at these resolutions has proven difficult due to the communication overhead required by data transpositions and the lack of cache data locality.

As an alternative numerical method, spectral elements maintain the accuracy and exponential convergence rate exhibited by the spectral transform method. Spectral elements also offer several computational advantages on microprocessors. The computations are naturally cache-blocked, and derivatives may be computed using nearest-neighbor communication. An explicit version of a spectral element atmospheric model has demonstrated linear scaling on a variety of parallel machines. Unfortunately, the explicit model suffers from severe time-step restrictions.

We have developed an efficient semi-implicit formulation of this spectral element model. Numerical innovations include a weak formulation of the governing equations and a block-Jacobi preconditioned conjugate gradient solver that is latency tolerant. The parallel implementation is a true hybrid MPI/OpenMP code, and the entire model time-step is threaded over elements using an SPMD parallel region. Cache-blocking in combination with looping over model layers between thread synchronizations for MPI calls results in a per node execution rate that is 25% of peak. We have achieved 361 Gflop/s sustained performance for this model on the NERSC IBM SP, qualifying us as finalists for the 2001 Gordon Bell award.

Significance
Scientific progress in climate modeling depends more on accelerating the integration rate than the resolution. A major goal of our work is to demonstrate that a climate simulation rate of over 100 years per wall clock day is possible on microprocessor-based clusters. This simulation rate is an order of magnitude faster than existing climate models and would represent a major advance in geophysical fluid dynamics.

Publications
S. J. Thomas and R. D. Loft, "Parallel semi-implicit spectral element methods for atmospheric general circulation models," Journal of Scientific Computing 15, 499 (2000).

R. D. Loft and S. J. Thomas, "Semi-implicit spectral element methods for atmospheric general circulation models," in Terascale Computing: The Use of Parallel Processors in Meteorology-Proceedings of the Ninth ECMWF Workshop on High-Performance Computing in Meteorology, November 2000, Reading, England (Singapore, World Scientific Publishers, 2000).

S. J. Thomas and R. D. Loft, "Parallel spectral element atmospheric model," in Proceedings of Parallel CFD 2000 Conference, Trondheim, Norway, May 22-25, 2000, pp. 331-337 (Elsevier North-Holland, 2001).

http://www.scd.ucar.edu/css/staff/thomas