NERSCPowering Scientific Discovery Since 1974

NERSC Initiative for Scientific Exploration (NISE) 2011 Awards

Performance Enhancements for Three-Dimensional Fast Fourier Transforms Library P3DFFT

Dmitry Pekurovsky, University of California San Diego

NISE project m1243.

NISE Award: 350,000 Hours
Award Date: March 2011

Three-dimensional Fast Fourier Transforms are used in a wide range of scientific computing applications, such as DNS turbulence, astrophysics, material science, oceanography, 3D tomography. They are often the bottleneck for performance of such codes on petascale systems. Any performance improvements in P3DFFT will translate directly into benefits for many users in these fields, since this open-source library is well-known and used as a building block in cutting-edge research codes.

P3DFFT is an open-source library for ultra-scalable FFTs in three dimensions ( It is used in many applications designed for large-scale simulations on Peta-scale systems. The library has demonstrated excellent scalability on up to O(10^5) cores on systems such as Jaguar and IBM BG.

This project will be focused on further improving performance and scalability of P3DFFT, putting it in a position to take advantage of the latest technological advances in supercomputer design. This work will follow two main directions.

  1. Currently P3DFFT relies on MPI_Alltoall(v) for a crucial global transpose operation. This is a blocking operation and does not allow any overlap of communication with computation.Such overlap will be explored, through use of one-sided communication (via MPI-2 or SHMEM).
  2. The multicore design of Hopper, as well as most other petascale platforms, necessitates adaptation of the default standard MPI programming model. In particular, a hybrid MPI/OpenMP implementation will be developed and tested,in order to make sure the nodes and interconnect are used in an optimal way.