NERSCPowering Scientific Discovery Since 1974

PETSc

Description and Overview

PETSc is a suite of data structures and routines for the scalable (parallel) solution of scientific applications modeled by partial differential equations. It employs the MPI standard for parallelism.

On Hopper and Edison PETSc is maintained as one of the Cray supported software packages.  There are also some legacy packages that were installed on Hopper and Carver as part of the DOE ACTS Collection.  As of January 2013, PETSc is no longer being supported by the ACTS group.  If there is a version of PETSc you need not installed on the NERSC systems, please contact the NERSC consultants, who will do their best to support your request.

Cray Versions

Cray PETSc includes the following packages:

  • MUMPS 4.8.3
  • SuperLU 3.1
  • SuperLU_dist 2.3
  • ParMETIS 3.1
  • HYPRE 2.4.0b

ACTS Versions

Note: The ACTS software may be available on NERSC systems but it is no longer supported.  Please transition to the Cray installation.

PETSc is available in many different configurations.  You will see many modulefiles of the form

petsc/X.Y.Z<spec>

where X.Y.Z is the PETSc software release version number and spec describes how the package was built as described below. All versions include a full PETSc implementation with internal linear and non-linear solvers.

spec = blank or (default)
A standard full version of PETSc with debugging flags (an alias for _g described below).
spec = -opkgs
These versions include external packages that interface with PETSc. Currently (Mar 12, 2012), the versions installed on Hopper are 3.2_p5 and these external packages are available: SuperLU, SuperLU_Dist, HYPRE, METIS, MUMPS, and PARMETIS. 
spec = _O (the capital letter "O")
A version built with compiler optimizations enabled.
spec = _g
A version built with the -g debugging flag and no compiler optimizations.
spec = _complex
Uses complex arithmetic.
spec = _c++
C++ bindings.

How to Access PETSc at NERSC

You must use the module command to access the PETSc libraries. To see all the available PETSc modules on a given machine use: module avail petsc.

When you run the module avail petsc command on Hopper you will see the Cray-supported versions listed under /opt/modulefiles and the ACTS-supported options listed under /usr/common/acts/Modules/modulefiles.

To select the default version use module load petsc.  To select a specific version use, for example, module load petsc/3.1
To get some specific usage information for a particular version use, for example, module help petsc/petsc/3.0.0_g_complex

 

Using PETSc on Edison

Note the change in the name of the modulefile on Edison!

module avail cray-petsc

module load cray-petsc

 

 

Using PETSc on Hopper

Cray Versions

The Cray (i.e., not ACTS) installation is the default one on Hopper.  All Cray versions should work under all programming environments, except as noted in the documentation (i.e., Pathscale is not supported in some versions). In addition to different modulefiles for different PETSc release versions, there are also versions for real and complex arithmetic.

By loading the PETSc modulefile, all header and library locations are automatically set corresponding to your environment.

Note: If you use the Cray Fortran compiler to create a PETSc executable, you need to either add the directive

!dir$ PREPROCESS EXPAND_MACROS 

to the source code or add the -F option to the ftn command line.

Using PETSC on Carver

The following is an example but you should look at the tutorials directories (see Documentation, below) for more extensive examples.

mpif90 -I$PETSC_DIR/$PETSC_ARCH/include -I$PETSC_DIR/include -o ex2f ex2f.F

Documentation

On the Cray systems, after you have loaded a Cray-installed PETSc modulefile, you can use man intro_petsc to get some basic information.  More info is available in Cray's Programming Environment User's Guide.  The base document number for this is S–2396.  Cray documents are generally of the form S-xxxx-nn, where nn relates to the release.  The June, 2010 release of the Programming Environment User's Guide is S–2396–50.

There are no man pages available with the ACTS installed versions; however, there are doc and tutorials directories with many useful documents and examples.  Use the module show version command to see the location.

Full documentation about PETSc is available on the PETSc home page.

For questions about using PETsc at NERSC contact the consultants, consult@nersc.gov

Using Packages Within the Cray PETSc Release

MUMPS

You can view a sample MUMPS Fortran example code here.  This code happens to use double precision MUMPS routines.  To compile this code on Hopper using the Cray installation of PETSc and MUMPS you might use:

module load petsc
ftn -Wl,-ydmumps_parallel_analysis_ dsimpletest.F

where the optional -Wl,-y compile flag simply tells the loader to state where the routine "dmumps_parallel_analysis" is loaded from and if you try this you'll see that it comes from the Cray "Third Party Software Library" (tpsl) libdmumps library, which is loaded automatically - both header files and library object archives - with the 'module load petsc' command.  A C example program is here.  Compile with cc simpletest.c.

HYPRE

A C example code is here.  Compile it on Hopper with cc ex11.c after loading the Cray PETSc modulefile. 

Availability

PackagePlatformCategoryVersionModuleInstall DateDate Made Default
petsc hopper libraries/ math 3.1 petsc/3.1_g 2011-02-09 2011-02-09
petsc hopper libraries/ math 3.1 petsc/3.1_O_c++ 2011-02-09
petsc hopper libraries/ math 3.1 petsc/3.1_O 2011-02-09
petsc hopper libraries/ math 3.1 petsc/3.1_g_c++ 2011-02-09
petsc hopper libraries/ math 3.1 petsc/3.1_O_complex 2011-02-09
petsc hopper libraries/ math 3.1 petsc/3.1_g_complex 2011-02-09
petsc hopper libraries/ I/O 3.1.04 petsc/3.1.04 2013-07-25
petsc hopper libraries/ I/O 3.2.02 cray-petsc/3.2.02 2013-07-26
petsc hopper libraries/ math 3.2.02 (cray) cray-petsc/3.2.02 2012-06-13 2012-08-15
petsc hopper libraries/ I/O 3.3.00 cray-petsc-complex/3.3.00 2013-12-04
petsc hopper libraries/ math 3.3.00 (cray) cray-petsc/3.3.00 2012-11-29 2013-02-27
petsc hopper libraries/ I/O 3.3.06 cray-petsc/3.3.06 2013-08-02
petsc hopper libraries/ I/O 3.3.06 cray-petsc-complex/3.3.06 2013-08-02
petsc hopper libraries/ I/O 3.4.2.0 cray-petsc-complex/3.4.2.0 2013-09-25 2013-12-11
petsc hopper libraries/ math 3.4.2.0 (cray) cray-petsc/3.4.2.0 2013-09-25 2013-12-11