Description and Overview
PETSc is a suite of data structures and routines for the scalable (parallel) solution of scientific applications modeled by partial differential equations. It employs the MPI standard for parallelism.
On Hopper and Edison PETSc is maintained as one of the Cray supported software packages. There are also some legacy packages that were installed on Hopper and Carver as part of the DOE ACTS Collection. As of January 2013, PETSc is no longer being supported by the ACTS group. If there is a version of PETSc you need not installed on the NERSC systems, please contact the NERSC consultants, who will do their best to support your request.
Cray PETSc includes the following packages:
- MUMPS 4.8.3
- SuperLU 3.1
- SuperLU_dist 2.3
- ParMETIS 3.1
- HYPRE 2.4.0b
Note: The ACTS software may be available on NERSC systems but it is no longer supported. Please transition to the Cray installation.
PETSc is available in many different configurations. You will see many modulefiles of the form
where X.Y.Z is the PETSc software release version number and spec describes how the package was built as described below. All versions include a full PETSc implementation with internal linear and non-linear solvers.
- spec = blank or (default)
- A standard full version of PETSc with debugging flags (an alias for _g described below).
- spec = -opkgs
- These versions include external packages that interface with PETSc. Currently (Mar 12, 2012), the versions installed on Hopper are 3.2_p5 and these external packages are available: SuperLU, SuperLU_Dist, HYPRE, METIS, MUMPS, and PARMETIS.
- spec = _O (the capital letter "O")
- A version built with compiler optimizations enabled.
- spec = _g
- A version built with the -g debugging flag and no compiler optimizations.
- spec = _complex
- Uses complex arithmetic.
- spec = _c++
- C++ bindings.
How to Access PETSc at NERSC
You must use the module command to access the PETSc libraries. To see all the available PETSc modules on a given machine use: module avail cray-petsc on Hopper and Edison and module avail petsc on Carver.
When you run the module avail petsc command on Hopper you will see the Cray-supported versions listed under /opt/modulefiles and the ACTS-supported options listed under /usr/common/acts/Modules/modulefiles. Remember, the ACTS versions are no longer supported at NERSC.
To select the default version use module load cray-petsc on Hopper and Edison. To select a specific version use, for example, module load petsc-complex/3.2.02 .
To get some specific usage information for a particular version use, for example, module help petsc-complex/3.2.02 .
Using PETSc on Edison
Note the change in the name of the modulefile on Edison!
module avail cray-petsc
module load cray-petsc
Using PETSc on Hopper
The Cray (i.e., not ACTS) installation is the default one on Hopper. All Cray versions should work under all programming environments, except as noted in the documentation (i.e., Pathscale is not supported in some versions). In addition to different modulefiles for different PETSc release versions, there are also versions for real and complex arithmetic.
By loading the PETSc modulefile, all header and library locations are automatically set corresponding to your environment.
Note: If you use the Cray Fortran compiler to create a PETSc executable, you need to either add the directive
!dir$ PREPROCESS EXPAND_MACROS
to the source code or add the -F option to the ftn command line.
Using PETSC on Carver
The following is an example but you should look at the tutorials directories (see Documentation, below) for more extensive examples.
mpif90 -I$PETSC_DIR/$PETSC_ARCH/include -I$PETSC_DIR/include -o ex2f ex2f.F
On the Cray systems, after you have loaded a Cray-installed PETSc modulefile, you can use man intro_petsc to get some basic information. More info is available in Cray's Programming Environment User's Guide. The base document number for this is S–2396. Cray documents are generally of the form S-xxxx-nn, where nn relates to the release. The June, 2010 release of the Programming Environment User's Guide is S–2396–50.
There are no man pages available with the ACTS installed versions; however, there are doc and tutorials directories with many useful documents and examples. Use the module show version command to see the location.
Full documentation about PETSc is available on the PETSc home page.
For questions about using PETsc at NERSC contact the consultants, email@example.com
Using Packages Within the Cray PETSc Release
You can view a sample MUMPS Fortran example code here. This code happens to use double precision MUMPS routines. To compile this code on Hopper using the Cray installation of PETSc and MUMPS you might use:
module load petsc
ftn -Wl,-ydmumps_parallel_analysis_ dsimpletest.F
where the optional -Wl,-y compile flag simply tells the loader to state where the routine "dmumps_parallel_analysis" is loaded from and if you try this you'll see that it comes from the Cray "Third Party Software Library" (tpsl) libdmumps library, which is loaded automatically - both header files and library object archives - with the 'module load petsc' command. A C example program is here. Compile with cc simpletest.c.
A C example code is here. Compile it on Hopper with cc ex11.c after loading the Cray PETSc modulefile.
|Package||Platform||Category||Version||Module||Install Date||Date Made Default|
|petsc||hopper||libraries/ math||3.2.02 (cray)||cray-petsc/3.2.02||2012-06-13||2012-08-15|
|petsc||hopper||libraries/ math||3.3.00 (cray)||cray-petsc/3.3.00||2012-11-29||2013-02-27|
|petsc||hopper||libraries/ math||126.96.36.199 (cray)||cray-petsc/188.8.131.52||2013-09-25||2013-12-11|
|petsc||hopper||libraries/ math||184.108.40.206 (cray)||cray-petsc/220.127.116.11||2013-12-18|
|petsc||hopper||libraries/ math||18.104.22.168 (cray)||cray-petsc-complex/22.214.171.124||2013-12-18|
|petsc||hopper||libraries/ math||126.96.36.199 (cray)||cray-petsc/188.8.131.52||2014-06-12|
|petsc||hopper||libraries/ I/O||184.108.40.206 (cray)||cray-petsc-complex/220.127.116.11||2014-06-12|