Hitchhiker's Guide

What's new in Parallel Environment 3.2?

This release of the Parallel Environment contains a number of functional enhancements, including:

• new PE Benchmarker tools
• improved parallel checkpointing capabilities
• support for 64-bit applications
• MPI enhancements

In addition, PE 3.2 includes these changes:

• PE now supports, and requires, AIX 5L^(TM) 5.1.
• PE Version 2, Release 2 is no longer supported.
• The Dynamic Probe Class Library (DPCL) is no longer a part of PE, though it is still shipped with PE. Instead, DPCL is now an open source offering that supports PE.
• The pedb debugger has been removed.
• The VT parallel tracing facility has been removed.

The following sections describe these functional enhancements and changes in more detail.

New PE Benchmarker tools

This release of PE contains a new suite of applications and utilities that you can use to analyze the performance of programs. This suite of tools is called PE Benchmarker and contains:

• the Performance Collection Tool, which enables you to collect either MPI and user event data, or else hardware and operating system profiles for one or more application processes. This tool is built on dynamic instrumentation technology (the Dynamic Probe Class Library, or DPCL) which enables you to make the decision of what data to collect at run time. Probes are placed in the running executable to collect just the information you requested. This ability typically results in a more acceptable intrusion cost than you would have with more traditional styles of instrumentation.
• a set of Unified Trace Environment (UTE) utilities for converting one or more AIX trace files (as output by the Performance Collection Tool) into UTE interval files. Additional UTE utilities enable you to:
  • generate statistics tables from UTE interval files.
  • convert UTE interval files into a single SLOG file. You can analyze an SLOG file using Jumpshot (a public domain tool developed by Argonne National Laboratory).
• the Profile Visualization Tool, for viewing hardware and operating system profiles collected by the Performance Collection Tool.

Improved parallel checkpointing capabilities

This release of PE includes more flexible parallel checkpoint/restart capabilities. In previous releases, only POE/MPI applications submitted under LoadLeveler^(R) in batch mode could be checkpointed, and there were significant limitations. What's more, a checkpoint sequence could be initiated only by all tasks in the parallel MPI program. In this release, PE's checkpointing capabilities have been extended to allow:

• a user to initiate a checkpoint sequence explicitly by issuing the new command poeckpt. Another new command, poerestart, enables a user to restart a POE job using a checkpoint file.
• a single task in a parallel MPI or LAPI job to initiate a checkpoint sequence. It is no longer necessary for all tasks in the job to call checkpointing functions.
• a system administrator or LoadLeveler to initiate a checkpoint sequence.

Also, many significant limitations of the checkpointing capabilities have been removed, and the checkpointing compiler scripts (mpcc_chkpt, for example) are no longer used.

MPI enhancements

With this release, PE MPI provides all of the functions in the MPI standard, except for the functionality defined in the "Process Creation and Management" chapter of MPI-2.

Additional command for starting MPI jobs

While you can continue to use the poe command to start MPI jobs, this release of PE provides support for the mpiexec command described in the MPI-2 standard. This command is not meant to replace the poe command; instead it is provided as a portable way to start MPI programs, and should prove helpful for applications that target multiple implementations of MPI.

Support for 64-bit applications

Certain architectures that PE supports, such as POWER3 SMP High Node, POWER3 SMP Thin Node, and POWER3 Wide Node, can run applications using a 64-bit address space. Accordingly, the tools that are provided with PE, as well as the MPI threads library, have been enhanced to support 64-bit applications on 64-bit processors.

MPI-IO performance enhancements

To improve the performance of IBM's implementation of MPI-IO, a variety of optimizations have been made to this release of PE. For example, two new POE environment variable/command-line flag pairs and one new file hint have been added. The two new POE environment variable/command-line flag pairs are the:

• MP_IO_BUFFER_SIZE environment variable (-io_buffer_size flag), which enables you to set the default size of buffers used by I/O agents.
• MP_IO_ERRLOG environment variable (-io_errlog flag), which enables you to log errors that occurred at the file system level throughout the MPI-IO application run.

The new file hint, IBM_sparse_access, enables you to specify whether the file access requests from participating tasks are sparse or dense.

Extended collective communication

This release provides the MPI_IN_PLACE and intercommunicator semantic that MPI-2 has added to a number of MPI 1.1 collective communication subroutines. The new subroutines, MPI_ALLTOALLW and MPI_EXSCAN, are also provided. The nonblocking collective subroutines, which have a prefix of MPE_I, are not enhanced.

C++ and FORTRAN90 support

This release provides the C++ bindings described by MPI-2. C++ programmers can now use PE MPI more naturally, as they no longer need to use the C bindings.

FORTRAN programs can now use the mpi module in place of mpif.h.

The nonblocking collective subroutines, which have a prefix of MPE_I, are not enhanced for C++.

MPI-2 external interfaces support

With this release, PE now provides full support of MPI-2 external interfaces in the MPI threads library. This support enables you to layer additional functionality on top of MPI with an interface that is similar to MPI's.

Miscellaneous MPI-2 enhancements

Some of the functions included in the "Miscellany" chapter of MPI-2 were provided in prior releases of PE. This release provides the rest of these miscellaneous functions.

DPCL is now an open source offering

The Dynamic Probe Class Library (DPCL) is no longer a part of the IBM PE for AIX licensed program, but it is still shipped with PE for convenience. Instead, DPCL is now available as an open source offering that supports PE. For more information on the DPCL open source project, go to this World Wide Web address:

 http://oss.software.ibm.com/developerworks/opensource/dpcl/ 

Removal of pedb debugger support

Beginning with this release, PE no longer includes the pedb parallel debugger. As a result, the pedb command is no longer available. The pdbx parallel debugger, however, is still supported; use it instead of pedb to debug your parallel applications.

Removal of VT trace collection support

Beginning with this release, PE no longer supports the parallel trace collection facility formerly used by the PE Visualization Tool (which was removed from PE in release 3.1). Because the usefulness of examining these traces independent of VT is limited, the VT trace facility has been replaced by the more robust tracing capabilities of the new PE Benchmarker suite of tools.

Commands no longer supported

Beginning with this release, PE no longer supports these commands:

• mpcc_chkpt
• mpCC_chkpt
• mpxlf_chkpt
• mpxlf90_chkpt
• mpxlf95_chkpt
• pedb