Home	About	News & Publications	HPC Users	Projects
Systems	Services	Analytics	Status & Statistics	Help

Package	Platform	Version	Module	Docs
IPM	bassi	0.918	ipm	Vendor
IPM	davinci	0.9122	ipm	Vendor
IPM	jacquard	0.918	ipm	Vendor

(*) Denotes limited support

Integrated Performance Monitoring (IPM)

IPM is a portable profiling infrastructure developed at NERSC which outputs a report on the execution of parallel jobs. IPM reports hardware counters data, MPI function timings, and memory usage. IPM is an enhanced replacement for NERSC's retired poe+ utility.

Usage

To use IPM load the ipm module before running your executable:

s00509> module load ipm

You do not need to recompile or relink your program to use IPM; just run your executable after loading the ipm module.

Trouble shooting tips if you are not getting IPM output:

If you are running in batch verify that your shell environment is used for the job. On Bassi, add the line "#@ environment = COPY_ALL" to your script. On Jacquard make sure you have the line "#PBS -V". If you are still having problems try explicitly adding "module load ipm" to your script right before the executable command.
ksh shell: There are a few issues we are currently working on to make module function definitions propagated into the script environment with the ksh shell. A work around until this gets fixed is to add the line ". $HOME/.profile" to your script before any module commands.
On Jacquard the mpiexec wrapper is currently not working, use mpirun instead
IPM writes its output to "standard out" (stdout). User applications should not redefine stdout. For programs written in Fortran, this will happen if unit 6 is opened or closed. For programs written in C, this will happen if file descriptor 1 is opened or closed (the corresponding < stdio.h > variable is FILE* stdout; similarly, the < iostream > variable is std::cout.

Output and Results

Once the module has been loaded each parallel code will, upon completion, print a concise report to standard out. Note: the program must complete normally in order for IPM to produce any results.

Results on the web

Detailed results are available the day after the job completed on the Bassi Completed Jobs page. Click on the StepID to view IPM information, if available.

Options

The interface to IPM is through environment variables and MPI_Pcontrol. The environment variable interface is selecetd at execute/submit time while the later allows for dynamic control of IPM. A description of environment variables supported on bassi is given below. A description of the MPI_Pcontrol interface is included in the main IPM documentation

Variable	Values	Description
IPM_REPORT	terse	(default) Aggregate wallclock time, memory usage and flops are reported along with the percentage of wallclock time spent in MPI calls.
	full	Each HPM counter is reported as are all of wallclock, user, system, and MPI time. The contribution of each MPI call to the communication time is given.
	none	No report
IPM_MPI_THRESHOLD	0.0 < x < 1.0	Only report MPI routines using more than a fraction x of the total MPI time.
IPM_HPM	1,2,3,4,scan

POWER5 Events

Event	Description
PM_FPU_1FLOP	An (either) FPU executed one flop instruction (does not include FMA instructions)
PM_FPU_FMA	An (either) FPU executed one multiply-add instruction
PM_INST_CMPL	Instructions completed
PM_LD_REF_L1	L1 Data cache load references
PM_RUN_CYC	Run cycles
PM_ST_REF_L1	L1 Data cache store references

Features and Futures

Items in plan to be completed on bassi:

LAPI and switch packet profiling
profiling of serial and non MPI/LAPI jobs
web interface to job profiles

IPM is newly developed Open Source software. If you have suggestions, questions, or bug reports please direct them to David Skinner

HPC Users

Resources

Software

Tools