View figure.
When you collect MPI and user event traces using the PCT (as described in Using the Performance Collection Tool), the collected information is saved, on each machine running instrumented processes, as a standard AIX event trace file. In order to view the information contained in these standard AIX trace files, you will first need to convert them into UTE (Unified Trace Environment) interval files. While an AIX event trace file has a time stamp indicating the point in time when an event occurred, UTE interval files take this information to also determine how long an event lasts. Because they include this duration information, UTE interval files are easier to visualize than traditional AIX event trace files. The UTE utilities are:
Figure 37. Unified Trace Environment (UTE) Utilities
View figure.
The preceding figure illustrates the UTE utilities you can use to either generate statistics tables from UTE interval files or view statistics graphically using Argonne National Laboratory's Jumpshot tool. Regardless of whether you want to view the statistics in simple tables or graphically in Jumpshot, the first thing you'll need to do is use the uteconvert utility to create UTE interval files from the AIX trace files ((a)). (See Converting AIX Trace Files Into UTE Interval Trace Files for more information.) Then, if you want to view the statistics in simple tables ((b)), you can use the utestats utility. You can optionally merge multiple UTE files into a single UTE file using the utemerge utility before using the utestats utility to generate the statistics tables. (See Generating Statistics Tables From UTE Interval Trace Files for more information.) If you instead want to view the information contained in the UTE interval files graphically ((c)), you can convert them into SLOG files using the slogmerge utility. The SLOG files are readable by Argonne National Laboratory's Jumpshot Tool. (See Converting UTE Interval Files Into SLOG Files Required By Argonne National Laboratory's Jumpshot Tool for more information.)
The following sections provide an overview of the UTE utilities. Note, however, that this section does not attempt to describe all the options available when using these utilities. For complete reference information on any of the utilities described in this section, refer to their man pages contained in Appendix A, Parallel environment tools commands.
Regardless of whether you want to view the statistics you have collected in simple tables, or graphically in Jumpshot, the first thing you'll need to do is use the uteconvert utility to create UTE interval files from the AIX trace files generated by the PCT. When you collect MPI and user event traces, the collected information is saved, on each machine running instrumented processes, as a standard AIX event trace file. The names of these individual trace files will consist of a common "base name" that you specified using the PCT, followed by a node-specific suffix supplied by the tool itself. Using the uteconvert utility, you can convert either a single AIX trace file into a UTE interval file, or a set of AIX trace files with the same prefix into a set of UTE interval files.
To convert a single AIX trace file into a UTE interval file, simply pass the uteconvert utility the name of the trace file located in the current directory. For example, to convert the AIX trace file mytrace into a UTE interval trace file, enter:
uteconvert mytrace
Using the -o flag, you can optionally specify the name of the output UTE interval file. For example, to specify that the output file should be named outute.
uteconvert -o outute mytrace
To convert a set of AIX trace files into a set of UTE interval files, simply specify the number of files using the -n option, and supply the common "base name" prefix shared by the files. For example, to convert five trace files with the prefix mytraces into UTE interval files, copy the trace files to a common directory and enter:
uteconvert -n 5 mytraces
You can optionally use the -o option to specify a file name prefix for the resulting UTE interval files.
uteconvert -n 5 -o outute mytraces
When you use the -n option, make sure you do not have any old AIX trace files from previous executions of the program still in the directory. The uteconvert utility will process the first n trace files it finds that match the base name prefix.
For complete reference information on the uteconvert utility, refer to its man page in Appendix A, Parallel environment tools commands. If you want to view the statistics information contained in the UTE file(s) in simple tables, refer to Generating Statistics Tables From UTE Interval Trace Files. If you want to view the statistics information contained in the UTE file(s) graphically, refer to Converting UTE Interval Files Into SLOG Files Required By Argonne National Laboratory's Jumpshot Tool.
Once you have created UTE interval trace files (as described in Converting AIX Trace Files Into UTE Interval Trace Files), you can generate statistical tables from them using the utestats utility. In addition to giving you a simple alternative to graphical analysis, the utestats utility can help you identify which traces you want to view in a graphical visualization tool like Jumpshot. This is useful, because you are often unable to view all process threads in a graphical visualization tool. Jumpshot, for example, supports only 64 threads. Using the utestats utility, you can determine which threads are of interest. In addition, if you do not wish to use a graphical visualization tool, you can analyze traces extensively using the utestats utility alone.
By default, six two-dimensional tables are generated. These tables are:
The computed statistic for all tables is the sum or the duration. A Node vs. Processor table would look like the following (where tabs have been replaced by spaces to make the column alignment clearer). The unit of measurement is seconds, so, for example, the accumulated duration of all interval records for CPU 1 of node 0 was 2.258315 seconds.
node/cpu 0 1 0 2.823739 2.258315 1 0.873746 4.241253 2 0.956515 4.322891 3 0.853188 4.334650
You can generate these statistics tables for a single UTE interval file or multiple UTE interval files. You can also generate these statistics tables for a merged UTE interval file. A merged UTE interval file is one that consists of multiple UTE interval files that have been merged into one file by the utemerge utility.
For example, to generate the statistics tables for the UTE interval file mytrace.ute, you would enter:
utestats mytrace.ute
By default, the statistics tables will be printed to standard output. You can, however, redirect them to a file using the -o option on the utestats command. For example, to redirect the statistics tables output by the utestats utility to the file stattables, you would enter:
utestats -o stattables mytrace.ute
As already stated, you can also specify multiple UTE interval files from which the statistics should be generated.
utestats mytrace.ute mytrace2.ute mytrace3.ute
Rather than specify multiple UTE interval trace file names on the utestats command, you could instead use the utemerge utility to first merge the multiple UTE interval trace files into a single UTE interval trace file. To do this, you use the -n option on the utemerge command to indicate the number of files you want to merge, and supply the common "base name" prefix shared by the files. For example:
utemerge -n 3 mytrace
The merged UTE interval file generated by the utemerge utility will, by default, be named trcfile.ute. To specify your own output file name, use the -o option.
utemerge -n 3 -o mergedtrc.ute mytrace
When you use the -n option, make sure you do not have any old UTE interval files from previous executions of the program still in the directory. The utemerge utility will process the first n interval files it finds that match the base name prefix.
You can then generate statistics for the merged UTE interval file using the utestats command.
utestats mergedtrc.ute
For complete reference information on the utestats and utemerge utilities, refer to their man pages in Appendix A, Parallel environment tools commands.
If you would like to view the traces collected by the PCT graphically, you can use the Jumpshot tool developed by Argonne National Laboratory. While Jumpshot is a public domain tool and not part of the PE Benchmarker Toolset, we do provide a utility -- slogmerge -- for converting UTE interval files into the SLOG files required by Jumpshot. You can use the slogmerge utility to:
If you are dealing with a massively parallel job, it is unlikely that you will be able to display all the process threads in Jumpshot. In fact, Jumpshot supports only 64 threads. Rather than merge all the trace files generated from such a job, you will instead want to merge selected trace files. To determine which files to merge, you can first use the utestats utility (as described in Generating Statistics Tables From UTE Interval Trace Files) to determine the characteristics of the files. By analyzing the files first using the utestats utility, you can determine which files contain the interesting information that you want to merge and view in Jumpshot.
To convert a single UTE interval file into a single SLOG file, pass the slogmerge command the name of the file located in the current directory. For example:
slogmerge mytrace.ute
By default, the SLOG file output by the slogmerge utility will be trcfile.slog. Using the -o option on the slogmerge command, however, you can specify an output file name. For example:
slogmerge -o mergedtrc.slog mytrace.ute
To merge multiple UTE interval files into a single SLOG file, use the -n option to indicate the number of files to merge and pass the slogmerge utility the common "base name" prefix of the files. For example, to merge 3 files whose prefix is mytrace, enter:
slogmerge -n 3 mergedtrc.slog mytrace
When you use the -n option, make sure you do not have any old UTE interval files from previous executions of the program still in the directory. The slogmerge utility will process the first n interval files it finds that match the base name prefix.
For complete reference information on the slogmerge utility, refer to its man page in Appendix A, Parallel environment tools commands.