NERSCPowering Scientific Discovery Since 1974

MAP

MAP from Allinea Software is a parallel profiler with a simple graphical user interface. It is installed on Hopper, Edison and Carver.

Note that the performance of the X Windows-based MAP Graphical User Interface can be greatly improved if used in conjunction with the free NX software.

Introduction

Allinea MAP is a parallel profiler with simple Graphical User Interface. MAP can be run with up to 512 processors, to profile serial, OpenMP and MPI codes.

The Allinea MAP web page and 'Allinea DDT and MAP User Guide' (available as $ALLINEA_TOOLS_DOCDIR/userguide.pdf after loading an allineatools module) are good resources for learning more about some of the advanced MAP features.

Loading the Allinea Tools Module

To use MAP, first load the 'allineatools' module to set the correct environment settings:

% module load allineatools

Compiling Code to Run with MAP

To collect performance data, MAP uses two small libraries: MAP sampler (map-sampler) and MPI wrapper (map-sampler-pmi) libraries. These must be used with your program, which can be done relatively easily with dynamic linking on Carver. There are somewhat strict rules regarding linking order among object codes and these libraries (please read the User Guide for detailed information). But if you follow the instructions printed by MAP utility scripts, then it is very likely your code will run with MAP.

Your program must be compiled with the -g option to keep debugging symbols, together with optimization flags that you would normally use. If you use the Cray compiler on the Cray machines, we recommend the -G2 option.

Below we show build instructions using a Fortran case, but the C or C++ usage is the same.

On Carver

To minimize problems you should stay with the default dynamic linking mode on Carver.  The libraries are built and preloaded at runtime, so the build procedure is relatively simple in this case.

% mpif90 -g -c testMAP.f
% mpif90 -o testMAP_ex testMAP.o

If you use the PGI compiler, you will need to add an additonal option:

% mpif90 -o testMAP_ex testMAP.o -Wl,--eh-frame-hdr        # with PGI

The MAP sampler and MPI wrapper libraries needed by MAP are built during runtime in the ~/.allinea/wrapper directory. However, if you prefer to build them in the current directory (to save runtime, for example), run the 'make-map-shared-libraries' command and link your code following the instructions printed by the command:

% make-map-shared-libraries
Created the MAP libraries in /your/directory:
libmap-sampler.so (and .so.1, .so.1.0, .so.1.0.0)
libmap-sampler-pmpi.so (and .so.1, .so.1.0, .so.1.0.0)

To instrument a program, add these compiler options:
compilation : -g (and -O3 etc.)
linking : -dynamic -L/your/directory -lmap-sampler-pmpi -lmap-sampler -Wl,--eh-frame-hdr

Note: These libraries must be on the same NFS / Lustre / GPFS filesystem as your
program.

Before running your program (interactively or from a queue), set
LD_LIBRARY_PATH:
export LD_LIBRARY_PATH=/your/directory:$LD_LIBRARY_PATH
mpirun ...

% mpif90 -g -c testMAP.f % mpif90 -o testMAP_ex testMAP.o -dynamic -L/your/directory -lmap-sampler-pmpi -lmap-sampler -Wl,--eh-frame-hdr

You can optionally provide an argument to the 'make-map-shared-libraries' command for a directory where the libraries are to be built.

Save the information about how to modify the LD_LIBRARY_PATH environment variable because you will need it when you run the program with MAP. Csh/tcsh users should use the equivalent 'setenv' command.

On Cray Machines

Buidling an executable for MAP is more complicated on Cray machines. First, you need to explicitly build the MAP sampler and MPI wrapper libraries using 'make-map-static-cray-libraries' for static libraries or 'make-map-cray-libraries' for shared libraries, and link them.

To build a statically-linked executable, follow this procedure:

% make-map-static-cray-libraries
Created the MAP libraries in /your/directory:
   libmap-sampler.a
   libmap-sampler-pmpi.a

To instrument a Cray XT/XK program, add these compiler options:
   compilation : -g (or '-G2' for native Cray fortran) (and -O3 etc.)
       linking : -L/your/directory -lmap-sampler-pmpi -Wl,--undefined,allinea_init_sampler_now -lmap-sampler -lstdc++ -lgcc_eh -Wl,--whole-archive -lpthread -Wl,--no-whole-archive -Wl,--eh-frame-hdr

% ftn -g -c testMAP.f        # Use -G2 instead of -g for the Cray compiler
% ftn -o testMAP_ex testMAP.o -L/your/directory -lmap-sampler-pmpi -Wl,--undefined,allinea_init_sampler_now -lmap-sampler -lstdc++ -lgcc_eh -Wl,--whole-archive -lpthread -Wl,--no-whole-archive -Wl,--eh-frame-hdr

To build a dynamically-linked executable, follow this procedure:

% make-map-cray-libraries
Created the MAP libraries in /your/directory:
   libmap-sampler.so       (and .so.1, .so.1.0, .so.1.0.0)
   libmap-sampler-pmpi.so  (and .so.1, .so.1.0, .so.1.0.0)

To instrument a Cray XT/XK program, add these compiler options:
   compilation : -g (or '-G2' for native Cray fortran) (and -O3 etc.)
       linking : -dynamic -L/your/directory -lmap-sampler-pmpi -lmap-sampler -Wl,--eh-frame-hdr

Note: These libraries must be on a lustre partition, just like your program.

Before running aprun (interactively or from a queue), set LD_LIBRARY_PATH:
   export LD_LIBRARY_PATH=/your/directory:$LD_LIBRARY_PATH
   aprun  ...
% ftn -c -g testMAP.f # Use -G2 for the Cray compiler % ftn -Bdynamic -o testMAP_ex testMAP.o -L/your/directory -lmap-sampler-pmpi -lmap-sampler -Wl,--eh-frame-hdr

Save the information about how to reset the LD_LIBRARY_PATH because you will need it before you run MAP.

Remember that you can provide an optional argument to 'make-map-static-cray-libraries' or 'make-map-cray-libraries' to build the libraries in a directory other than the current working directory.

Starting a Job with MAP

You must log in with an X window forwarding enabled.  One way of ensuring this is to use the -XY flag with the ssh command.

% ssh -XY username@hopper.nersc.gov

After loading the allineatools module and compiling with the -g option, request an interactive batch session on Hopper, Edison and Carver.

% qsub -I -V -q debug -lmppwidth=numCores                      # on Hopper or Edison

% qsub -I -V -q debug -lnodes=numNodes:ppn=numTasksPerNode     # on Carver

Load the 'allineatools' module if you haven't loaded it yet:

% module load allineatools

If you are profiling with a dynamically linked executable and you explicitly created the libraries that MAP needs, using a make-map-* commmand, run the command to modify the LD_LIBRARY_PATH that you saved when you ran the command:

% setenv LD_LIBRARY_PATH /your/directory:$LD_LIBRARY_PATH     # for csh/tcsh

$ export LD_LIBRARY_PATH=/your/directory:$LD_LIBRARY_PATH     # for bash/sh/ksh

If you are using a dynamically linked executable on a Cray machine, you will have to set CRAY_ROOTFS to DSL:

% setenv CRAY_ROOTFS DSL                                      # for csh/tcsh

$ export CRAY_ROOTFS=DSL                                      # for bash/sh/ksh

Then, run the map command followed by the name of the executable to profile:

% map ./testMAP_ex     # or 'map aprun -n ... ./testMAP_ex', 'map mpirun -np ... ./testMAP_ex'

The MAP GUI will pop up with a start up menu.  For profiling choose the option Profile.  You can also choose to Load Profile Data File to view profiling results saved in a file created in a previous MAP run.

Next a submission window will appear with a prefilled path to the executable to run. Select the number of processors on which to run and press Run. To pass command line arguments to a program enter them in the aprun arguments box.

DDT Submit window

MAP will start your program and collect performance data from all processes.

DDT Submit window

By default, MAP lets your program run to completion and will display data for the entire run.  You can also use the 'Stop and Analyze' button and the menu beneath it to control how long to profile your program.

Profiling Results

After completing the run, MAP displays the collected perfromance data using GUI.

DDT Submit window

The window is made of a few sections, providing different view points in presenting collected performance data.

Metrics View

The top section shows the "Metrics view," displaying a timeline of a few selected performance data. By default it shows 'Memory usage (M)' for each task's memorage usage, 'MPI call duration (ms)' for the time spent in MPI calls, and 'CPU floating-point (%)' for the percentage of time each rank spends in floating-point CPU instruction.

Each vertical slice shows the distribution of values across (MPI) tasks at the moment. The minimum, maximum and the mean are displayed, and shading gives you an idea about how data is clustered. A region of large load imbalnce can be visually identified with a fat shaded region.

You can add more metrics to the view area by clicking the 'Metrics' button at the bottom and then adding the ones from the list that interest you. Some availalbe ones are 'Memory usage', 'MPI call duration', 'MPI sent', 'MPI received', 'MPI calls', 'MPI point-to-point', 'MPI collectives', 'CPU time', 'CPU memory access', 'CPU floating-point', 'CPU floating point vector'.  The manual gives an explanation of the metrics.

Source Code View

The center pane shows the source code, annotated with performance information to the left of each line. It shows how much total time was spent computing (green) and communicating (blue) on that line. (This coloring scheme applies to the other area, too.) Only lines that spent at least 0.1% of the total time get charts.

Parallel Stack View

The "Parallel Stack View" area (shown when selecting the 'Parallel Stack View ' tab in the bottom pane) lists the lines where a large wall time was spent, sorted by wallclock time. Clicking on any line jumps the code view to that position in the source code pane.

Project Files View

The "Project Files View" area (shown when selecting the 'Project Files' tab) offers a bottom-up view of the performance of your program. Each file, function or folder comes with a time chart that summarily shows how much wall clock time was spent executing code inside that file/function/folder. 'External Code' is typically system libraries.

DDT Submit window

 When you hover your mouse over the metrics view area, a thin hairline will appear and distribution information for the selected performance metric (i.e. the metric window where your mouse's cursor is located) will be displayed at the bottom of the metrics view area. Similar hairlines will appear in the source code pane and the bottom pane, and they move in sync with the top hairline.

One can also select a region of interest in the horizontal axis (wallclock time) by clicking the left mouse botton, dragging the mouse and then releasing the mouse button. The selected region will appear highlighted. The center and bottom pane's contents will be adjusted by the selection.

DDT Submit window

MAP saves profiling results in a file, 'executablename_#p_yyyy-mm-dd_HH-MM.map' where '#' is for the process count and yyyy-mm-dd_HH-MM' is the time stamp.

% ls -l
...
-rw-------   1 wyang wyang   175839 Aug 16 12:05 jacobi_mpi+hopper_24p_2013-08-16_12-04.map

You can save this file to run MAP on it to examine the profiling results later:

% map jacobi_mpi+hopper_24p_2013-08-16_12-04.map

Running in Command Line Mode

MAP can be run from the command line without GUI, by using the '-profile' option. You can submit a batch job as follows:

% cat runit
#!/bin/csh
#PBS -l mppwidth=24
#PBS -q debug
#PBS -l walltime=10:00
#PBS -j oe

cd $PBS_O_WORKDIR
module load allineatools
map -profile aprun -n 24 ./jacobi_mpi+hopper

% qsub runit
6353383.hopque01

% cat runit.6353383
...
Allinea MAP 4.1-32296, (c) Allinea Software Ltd
Profiling                    : /global/scratch/sd/wyang/MAP/jacobi_mpi+hopper
Processes                    : 24
MPI                          : Cray XT/XE/XK/XC (MPI/shmem/UPC/CAF)
MAP sampler                  : statically linked
MAP MPI wrapper              : statically linked

MAP starting profiling...
aprun:             1    27.55752    
...  
aprun:            10    5.445424    
MAP gathering samples...

MAP analysing program...
MAP generated jacobi_mpi+hopper_24p_2013-08-16_14-19.map
...

% ls -l
...
-rw-------   1 wyang wyang    49029 2013-08-16 14:19 jacobi_mpi+hopper_24p_2013-08-16_14-19.map

Trouble Shooting

If you are having trouble launching MAP try these steps.

Make sure you have the most recent version of the system.config configuration file. The first time you run MAP, you pick up a master template which then gets stored locally in your home directory in ~/.allinea/${NERSC_HOST}/system.config where ${NERSC_HOST} is the machine name: hopper, edison or carver. If you are having problems launching MAP you could be using an older verion of the system.config file and you may want to remove the entire directory:

% rm -rf ~/.allinea/${NERSC_HOST}  

Remove any stale processes that may have been left by MAP.

% rm -rf $TMPDIR/allinea-$USER 

In case of a font problem where every character is displayed as a square, please delete the .fontconfig directory in your home directory and restart ddt.

% rm -rf ~/.fontconfig

Make sure you are requesting an interactive batch session on Hopper, Edison and Carver. NERSC has configured MAP to run from the interactive batch jobs.

% qsub -I -V -q debug -lmppwidth=numCores                      # Hopper or Edison
% qsub -I -V -q debug -lnodes=numNodes:ppn=numTasksPerNode     # on Carver

Finally make sure you have compiled your code with -g. If none of these tips help, please contact the consultants at consult@nersc.gov. 

Installed Versions

PackagePlatformCategoryVersionModuleInstall DateDate Made Default
Allinea tools carver applications/ debugging 4.1-32834 allineatools/4.1-32834 2013-09-02
Allinea tools carver applications/ debugging 4.2-34164 allineatools/4.2-34164 2013-12-16 2013-12-16
Allinea tools carver applications/ debugging 4.2-34404 allineatools/4.2-34404 2014-01-21 2014-02-21
Allinea tools edison applications/ debugging 4.1-33167 allineatools/4.1-33167 2013-09-24 2013-09-24
Allinea tools edison applications/ debugging 4.2-34164 allineatools/4.2-34164 2013-12-18 2013-12-18
Allinea tools edison applications/ debugging 4.2-34404 allineatools/4.2-34404 2014-01-21 2014-02-21
Allinea tools hopper applications/ debugging 4.1-33167 allineatools/4.1-33167 2013-09-24 2013-09-24
Allinea tools hopper applications/ debugging 4.2-34164 allineatools/4.2-34164 2013-12-16 2013-12-16
Allinea tools hopper applications/ debugging 4.2-34404 allineatools/4.2-34404 2014-01-21 2014-02-21