NERSCPowering Scientific Discovery Since 1974

Totalview

Description

TotalView from Rogue Wave Software is a parallel debugging tool that can be run with up to 512 processors.

It provides both X Windows-based Graphical User Interface (GUI) and command line interface (CLI) environments for debugging. The performance of the GUI can be greatly improved if used in conjunction with free NX software.

The TotalView documentation web page is a good resource for learning more about some of the advanced TotalView features.

Accessing Totalview at NERSC

To use TotalView at NERSC, first load the TotalView modulefile to set the correct environment settings with the following command:

% module load totalview

Compiling Code to Run with TotalView

In order to use TotalView, code must be compiled with the -g option. We also recommend that you do not run with optimization turned on, flags such as -fast.

A Fortran example:

% ftn -g -o testTV_ex testTV.f          # on Hopper or Edison
% mpif90 -g -o testTV_ex testTV.f       # on Carver

A C example:

% cc -g -o testTV_ex testTV.c           # on Hopper or Edison
% mpicc -g -o testTV_ex testTV.c        # on Carver

Starting a Job with TotalView

Be sure to log in with an X window forwarding enabled. This could mean using the -X or -Y option to ssh. The -Y option often works better for Mac OSX.

% ssh -X username@hopper.nersc.gov

After loading the TotalView module and compiling with the -g option, request and interactive batch session on Hopper, Edison or Carver.

% qsub -I -V -q debug -lmppwidth=numCores                  # on Hopper or Edison
% qsub -I -V -q debug -lnodes=numNodes:ppn=numTasksPerNode # on Carver

Then launch the debugger with the totalview command followed by the name of the executable to debug:

% totalview aprun -a -n numTasks ./testTV_ex               # on Hopper or Edison
% totalview mpirun -a -np numTasks ./testTV_ex             # on Carver

In case of debugging a serial code on Carver, you can run as follows:

% totalview ./testTV_ex                                    # on Carver

 The TotalView startup GUI will pop up and show debugging startup parameters. After reviewing them, click OK.

startup parameters

Go to the process window, and click the Go button.

Process window

A small window will then pop up, asking whether the aprun process (or mpirun process on Carver in case of parallel debugging) should be stopped. Click 'Yes'.

Then, the process window will show the source code, waiting for your command.

If the user source code is not displayed in the process window at this stage, select the main routine in the stack trace pane (the upper left frame) of the process window.

To start debugging, create a break point by clicking on a line number in the source pane, and click 'Go'. After that, you can use other buttons ('Next', 'Step', 'Out', etc.).

There is a smaller window called the 'root window' beside the process window, displaying all the (MPI) processes and threads created for the debugging application. You can see MPI task id (rank) and its status for each process or thread from the window.

 You will want to check the value of variables from time to time in order to see whether your program is running as expected. If you want to check a local variable, you can simply look up the 'Stack Frame' section in the process window. For any variable, right-click on the variable in the source pane and select 'Dive'.

Select 'Across Processes' to view the value across processes. If you are in a threaded region and want to view the value across threads, then select the 'Across Threads' menu.

 

Known Limitations

TotalView does not support on Cray systems:

  • Debugging MPI_Spawn(), OpenMP, or Cray SHMEM programs
  • Compiled EVAL points and expressions.
  • Type transformations for the PGI C++ compiler standard template library collection classes
  • Exception handling for the PGI C++ compiler run time library
  • Spawing a process onto the compute processors
  • Machine partitioning schemes, gang scheduling, or batch systems

Memory debugging (MemoryScape) doesn't work on Hopper. Rogue Wave Software is working on the problem.

Availability

PackagePlatformCategoryVersionModuleInstall DateDate Made Default
TotalView babbage applications/ debugging 8T.12.0-1 totalview/8T.12.0-1 2014-01-08 2014-01-08
TotalView carver applications/ debugging 8.11.0 totalview/8.11.0 2013-08-09
TotalView carver applications/ debugging 8.12.0 totalview/8.12.0 2013-08-09 2013-08-12
TotalView carver applications/ debugging 8.13.0 totalview/8.13.0 2014-03-21
TotalView carver applications/ debugging 8.14.0 totalview/8.14.0 2014-08-04 2014-08-04
totalview carver applications/ debugging 8.9.1-0 totalview/8.9.1 2011-06-13 2011-07-07
TotalView carver_sl6 applications/ debugging 8.14.0 totalview/8.14.0 2014-08-10 2014-08-10
TotalView edison applications/ debugging 8.11.0 totalview/8.11.0 2013-01-19 2013-01-19
TotalView edison applications/ debugging 8.12.0 totalview/8.12.0 2013-08-09 2013-08-09
TotalView edison applications/ debugging 8.12.0 totalview-mem-debug/8.12.0 2014-03-21
TotalView edison applications/ debugging 8.13.0 totalview/8.13.0 2014-03-21 2014-03-21
TotalView edison applications/ debugging 8.13.0 totalview-mem-debug/8.13.0 2014-03-21 2014-03-21
totalview hopper applications/ debugging 1.0.6 totalview-support/1.0.6 2011-02-27
TotalView hopper applications/ debugging 8.11.0 totalview/8.11.0 2013-01-18 2013-01-18
TotalView hopper applications/ debugging 8.11.0 totalview-mem-debug/8.11.0 2014-03-21
TotalView hopper applications/ debugging 8.12.0 totalview/8.12.0 2013-08-09 2013-08-09
TotalView hopper applications/ debugging 8.12.0 totalview-mem-debug/8.12.0 2014-03-21
TotalView hopper applications/ debugging 8.13.0 totalview/8.13.0 2014-03-21 2014-03-21
TotalView hopper applications/ debugging 8.13.0 totalview-mem-debug/8.13.0 2014-03-21 2014-03-21
totalview hopper applications/ debugging 8.9.1-0 totalview/8.9.1 2011-06-13 2011-07-07
TotalView hopper applications/ debugging 8.9.1-1 totalview_ccm/8.9.1-1 2012-04-04 2012-04-04