Using the Totalview Debugger at NERSC

• Introduction
• Documentation
• Hopper and Franklin
• Bassi
• Jacquard

Introduction

The Totalview is an X Windows-based graphical parallel debugger. Typically a code is compiled for debugging by adding the -g compiler option when building a code. The code can then be run under totalview to step through the program, set break points, examine variables, etc., while viewing the source code. Totalview can also be used to examine core files and attach to running programs.

TotalView Documentation

Totalview Technologies provides a documentation web site. Additional instructions specific to NERSC platforms are described below.

Hopper and Franklin

NERSC has a license to debug applications using up to 512 cores on Hopper and Franklin. If you need to debug an application on more than 512 cores please email the consultants at consult@nersc.gov.

1. Log into Franklin or Hopper with X window forwarding enabled

   This could mean using the -X or -Y option to ssh. The -Y option often works better for Mac OSX.

   % ssh -Y username@franklin.nersc.gov
   

2. Load the Totalview module

   franklin% module load totalview
   

3. Compile your code with debugging flags

   In order to use Totalview (or any debugger), your code must be compiled with the -g option. We also recommend that you do not run with optimization turned on, flags such as -fast.

   A Fortran example:

   franklin% ftn -g -o test.x test.f
   

   A C example:

   franklin% cc -g -o test.x test.c
   

4. Request and interactive batch session

   franklin% qsub -I -V -q debug -lmppwidth=numCores 
   

5. Launch the debugger

   franklin% totalview aprun.tv -a -n [numProcs] test.x
   

   Notice you are using aprun.tv rather than aprun

6. Debug your application

   Once totalview starts, click on "Go", the green arrow, and Totalview will tell you that "aprun.tv" is a parallel job and ask if you want to stop the parallel program. Answer "Yes" and you will be positioned at the beginning of your code. You can then set break points, etc. Refer to the Totalview Technologies documentation web site to learn how to use the advanced features of Totalview.

Bassi

First load the totalview module:

% module load totalview

The first time you load the module an SSH key will be automatically created; this may take up to a minute or so. The key is used internally to launch totalview on batch nodes.

Serial codes

A serial code with an executable named program.x can be debugged with the following command.

% totalview ./program.x <-a optional program args>

Parallel programs

Parallel programs can be debugged interactively using the command line:

% totalview poe -a ./program.x  <optional program args> -nodes num_nodes -tasks num_tasks 

Once totalview starts, click on "Go" and totalview will tell you that "poe" is a parallel job and ask if you want to stop the parallel program. Answer "Yes" and you will be positioned at the beginning of your code. You can then set break points, etc. Refer to the Totalview Technologies documentation web site to learn how to use totalview.

Batch scripts

You can also run totalview in a batch script. Assuming you have used SSH to connect to Bassi and enabled X authentication forwarding (often the default, otherwise enabled with the -X flag on most UNIX systems), the totalview window will open on your display once the batch jobs starts. Obviously, this is not useful if you submit to a batch queue that has a long wait time (and unpredictable start time).

Licenses

The license for Bassi supports debugging parallel codes using up to 64 tasks.

Jacquard

Please see the Jacquard totalview page.

Licenses

Totalview on Jacquard supports debugging parallel programs using up to 32 tasks.