NERSCPowering Scientific Discovery Since 1974

Debugging and Profiling

NERSC provides many popular debugging and profiling tools. Some of them are general-purpose tools and others are geared toward more specific tasks.

A quick guideline on when to use which debugging tool is as follows:

  • DDT and TotalView: general purpose parallel debuggers allowing users to interactively control the pace of execution of a program using a graphical user interface
  • gdb: serial command line mode debugger; can be useful in quickly examining core files to see where the code crashed (DDT and TotalView can be used for this purpose, too)
  • STAT: used for obtaining call backtraces for all parallel tasks from a live parallel application and displaying a call tree graphically, showing where each task is executing; useful in debugging a hung application

  • ATP: used for generating call backtraces for all parallel tasks when a code crashes; useful in debugging a hung appliation; can be a good starting point if a code crashes with little hint left behind

  • CCDB and lgdb: the unique and great feature is to run two versions of a code (e.g., one working version and an incorrect version, or a code run with two different numbers of tasks) side by side to find out where the two runs start to generate diverging results

  •  Valgrind: a suite of debugging and profiling tools; the best known tool is memcheck which can detect memory errors or memory leaks; other tools include cache profiling, heap memory profiling tools and more

A quick guideline for performance analysis tools below:

  • IPM: a low-overhead easy-to-use tool for getting hardware counters data, MPI function timings, and memory usage
  • CrayPat: a suite of sophisticated Cray tools for a more detailed performance analysis which can show routine-based hardware counters data, MPI message statistics, I/O statistics, etc; in addition to getting performance data deduced from a sampling method, tracing of certain routines (or library routines) can be performed for better understanding of performance statistics associated with the selected routines
  • MAP: a sampling tool for performance metrics; time series of the collected data for the entire run of the code is displayed graphically, and the source code lines are annotated with performance metrics
  • Intel VTune Amplifier XE: a GUI-based tool that can find performance bottlenecks

For more information about how to use a tool, click on the relevant item below.


Darshan is a light weight I/O profiling tool capable of profiling POSIX IO, MPIIO and HDF5 IO. Read More »