NERSCPowering Scientific Discovery Since 1974

darshan

Darshan has been turned on by default on Hopper and Edison

Description

Darshan is a light weight IO profiling tool capable of profiling POSIX IO, MPIIO and HDF5 IO. We encourage all users to turn on Darshan for their application running on Hopper. Darshan will not only help the users to identify IO bottleneck and improve performance, but also help NERSC to better understand the IO usage of its users and shape its future plans. 

Darshan Enabled on Hopper and Edison

The Darshan module is loaded by default for all Hopper users. Simply recompile your code to allow Darshan to collect statistics. 

Darshan Results

Hopper Completed Jobs Page

On the Completed Jobs page, a Darshan summary is automatically generated for all jobs logged by Darshan. Below shows an example of a job's I/O summary:

 

Locating Logs

Hopper: /scratch2/scratchdirs/darshanlogs/year/month/day/user-executablename_idPBS_JOBID_xxx.gz
Edison: /scratch1/scratchdirs/darshanlogs/year/month/day/user-executablename_idPBS_JOBID_xxx.gz

Note that each aprun command will produce a separate log file. The raw log files are kept for 1 month before deleted from the central location. The log summary is stored in a database for 24/7 access via the web. 

Understanding Results

You can use the darshan-parser tool to analyze your log. To show how much data was read or written in a run:

darshan-parser --total logfile | grep BYTES_READ
darshan-parser --total logfile | grep BYTES_WRITTEN

To show the number of Read/Write operations in the run:

darshan-parser --total logfile | grep POSIX| grep READS
darshan-parser --total logfile | grep POSIX| grep WRITES 

You can also get a distribution of the transaction size, for POSIX read/write:

% darshan-parser --total logfile | grep SIZE_READ|grep -v AGG 

total_CP_SIZE_READ_0_100: 44
total_CP_SIZE_READ_100_1K: 12
total_CP_SIZE_READ_1K_10K: 16
total_CP_SIZE_READ_10K_100K: 12
total_CP_SIZE_READ_100K_1M: 28
total_CP_SIZE_READ_1M_4M: 8
total_CP_SIZE_READ_4M_10M: 8
total_CP_SIZE_READ_10M_100M: 12
total_CP_SIZE_READ_100M_1G: 12
total_CP_SIZE_READ_1G_PLUS: 4 

Or for MPIIO:

% darshan-parser --total logfile | grep SIZE_READ|grep AGG 

total_CP_SIZE_READ_AGG_0_100: 4
total_CP_SIZE_READ_AGG_100_1K: 12
total_CP_SIZE_READ_AGG_1K_10K: 16
total_CP_SIZE_READ_AGG_10K_100K: 12
total_CP_SIZE_READ_AGG_100K_1M: 20
total_CP_SIZE_READ_AGG_1M_4M: 8
total_CP_SIZE_READ_AGG_4M_10M: 8
total_CP_SIZE_READ_AGG_10M_100M: 12
total_CP_SIZE_READ_AGG_100M_1G: 12
total_CP_SIZE_READ_AGG_1G_PLUS: 4

Example

The following script will setup a test directory in $SCRATCH, automatically compile and run it, and point you to the profiling output. 

#if you use bash or sh
/usr/common/usg/darshan/example/rundarshanExample.sh

#if you use tcsh or csh
/usr/common/usg/darshan/example/rundarshanExample.csh

#if you don't know which shell you have
env|grep SHELL 

Overhead of Darshan

The overhead of Darshan is negligible to most of jobs. The plot below shows the overhead of darshan with respect to the job concurrency. The blue markers shows the overhead when each MPI task is reading/writing its own file. The red markers shows the overhead when all MPI tasks are read/writing the same file. For big jobs, it is suggested to use shared file IO (E.g. MPIIO) instead of file pre process. "Overhead" is defined as the time that Darshan uses to communicate the result and write the log file. The overhead of darshan should be un-noticeable for jobs with less than 10000 MPI tasks.  

Versions

PackagePlatformCategoryVersionModuleInstall DateDate Made Default
Darshan edison applications/ debugging 2.2.4 darshan/2.2.4 2013-01-22 2013-01-22
Darshan edison applications/ debugging 2.2.4-old darshan/2.2.4-old 2013-01-31
Darshan edison applications/ debugging 2.2.5-pre1 darshan/2.2.5-pre1 2013-02-04 2013-02-04
Darshan edison applications/ debugging 2.2.5-pre3 darshan/2.2.5-pre3 2013-02-23 2013-02-23
Darshan edison applications/ debugging 2.2.5-pre3-104 darshan/2.2.5-pre3-104 2013-06-10 2013-06-11
Darshan edison applications/ debugging 2.2.6 darshan/2.2.6 2013-05-07
Darshan edison applications/ debugging 2.2.7-105 darshan/2.2.7-105 2013-07-15 2013-07-15
Darshan edison applications/ debugging 2.2.7-106 darshan/2.2.7-106 2013-08-29 2013-08-29
Darshan edison applications/ debugging 2.2.8 darshan/2.2.8 2014-02-25
Darshan hopper applications/ debugging 2.2.3 darshan/2.2.3 2012-11-02
Darshan hopper applications/ debugging 2.2.3-pre1 darshan/2.2.3-pre1 2012-08-22 2012-08-22
Darshan hopper applications/ debugging 2.2.3-pre1-nostat darshan/2.2.3-pre1-nostat 2012-10-27
Darshan hopper applications/ debugging 2.2.4 darshan/2.2.4 2013-01-09 2013-01-09
Darshan hopper applications/ debugging 2.2.4-pre1-bench darshan/2.2.4-pre1-bench 2012-11-09
Darshan hopper applications/ debugging 2.2.4-pre2 darshan/2.2.4-pre2 2012-11-09 2012-11-09
Darshan hopper applications/ debugging 2.2.4-pre3 darshan/2.2.4-pre3 2012-11-13 2013-01-09
Darshan hopper applications/ debugging 2.2.4-pre3-2 darshan/2.2.4-pre3-2 2013-01-09
Darshan hopper applications/ debugging 2.2.5-pre1 darshan/2.2.5-pre1 2013-02-21
Darshan hopper applications/ debugging 2.2.5-pre3 darshan/2.2.5-pre3 2013-02-22
Darshan hopper applications/ debugging 2.2.5-pre3-cle4 darshan/2.2.5-pre3-cle4 2013-02-23 2013-05-17
Darshan hopper applications/ debugging 2.2.6 darshan/2.2.6 2013-05-07 2013-06-11
Darshan hopper applications/ debugging 2.2.8 darshan/2.2.8 2014-03-17

Contact

If you have any question about Darshan at NERSC, please email consult@nersc.gov