NERSCPowering Scientific Discovery Since 1974

GTC Medium IPM Profile on Bassi

 

Step ID b0201.214365.0 Job Name gtcMediumIPM
Owner hjw Account mpccc Status Completed
LL class reg_1 Submit class regular Job type Parallel
Nodes 8 Wall secs 1,809 Wall hrs 0.50
MPP secs 694,656 MPP hrs 192.96 Raw Secs 115,776
Submit Jul-16-07 07:24:09 Start Jul-16-07 18:42:24 Wait 11:18:15
Completion Jul-16-07 19:12:34 systime 624 usrtime 112,081
Task Max. Memory (GB) 2.582e-01  
Nodelist b0901, b0801, b0406, b0405, b0805, b0409, b0702, b0205
*Indicates dispatch time

Summary | Hardware Counters | MPI Statistics

  

IPM summary

Executable ./gtcmpi
Number of tasks 64 Aggregate GFlop/sec 39.9283 Average GFlop/sec/task 0.6239
Average wall secs 1.718e+03 Aggregate memory (GB) 16.5175 Average memory/task (GB) 0.2581
Average MPI secs/task 8.372e+01 MPI time % 4.88 Aggregate MPI calls made 6.370e+06

IPM summary statistics - 64 tasks

MetricSum over all tasksAverage (per task)Task CV (%)Task MinimumTask Maxium
Aggregate Floating Point Operations (Flop x 10**9) 6.858e+04 1.072e+03 0.07 1.070e+03 1.073e+03
GFlop/sec 3.993e+01 6.239e-01 0.07 6.230e-01 6.247e-01
Maximum Memory Usage (GBytes) 1.652e+01 2.581e-01 0.00 2.581e-01 2.581e-01
Time Spent in MPI Routines (sec) 5.358e+03 8.372e+01 10.52 4.360e+01 9.520e+01
Wallclock Time (sec) 1.099e+05 1.718e+03 0.01 1.717e+03 1.718e+03
Memory in units of gigabytes; time in seconds.
  

Hardware counter statistics - 64 tasks

Counter NameSum over all tasksAverage (per task)Task CV (%)Task MinimumTask Maxium
PM_FPU_1FLOP 2.791294e+13 4.361397e+11 0.17 4.349208e+11 4.373802e+11
PM_FPU_FMA 2.033260e+13 3.176969e+11 0.01 3.176117e+11 3.177840e+11
PM_INST_CMPL 1.728802e+14 2.701253e+12 0.67 2.622625e+12 2.722852e+12
PM_LD_REF_L1 5.024372e+13 7.850581e+11 0.80 7.606230e+11 7.971929e+11
PM_RUN_CYC 2.074761e+14 3.241814e+12 0.18 3.227409e+12 3.254329e+12
PM_ST_REF_L1 1.701006e+13 2.657822e+11 2.36 2.576961e+11 2.826931e+11
  

MPI time statistics - 64 tasks

CallSum over all tasksAverage (per task)Task CV (%)Task MinimumTask Maxium% of MPI% of wall
MPI_Allreduce 3.450e+03 5.390e+01 21.71 2.202e+01 7.146e+01 64.386 3.139
MPI_Sendrecv 1.733e+03 2.707e+01 39.29 1.391e+01 5.776e+01 32.334 1.576
MPI_Gather 1.712e+02 2.676e+00 24.74 1.330e+00 3.154e+00 3.196 0.156
MPI_Bcast 3.334e+00 5.209e-02 14.04 4.392e-04 6.116e-02 0.062 0.003
MPI_Reduce 1.175e+00 1.836e-02 98.20 5.852e-03 8.697e-02 0.022 0.001
MPI_Recv 2.579e-02 4.029e-04 0.00 2.579e-02 2.579e-02 0.000 0.000
MPI_Send 6.941e-03 1.085e-04 116.24 3.386e-05 6.030e-04 0.000 0.000
MPI_Comm_size 1.209e-04 1.889e-06 18.32 1.431e-06 2.623e-06 0.000 0.000
MPI_Comm_rank 7.248e-05 1.132e-06 21.55 7.153e-07 1.669e-06 0.000 0.000

Percent of total MPI Time

Pct. MPI Time by function

MPI call frequency statistics - 64 tasks

CallTasksNumber of Calls (per task)Buffer Size (Bytes)
SumAverageCV (%)MinimumMaxiumSumAvg. per taskAvg. per call
MPI_Allreduce 64 6.720e+05 1.050e+04 0.00e+00 1.050e+04 1.050e+04 0.000e+00 0.000e+00 0.000e+00
MPI_Sendrecv 64 3.584e+06 5.600e+04 0.00e+00 5.600e+04 5.600e+04 0.000e+00 0.000e+00 0.000e+00
MPI_Gather 64 2.080e+06 3.250e+04 0.00e+00 3.250e+04 3.250e+04 0.000e+00 0.000e+00 0.000e+00
MPI_Bcast 64 1.920e+02 3.000e+00 0.00e+00 3.000e+00 3.000e+00 0.000e+00 0.000e+00 0.000e+00
MPI_Reduce 64 3.270e+04 5.110e+02 0.00e+00 5.110e+02 5.110e+02 0.000e+00 0.000e+00 0.000e+00
MPI_Recv 1 5.000e+02 7.813e+00 0.00e+00 5.000e+02 5.000e+02 0.000e+00 0.000e+00 0.000e+00
MPI_Send 36 5.000e+02 7.813e+00 1.14e+02 4.000e+00 4.200e+01 0.000e+00 0.000e+00 0.000e+00
MPI_Comm_size 64 1.920e+02 3.000e+00 0.00e+00 3.000e+00 3.000e+00 0.000e+00 0.000e+00 0.000e+00
MPI_Comm_rank 64 1.920e+02 3.000e+00 0.00e+00 3.000e+00 3.000e+00 0.000e+00 0.000e+00 0.000e+00

Memory Snapshots

Memory usage snapshots are taken every 15 minutes on the nodes. This job was running on the nodes listed below when snapshots were taken. A yellow background warns that a node was running very low on small page memory; a pink background indicates a shortage that likely forced memory paging. Memory paging leads to a severe performance penalty and may cause the application to die and perhaps take the node with it.

Node NameTotal GB UsedLarge Page GB UsedLarge Page GB FreeSmall Page GB UsedSmall Page GB FreeTimeNotes
b0205 1.02 0.00 18.92 1.02 7.29 2007-07-16 18:45:00  
b0405 0.40 0.00 18.92 0.40 7.91 2007-07-16 18:45:00  
b0406 4.34 3.50 15.42 0.84 7.47 2007-07-16 18:45:00  
b0409 1.19 0.50 18.42 0.69 7.62 2007-07-16 18:45:00  
b0702 2.51 1.25 17.67 1.26 7.05 2007-07-16 18:45:00  
b0801 2.99 2.00 16.92 0.99 7.32 2007-07-16 18:45:00  
b0805 2.23 1.25 17.67 0.98 7.33 2007-07-16 18:45:00  
b0901 2.91 0.25 18.67 2.66 5.65 2007-07-16 18:45:00  
Node NameTotal GB UsedLarge Page GB UsedLarge Page GB FreeSmall Page GB UsedSmall Page GB FreeTimeNotes
b0205 3.38 2.30 16.62 1.08 7.23 2007-07-16 19:00:01  
b0405 2.77 2.30 16.62 0.47 7.84 2007-07-16 19:00:01  
b0406 6.71 5.80 13.12 0.91 7.40 2007-07-16 19:00:01  
b0409 3.56 2.80 16.12 0.76 7.55 2007-07-16 19:00:01  
b0702 4.88 3.55 15.38 1.33 6.98 2007-07-16 19:00:01  
b0801 5.36 4.30 14.62 1.06 7.25 2007-07-16 19:00:01  
b0805 4.59 3.55 15.38 1.04 7.27 2007-07-16 19:00:01  
b0901 5.78 3.05 15.88 2.73 5.58 2007-07-16 19:00:01