NERSCPowering Scientific Discovery Since 1974

MADBench Medium IPM Profile on Bassi

 

Step ID MadBenchMedium Job Name MADmediumIPM
Owner hjw Account mpccc Status Completed
LL class reg_1 Submit class regular Job type Parallel
Nodes 8 Wall secs 1,085 Wall hrs 0.30
MPP secs 416,640 MPP hrs 115.73 Raw Secs 69,440
Submit Jul-12-07 09:01:14 Start Jul-12-07 22:27:43 Wait 13:26:29
Completion Jul-12-07 22:51:51 systime 2,544 usrtime 74,887
Task Max. Memory (GB) 5.041e-01  
Nodelist b0408, b0312, b0912, b0407, b0802, b0412, b0901, b0204
*Indicates dispatch time

Summary | Hardware Counters | MPI Statistics

  

IPM summary

Executable ../madbench.x 18000 24 1 128 4 4
Number of tasks 64 Aggregate GFlop/sec 216.4009 Average GFlop/sec/task 3.3813
Average wall secs 1.334e+03 Aggregate memory (GB) 32.2581 Average memory/task (GB) 0.5040
Average MPI secs/task 3.427e+02 MPI time % 25.69 Aggregate MPI calls made 6.831e+06

IPM summary statistics - 64 tasks

MetricSum over all tasksAverage (per task)Task CV (%)Task MinimumTask Maxium
Aggregate Floating Point Operations (Flop x 10**9) 2.886e+05 4.510e+03 3.75 4.212e+03 4.735e+03
GFlop/sec 2.164e+02 3.381e+00 3.75 3.158e+00 3.549e+00
Maximum Memory Usage (GBytes) 3.226e+01 5.040e-01 0.00 5.040e-01 5.041e-01
Time Spent in MPI Routines (sec) 2.193e+04 3.427e+02 12.51 2.560e+02 4.315e+02
Wallclock Time (sec) 8.536e+04 1.334e+03 0.02 1.333e+03 1.334e+03
Memory in units of gigabytes; time in seconds.
  

Hardware counter statistics - 64 tasks

Counter NameSum over all tasksAverage (per task)Task CV (%)Task MinimumTask Maxium
PM_FPU_1FLOP 3.920615e+11 6.125960e+09 3.73 5.714366e+09 6.438386e+09
PM_FPU_FMA 1.441144e+14 2.251787e+12 3.75 2.103315e+12 2.364198e+12
PM_INST_CMPL 3.090704e+14 4.829224e+12 3.70 4.457525e+12 5.145855e+12
PM_LD_REF_L1 9.422183e+13 1.472216e+12 3.82 1.361713e+12 1.566510e+12
PM_RUN_CYC 1.361351e+14 2.127111e+12 6.04 1.923897e+12 2.352402e+12
PM_ST_REF_L1 1.175324e+13 1.836444e+11 14.67 1.301904e+11 2.332761e+11
  

MPI time statistics - 64 tasks

CallSum over all tasksAverage (per task)Task CV (%)Task MinimumTask Maxium% of MPI% of wall
MPI_Recv 1.331e+04 2.080e+02 24.83 9.639e+01 2.878e+02 60.714 15.598
MPI_Barrier 5.672e+03 8.863e+01 32.30 5.086e+01 1.415e+02 25.865 6.645
MPI_Send 2.283e+03 3.568e+01 48.02 1.469e+01 8.049e+01 10.412 2.675
MPI_Reduce 4.379e+02 6.842e+00 34.75 3.459e+00 1.487e+01 1.997 0.513
MPI_Bcast 2.069e+02 3.232e+00 28.31 1.363e+00 4.961e+00 0.943 0.242
MPI_Testall 5.468e+00 8.543e-02 38.78 3.696e-04 1.185e-01 0.025 0.006
MPI_Allreduce 5.421e+00 8.470e-02 58.25 7.651e-03 2.311e-01 0.025 0.006
MPI_Isend 2.524e+00 3.944e-02 44.75 3.905e-04 7.011e-02 0.012 0.003
MPI_Sendrecv 1.544e+00 2.412e-02 0.97 2.379e-02 2.453e-02 0.007 0.002
MPI_Comm_rank 2.196e-03 3.432e-05 10.84 2.646e-05 4.840e-05 0.000 0.000
MPI_Comm_size 8.297e-05 1.296e-06 26.39 7.153e-07 2.384e-06 0.000 0.000

Percent of total MPI Time

Pct. MPI Time by function

MPI call frequency statistics - 64 tasks

CallTasksNumber of Calls (per task)Buffer Size (Bytes)
SumAverageCV (%)MinimumMaxiumSumAvg. per taskAvg. per call
MPI_Recv 64 1.694e+06 2.647e+04 2.66e+00 2.457e+04 2.684e+04 0.000e+00 0.000e+00 0.000e+00
MPI_Barrier 64 9.472e+03 1.480e+02 0.00e+00 1.480e+02 1.480e+02 0.000e+00 0.000e+00 0.000e+00
MPI_Send 64 1.583e+06 2.474e+04 2.64e-01 2.456e+04 2.486e+04 0.000e+00 0.000e+00 0.000e+00
MPI_Reduce 64 7.751e+04 1.211e+03 1.09e+00 1.183e+03 1.235e+03 0.000e+00 0.000e+00 0.000e+00
MPI_Bcast 64 3.790e+04 5.921e+02 1.32e+00 5.790e+02 6.000e+02 0.000e+00 0.000e+00 0.000e+00
MPI_Testall 64 3.303e+06 5.161e+04 3.88e+01 1.240e+02 6.415e+04 0.000e+00 0.000e+00 0.000e+00
MPI_Allreduce 64 1.984e+03 3.100e+01 2.58e+01 2.800e+01 5.200e+01 0.000e+00 0.000e+00 0.000e+00
MPI_Isend 64 1.112e+05 1.737e+03 3.67e+01 6.400e+01 2.037e+03 0.000e+00 0.000e+00 0.000e+00
MPI_Sendrecv 64 9.984e+03 1.560e+02 0.00e+00 1.560e+02 1.560e+02 0.000e+00 0.000e+00 0.000e+00
MPI_Comm_rank 64 1.920e+03 3.000e+01 0.00e+00 3.000e+01 3.000e+01 0.000e+00 0.000e+00 0.000e+00
MPI_Comm_size 64 1.280e+02 2.000e+00 0.00e+00 2.000e+00 2.000e+00 0.000e+00 0.000e+00 0.000e+00

Memory Snapshots

Memory usage snapshots are taken every 15 minutes on the nodes. This job was running on the nodes listed below when snapshots were taken. A yellow background warns that a node was running very low on small page memory; a pink background indicates a shortage that likely forced memory paging. Memory paging leads to a severe performance penalty and may cause the application to die and perhaps take the node with it.

Node NameTotal GB UsedLarge Page GB UsedLarge Page GB FreeSmall Page GB UsedSmall Page GB FreeTimeNotes
b0204 -2.59 0.50 18.42 -3.09 7.40 2007-07-12 22:30:01  
b0312 -0.78 0.50 18.42 -1.28 5.59 2007-07-12 22:30:01  
b0407 0.76 2.25 16.67 -1.49 5.80 2007-07-12 22:30:01  
b0408 -2.68 0.50 18.42 -3.18 7.49 2007-07-12 22:30:01  
b0412 -2.86 0.00 18.92 -2.86 7.17 2007-07-12 22:30:01  
b0802 -0.01 0.00 18.92 -0.01 4.32 2007-07-12 22:30:01  
b0901 -1.16 0.25 18.67 -1.41 5.72 2007-07-12 22:30:01  
b0912 -1.08 2.00 16.92 -3.08 7.39 2007-07-12 22:30:01  
Node NameTotal GB UsedLarge Page GB UsedLarge Page GB FreeSmall Page GB UsedSmall Page GB FreeTimeNotes
b0204 1.76 4.80 14.12 -3.04 7.35 2007-07-12 22:45:01  
b0312 3.58 4.80 14.12 -1.22 5.53 2007-07-12 22:45:01  
b0407 5.13 6.55 12.38 -1.42 5.73 2007-07-12 22:45:01  
b0408 2.16 5.30 13.62 -3.14 7.45 2007-07-12 22:45:01  
b0412 1.49 4.30 14.62 -2.81 7.12 2007-07-12 22:45:01  
b0802 4.38 4.30 14.62 0.08 4.23 2007-07-12 22:45:01  
b0901 3.20 4.55 14.38 -1.35 5.66 2007-07-12 22:45:01  
b0912 3.26 6.30 12.62 -3.04 7.35 2007-07-12 22:45:01