NERSCPowering Scientific Discovery Since 1974

Paratec Large IPM Profile on Bassi

Step ID b0201.211957.0 Job Name ParatecLargeIPM
Owner hjw Account mpccc Status Completed
LL class reg_32 Submit class regular Job type Parallel
Nodes 32 Wall secs 1,173 Wall hrs 0.33
MPP secs 1,801,728 MPP hrs 500.48 Raw Secs 300,288
Submit Jul-12-07 08:29:13 Start Jul-15-07 03:10:43 Wait 66:41:30
Completion Jul-15-07 03:43:42 systime 18,260 usrtime 397,380
Task Max. Memory (GB) 2.010e+00  
Nodelist b0705, b0711, b1108, b0411, b0910, b0702, b1201, b0308, b0410, b0912, b0305, b0208, b0507, b0505, b0205, b1210, b0712, b0801, b0406, b0807, b0911, b0210, b1203, b1005, b0412, b0809, b0407, b0304, b1109, b1107, b1211, b0310
*Indicates dispatch time

Summary | Hardware Counters | MPI Statistics

  

IPM summary

Executable ../bin/sp3nersc/paratec.mpi
Number of tasks 256 Aggregate GFlop/sec 644.4357 Average GFlop/sec/task 2.5173
Average wall secs 1.106e+03 Aggregate memory (GB) 233.7110 Average memory/task (GB) 0.9129
Average MPI secs/task 5.588e+02 MPI time % 50.52 Aggregate MPI calls made 1.769e+10

IPM summary statistics - 256 tasks

MetricSum over all tasksAverage (per task)Task CV (%)Task MinimumTask Maxium
Aggregate Floating Point Operations (Flop x 10**9) 7.128e+05 2.784e+03 0.56 2.745e+03 2.802e+03
GFlop/sec 6.444e+02 2.517e+00 0.56 2.481e+00 2.533e+00
Maximum Memory Usage (GBytes) 2.337e+02 9.129e-01 45.37 7.566e-01 2.008e+00
Time Spent in MPI Routines (sec) 1.430e+05 5.588e+02 0.92 5.463e+02 5.720e+02
Wallclock Time (sec) 2.832e+05 1.106e+03 0.02 1.106e+03 1.106e+03
Memory in units of gigabytes; time in seconds.
  

Hardware counter statistics - 256 tasks

Counter NameSum over all tasksAverage (per task)Task CV (%)Task MinimumTask Maxium
PM_FPU_1FLOP 5.952263e+12 2.325103e+10 0.96 2.290482e+10 2.408109e+10
PM_FPU_FMA 3.534382e+14 1.380618e+12 0.56 1.360846e+12 1.389237e+12
PM_INST_CMPL 7.761684e+14 3.031908e+12 1.61 2.983507e+12 3.179853e+12
PM_LD_REF_L1 2.232065e+14 8.719004e+11 1.92 8.540956e+11 9.234264e+11
PM_RUN_CYC 5.084702e+14 1.986212e+12 1.62 1.957901e+12 2.082010e+12
PM_ST_REF_L1 5.214635e+13 2.036967e+11 5.99 1.920569e+11 2.427347e+11
  

MPI time statistics - 256 tasks

CallSum over all tasksAverage (per task)Task CV (%)Task MinimumTask Maxium% of MPI% of wall
MPI_Wait 9.708e+04 3.792e+02 1.04 3.672e+02 3.913e+02 67.866 34.284
MPI_Allreduce 1.819e+04 7.106e+01 5.37 5.931e+01 8.004e+01 12.717 6.424
MPI_Isend 1.642e+04 6.416e+01 2.56 6.026e+01 7.008e+01 11.481 5.800
MPI_Irecv 4.526e+03 1.768e+01 2.91 1.647e+01 1.929e+01 3.164 1.598
MPI_Bcast 4.088e+03 1.597e+01 4.74 1.165e+01 1.704e+01 2.858 1.444
MPI_Recv 1.631e+03 6.370e+00 18.48 3.872e+00 1.014e+01 1.140 0.576
MPI_Waitall 3.939e+02 1.539e+00 77.62 1.573e-01 6.209e+00 0.275 0.139
MPI_Gather 3.474e+02 1.357e+00 0.58 1.334e+00 1.368e+00 0.243 0.123
MPI_Reduce 3.079e+02 1.203e+00 49.99 1.670e-01 2.691e+00 0.215 0.109
MPI_Barrier 2.982e+01 1.165e-01 9.04 5.787e-02 1.364e-01 0.021 0.011
MPI_Send 2.271e+01 8.869e-02 102.90 3.908e-02 7.987e-01 0.016 0.008
MPI_Testall 5.469e+00 2.136e-02 27.43 1.784e-03 3.362e-02 0.004 0.002
MPI_Rsend 3.119e-01 1.218e-03 160.85 2.222e-04 2.281e-02 0.000 0.000
MPI_Comm_rank 9.869e-03 3.855e-05 80.99 2.194e-05 3.226e-04 0.000 0.000
MPI_Comm_size 4.137e-04 1.616e-06 19.52 9.537e-07 2.623e-06 0.000 0.000

Percent of total MPI Time

Pct. MPI Time by function

MPI call frequency statistics - 256 tasks

CallTasksNumber of Calls (per task)Buffer Size (Bytes)
SumAverageCV (%)MinimumMaxiumSumAvg. per taskAvg. per call
MPI_Wait 256 2.147e+09 3.431e+07 0.00e+00 3.431e+07 3.431e+07 0.000e+00 0.000e+00 0.000e+00
MPI_Allreduce 256 2.836e+05 1.108e+03 0.00e+00 1.108e+03 1.108e+03 0.000e+00 0.000e+00 0.000e+00
MPI_Isend 256 2.147e+09 1.732e+07 2.13e-01 1.722e+07 1.739e+07 0.000e+00 0.000e+00 0.000e+00
MPI_Irecv 256 2.147e+09 1.715e+07 4.59e-04 1.715e+07 1.715e+07 0.000e+00 0.000e+00 0.000e+00
MPI_Bcast 256 8.477e+06 3.311e+04 1.17e+00 3.223e+04 3.398e+04 0.000e+00 0.000e+00 0.000e+00
MPI_Recv 256 4.636e+07 1.811e+05 2.05e+01 7.607e+04 2.586e+05 0.000e+00 0.000e+00 0.000e+00
MPI_Waitall 256 2.017e+07 7.879e+04 6.22e-01 7.765e+04 7.989e+04 0.000e+00 0.000e+00 0.000e+00
MPI_Gather 256 7.864e+05 3.072e+03 0.00e+00 3.072e+03 3.072e+03 0.000e+00 0.000e+00 0.000e+00
MPI_Reduce 256 2.616e+05 1.022e+03 1.46e+01 7.750e+02 1.239e+03 0.000e+00 0.000e+00 0.000e+00
MPI_Barrier 256 9.216e+03 3.600e+01 0.00e+00 3.600e+01 3.600e+01 0.000e+00 0.000e+00 0.000e+00
MPI_Send 256 2.921e+06 1.141e+04 9.32e+00 6.260e+03 1.268e+04 0.000e+00 0.000e+00 0.000e+00
MPI_Testall 256 1.884e+06 7.360e+03 2.63e+01 8.810e+02 1.235e+04 0.000e+00 0.000e+00 0.000e+00
MPI_Rsend 192 8.367e+04 3.268e+02 1.01e+02 8.200e+01 1.238e+03 0.000e+00 0.000e+00 0.000e+00
MPI_Comm_rank 256 4.608e+03 1.800e+01 0.00e+00 1.800e+01 1.800e+01 0.000e+00 0.000e+00 0.000e+00
MPI_Comm_size 256 7.680e+02 3.000e+00 0.00e+00 3.000e+00 3.000e+00 0.000e+00 0.000e+00 0.000e+00

Memory Snapshots

Memory usage snapshots are taken every 15 minutes on the nodes. This job was running on the nodes listed below when snapshots were taken. A yellow background warns that a node was running very low on small page memory; a pink background indicates a shortage that likely forced memory paging. Memory paging leads to a severe performance penalty and may cause the application to die and perhaps take the node with it.

Node NameTotal GB UsedLarge Page GB UsedLarge Page GB FreeSmall Page GB UsedSmall Page GB FreeTimeNotes
b0205 8.60 7.55 11.38 1.05 7.26 2007-07-15 03:15:01  
b0208 10.40 7.80 11.25 2.60 5.71 2007-07-15 03:15:01  
b0210 8.56 7.80 11.12 0.76 7.55 2007-07-15 03:15:01  
b0304 11.24 8.80 10.12 2.44 5.87 2007-07-15 03:15:01  
b0305 9.66 8.80 10.12 0.86 7.45 2007-07-15 03:15:01  
b0308 8.02 7.55 11.38 0.47 7.84 2007-07-15 03:15:01  
b0310 8.31 7.55 11.38 0.76 7.55 2007-07-15 03:15:01  
b0406 11.92 11.05 7.88 0.87 7.44 2007-07-15 03:15:01  
b0407 12.41 9.80 9.12 2.61 5.70 2007-07-15 03:15:01  
b0410 15.81 12.05 6.88 3.76 4.55 2007-07-15 03:15:01  
b0411 9.06 7.55 11.38 1.51 6.80 2007-07-15 03:15:01  
b0412 8.76 7.55 11.38 1.21 7.10 2007-07-15 03:15:01  
b0505 9.43 8.55 10.38 0.88 7.43 2007-07-15 03:15:01  
b0507 8.65 7.55 11.38 1.10 7.21 2007-07-15 03:15:01  
b0702 10.10 8.80 10.12 1.30 7.01 2007-07-15 03:15:01  
b0705 12.61 9.80 9.12 2.81 5.50 2007-07-15 03:15:01  
b0711 8.45 7.55 11.38 0.90 7.41 2007-07-15 03:15:01  
b0712 10.72 9.55 9.38 1.17 7.14 2007-07-15 03:15:01  
b0801 10.57 9.55 9.38 1.02 7.29 2007-07-15 03:15:01  
b0807 8.13 7.55 11.38 0.58 7.73 2007-07-15 03:15:01  
b0809 13.17 10.05 8.88 3.12 5.19 2007-07-15 03:15:01  
b0910 10.60 10.05 8.88 0.55 7.76 2007-07-15 03:15:01  
b0911 11.42 10.30 8.62 1.12 7.19 2007-07-15 03:15:01  
b0912 10.53 9.55 9.38 0.98 7.33 2007-07-15 03:15:01  
b1005 8.61 7.80 11.12 0.81 7.50 2007-07-15 03:15:01  
b1107 8.64 7.80 11.12 0.84 7.47 2007-07-15 03:15:01  
b1108 8.42 7.55 11.38 0.87 7.44 2007-07-15 03:15:01  
b1109 11.65 7.80 11.12 3.85 4.46 2007-07-15 03:15:01  
b1201 8.36 7.80 11.12 0.56 7.75 2007-07-15 03:15:01  
b1203 10.45 9.80 9.12 0.65 7.66 2007-07-15 03:15:01  
b1210 12.74 9.55 9.38 3.19 5.12 2007-07-15 03:15:01  
b1211 9.17 7.55 11.38 1.62 6.69 2007-07-15 03:15:01  
Node NameTotal GB UsedLarge Page GB UsedLarge Page GB FreeSmall Page GB UsedSmall Page GB FreeTimeNotes
b0205 8.60 7.55 11.38 1.05 7.26 2007-07-15 03:30:01  
b0208 10.40 7.80 11.25 2.60 5.71 2007-07-15 03:30:01  
b0210 8.56 7.80 11.12 0.76 7.55 2007-07-15 03:30:01  
b0304 11.24 8.80 10.12 2.44 5.87 2007-07-15 03:30:01  
b0305 9.66 8.80 10.12 0.86 7.45 2007-07-15 03:30:01  
b0308 8.02 7.55 11.38 0.47 7.84 2007-07-15 03:30:01  
b0310 8.31 7.55 11.38 0.76 7.55 2007-07-15 03:30:01  
b0406 11.92 11.05 7.88 0.87 7.44 2007-07-15 03:30:01  
b0407 12.41 9.80 9.12 2.61 5.70 2007-07-15 03:30:01  
b0410 15.82 12.05 6.88 3.77 4.54 2007-07-15 03:30:01  
b0411 9.06 7.55 11.38 1.51 6.80 2007-07-15 03:30:01  
b0412 8.76 7.55 11.38 1.21 7.10 2007-07-15 03:30:01  
b0505 9.43 8.55 10.38 0.88 7.43 2007-07-15 03:30:01  
b0507 8.65 7.55 11.38 1.10 7.21 2007-07-15 03:30:01  
b0702 10.10 8.80 10.12 1.30 7.01 2007-07-15 03:30:01  
b0705 13.36 10.55 8.38 2.81 5.50 2007-07-15 03:30:01  
b0711 8.46 7.55 11.38 0.91 7.40 2007-07-15 03:30:01  
b0712 10.72 9.55 9.38 1.17 7.14 2007-07-15 03:30:01  
b0801 10.58 9.55 9.38 1.03 7.28 2007-07-15 03:30:01  
b0807 8.13 7.55 11.38 0.58 7.73 2007-07-15 03:30:01  
b0809 13.17 10.05 8.88 3.12 5.19 2007-07-15 03:30:01  
b0910 10.60 10.05 8.88 0.55 7.76 2007-07-15 03:30:01  
b0911 11.42 10.30 8.62 1.12 7.19 2007-07-15 03:30:01  
b0912 10.53 9.55 9.38 0.98 7.33 2007-07-15 03:30:01  
b1005 8.61 7.80 11.12 0.81 7.50 2007-07-15 03:30:01  
b1107 8.64 7.80 11.12 0.84 7.47 2007-07-15 03:30:01  
b1108 8.42 7.55 11.38 0.87 7.44 2007-07-15 03:30:01  
b1109 11.65 7.80 11.12 3.85 4.46 2007-07-15 03:30:01  
b1201 8.36 7.80 11.12 0.56 7.75 2007-07-15 03:30:01  
b1203 10.45 9.80 9.12 0.65 7.66 2007-07-15 03:30:01  
b1210 12.74 9.55 9.38 3.19 5.12 2007-07-15 03:30:01  
b1211 9.17 7.55 11.38 1.62 6.69 2007-07-15 03:30:01