NERSCPowering Scientific Discovery Since 1974

PMEM3D Medium IPM Profile on Bassi

Step ID b0201.212766.0 Job Name pmemd64IPM
Owner hjw Account mpccc Status Completed
LL class debug Submit class debug Job type Parallel
Nodes 8 Wall secs 720 Wall hrs 0.20
MPP secs 276,480 MPP hrs 76.80 Raw Secs 46,080
Submit Jul-13-07 17:22:36 Start Jul-13-07 17:33:35 Wait 00:10:59
Completion Jul-13-07 17:52:49 systime 677 usrtime 66,772
Task Max. Memory (GB) 2.656e-01  
Nodelist b0506, b1005, b0307, b0311, b1211, b1205, b0204, b0312
*Indicates dispatch time

Summary | Hardware Counters | MPI Statistics

  

IPM summary

Executable ../exe/pmemd -O -c inpcrd -o res.jc
Number of tasks 64 Aggregate GFlop/sec 15.2381 Average GFlop/sec/task 0.2381
Average wall secs 1.055e+03 Aggregate memory (GB) 16.8076 Average memory/task (GB) 0.2626
Average MPI secs/task 1.084e+02 MPI time % 10.28 Aggregate MPI calls made 2.397e+08

IPM summary statistics - 64 tasks

MetricSum over all tasksAverage (per task)Task CV (%)Task MinimumTask Maxium
Aggregate Floating Point Operations (Flop x 10**9) 1.607e+04 2.511e+02 15.96 2.036e+02 3.064e+02
GFlop/sec 1.524e+01 2.381e-01 15.96 1.931e-01 2.905e-01
Maximum Memory Usage (GBytes) 1.681e+01 2.626e-01 0.02 2.625e-01 2.628e-01
Time Spent in MPI Routines (sec) 6.939e+03 1.084e+02 21.47 8.111e+01 1.386e+02
Wallclock Time (sec) 6.751e+04 1.055e+03 0.01 1.055e+03 1.056e+03
Memory in units of gigabytes; time in seconds.
  

Hardware counter statistics - 64 tasks

Counter NameSum over all tasksAverage (per task)Task CV (%)Task MinimumTask Maxium
PM_FPU_1FLOP 8.779579e+12 1.371809e+11 16.92 1.099774e+11 1.665629e+11
PM_FPU_FMA 3.646558e+12 5.697746e+10 14.84 4.682335e+10 6.993149e+10
PM_INST_CMPL 9.370561e+13 1.464150e+12 4.80 1.378866e+12 1.546138e+12
PM_LD_REF_L1 4.189077e+13 6.545433e+11 2.05 6.389105e+11 6.736872e+11
PM_RUN_CYC 1.270429e+14 1.985046e+12 0.19 1.976312e+12 1.994665e+12
PM_ST_REF_L1 1.455681e+13 2.274501e+11 3.79 2.172774e+11 2.424742e+11
  

MPI time statistics - 64 tasks

CallSum over all tasksAverage (per task)Task CV (%)Task MinimumTask Maxium% of MPI% of wall
MPI_Waitany 3.660e+03 5.720e+01 33.52 3.246e+01 8.298e+01 52.754 5.423
MPI_Allreduce 1.558e+03 2.435e+01 13.31 1.365e+01 2.956e+01 22.455 2.308
MPI_Allgatherv 7.907e+02 1.235e+01 0.54 1.222e+01 1.252e+01 11.395 1.171
MPI_Isend 7.106e+02 1.110e+01 51.53 4.309e+00 1.849e+01 10.240 1.053
MPI_Bcast 9.566e+01 1.495e+00 10.79 2.256e-01 1.526e+00 1.379 0.142
MPI_Irecv 8.380e+01 1.309e+00 42.61 6.374e-01 1.885e+00 1.208 0.124
MPI_Waitall 3.672e+01 5.737e-01 40.72 3.378e-01 1.677e+00 0.529 0.054
MPI_Gatherv 1.842e+00 2.879e-02 66.27 1.357e-02 8.464e-02 0.027 0.003
MPI_Gather 9.005e-01 1.407e-02 46.15 6.375e-03 4.909e-02 0.013 0.001
MPI_Send 4.470e-04 6.985e-06 18.47 4.768e-06 9.298e-06 0.000 0.000
MPI_Recv 9.179e-05 1.434e-06 0.00 9.179e-05 9.179e-05 0.000 0.000
MPI_Comm_rank 6.294e-05 9.835e-07 16.45 7.153e-07 1.431e-06 0.000 0.000
MPI_Comm_size 3.171e-05 4.955e-07 19.75 2.384e-07 7.153e-07 0.000 0.000
INTERNAL_ERROR 0.000e+00 0.000e+00 0.00 0.000e+00 0.000e+00 0.000 0.000

Percent of total MPI Time

Pct. MPI Time by function

MPI call frequency statistics - 64 tasks

CallTasksNumber of Calls (per task)Buffer Size (Bytes)
SumAverageCV (%)MinimumMaxiumSumAvg. per taskAvg. per call
MPI_Waitany 64 7.700e+07 1.203e+06 4.42e+01 5.605e+05 1.702e+06 0.000e+00 0.000e+00 0.000e+00
MPI_Allreduce 64 1.922e+06 3.003e+04 0.00e+00 3.003e+04 3.003e+04 0.000e+00 0.000e+00 0.000e+00
MPI_Allgatherv 64 1.805e+05 2.821e+03 0.00e+00 2.821e+03 2.821e+03 0.000e+00 0.000e+00 0.000e+00
MPI_Isend 64 7.700e+07 1.203e+06 4.45e+01 5.600e+05 1.705e+06 0.000e+00 0.000e+00 0.000e+00
MPI_Bcast 64 1.427e+04 2.230e+02 0.00e+00 2.230e+02 2.230e+02 0.000e+00 0.000e+00 0.000e+00
MPI_Irecv 64 7.700e+07 1.203e+06 4.42e+01 5.605e+05 1.702e+06 0.000e+00 0.000e+00 0.000e+00
MPI_Waitall 64 3.000e+06 4.688e+04 3.20e+01 3.000e+04 6.000e+04 0.000e+00 0.000e+00 0.000e+00
MPI_Gatherv 64 3.840e+03 6.000e+01 0.00e+00 6.000e+01 6.000e+01 0.000e+00 0.000e+00 0.000e+00
MPI_Gather 64 1.165e+04 1.820e+02 0.00e+00 1.820e+02 1.820e+02 0.000e+00 0.000e+00 0.000e+00
MPI_Send 63 6.300e+01 9.844e-01 1.27e+01 1.000e+00 1.000e+00 0.000e+00 0.000e+00 0.000e+00
MPI_Recv 1 6.300e+01 9.844e-01 0.00e+00 6.300e+01 6.300e+01 0.000e+00 0.000e+00 0.000e+00
MPI_Comm_rank 64 6.400e+01 1.000e+00 0.00e+00 1.000e+00 1.000e+00 0.000e+00 0.000e+00 0.000e+00
MPI_Comm_size 64 6.400e+01 1.000e+00 0.00e+00 1.000e+00 1.000e+00 0.000e+00 0.000e+00 0.000e+00
INTERNAL_ERROR 40 3.572e+06 5.582e+04 7.90e+01 8.690e+04 1.006e+05 0.000e+00 0.000e+00 0.000e+00

Memory Snapshots

Memory usage snapshots are taken every 15 minutes on the nodes. This job was running on the nodes listed below when snapshots were taken. A yellow background warns that a node was running very low on small page memory; a pink background indicates a shortage that likely forced memory paging. Memory paging leads to a severe performance penalty and may cause the application to die and perhaps take the node with it.

Node NameTotal GB UsedLarge Page GB UsedLarge Page GB FreeSmall Page GB UsedSmall Page GB FreeTimeNotes
b0204 3.85 2.80 16.12 1.05 7.26 2007-07-13 17:45:01  
b0307 6.05 2.30 16.62 3.75 4.56 2007-07-13 17:45:01  
b0311 5.16 2.55 16.38 2.61 5.70 2007-07-13 17:45:01  
b0312 5.66 2.80 16.12 2.86 5.45 2007-07-13 17:45:01  
b0506 3.66 2.80 16.12 0.86 7.45 2007-07-13 17:45:01  
b1005 3.11 2.30 16.62 0.81 7.50 2007-07-13 17:45:01  
b1205 4.92 3.80 15.12 1.12 7.19 2007-07-13 17:45:01  
b1211 3.95 2.30 16.62 1.65 6.66 2007-07-13 17:45:01