NERSCPowering Scientific Discovery Since 1974



The code mpimemu is a simple tool that helps approximate MPI library memory usage as a function of scale.  It takes samples of /proc/meminfo (node level) and /proc/self/status (process level) and outputs the min, max and avg values for a specified period of time. More information can be found in the README and README.QUICKSTART files.


mpimemu tar file (updated July 5, README only)

How to Build

./configure CC=mpicc (or your appropriate MPI C compiler)

How to Run 

Mpimemu can be run directly, but to peform a scaled study it is necessary to use the script src/mpimemu-run. But before using mpimemu-run it is necessary to source environment variables contatined in util/env-setup-bash. If you don't like bash, modify it to your appropriate shell commands.

In order to run a scaled study, it is necessary to set MPIMEMU_PPN to the number of MPI ranks per node you would like to run. As a guide in choosing the number of MPI ranks per node, the desired choice would be to use the maximum number of ranks per node that you typically use to achieve good performance on applications as part of a performance projection study. Once the number of ranks per node is set, the overall concurrency
will be the number of ranks per node multiplied by the total number of nodes.

MPIMEMU_MAX_PES is the upper limit of the scaling run.
So for example, if MPIMEMU_PPN=16 and MPIMEMU_MAX_PE=1024 and MPIMEMU_NUMPE_FUN="2**X * MPIMEMU_PPN" then mpimemu-run will
collect data for N=16,32,64,128,256,512 and 1024 PEs. Note that MPIMEMU_RUN_CMD needs to include the necessary flags for your
architecture to run MPIMEMU_PPN MPI ranks per node.


Edit the file util/env-setup-bash to specify a scaled study from 1 node to 2048 nodes. For example, on NERSC's Hopper with 24 MPI ranks per node (one rank per core) the following environment variables might be used:

MPIMEMU_RUN_CMD="aprun -cc cpu -n nnn aaa"

Source the appropriate environment variables and run the scaling script:
$ . util/env-setup-bash
$ ./src/mpimemu-run 


Mpimemu-run places results a directory at the location specified by MPIMEMU_DATA_DIR_PREFIX. The results can be parsed into a CSV list using the script src/mpimemu-mkstats.

For example, mpimemu-run creates results in the directory mpimemu-username-01112013,

$ ./src/mpimemu-mkstats -i mpimemu-username-01112013

creates two files, 01112013-node-mem-usage-xxx.csv and 01112013-proc-mem-usage-xxx.csv.

For Trinity and NERSC-8, the value ((Post-Init MemUsed)-(Pre-Init MemUsed)) from the node report is to be reported.  Use the script "mpimemu-mkstats" in the 'src' directory to convert the data to CSV format and copy the CSV-format data into the results spreadsheet.


Samuel Gutierrez - Los Alamos National Laboratory
Nathan T. Hjelm - Los Alamos National Laboratory