mpimemu
Description
The code mpimemu is a simple tool that helps approximate MPI library memory usage as a function of scale. It takes samples of /proc/meminfo (node level) and /proc/self/status (process level) and outputs the min, max and avg values for a specified period of time. More information can be found in the README and README.QUICKSTART files.
Download
How to Build
./configure CC=mpicc (or your appropriate MPI C compiler)
make
How to Run
Mpimemu can be run directly, but to peform a scaled study it is necessary to use the script src/mpimemu-run. But before using mpimemu-run it is necessary to source environment variables contatined in util/env-setup-bash. If you don't like bash, modify it to your appropriate shell commands.
In order to run a scaled study, it is necessary to set MPIMEMU_PPN to the number of MPI ranks per nodes you would like to run. MPIMEMU_MAX_PES is the upper limit of the scaling run. So for example, if MPIMEMU_PPN=16 and MPIMEMU_MAX_PE=1024 and MPIMEMU_NUMPE_FUN="2**X * MPIMEMU_PPN" then mpimemu-run will collect data for N=16,32,64,128,256,512 and 1024 PEs. Note that MPIMEMU_RUN_CMD needs to include the necessary flags for your architecture to run MPIMEMU_PPN MPI ranks per node.
Edit the file util/env-setup-bash to specify a scaled study from 1 node to 2048 nodes. For example, on NERSC's Hopper with 24 MPI ranks per node (one rank per core) the following environment variables might be used:
MPIMEMU_PPN=24
MPIMEMU_MAX_PES=49152
MPIMEMU_RUN_CMD="aprun -cc cpu -n nnn aaa"
MPIMEMU_NUMPE_FUN="2**X * $MPIMEMU_PPN"
Source the appropriate environment variables and run the scaling script:
$ . util/env-setup-bash
$ ./src/mpimemu-run
Reporting
Mpimemu-run places results a directory at the location specified by MPIMEMU_DATA_DIR_PREFIX. The results can be parsed into a CSV list using the script src/mpimemu-mkstats.
For example, mpimemu-run creates results in the directory mpimemu-username-01112013,
$ ./src/mpimemu-mkstats -i mpimemu-username-01112013
creates two files, 01112013-node-mem-usage-xxx.csv and 01112013-proc-mem-usage-xxx.csv.
For Trinity and NERSC-8, the value ((Post-Init MemUsed)-(Pre-Init MemUsed)) from the node report is to be reported.
Authorship
Samuel Gutierrez - Los Alamos National Laboratory
Nathan T. Hjelm - Los Alamos National Laboratory
