Memory Usage Considerations on Franklin
Each Franklin compute node has 8 GB (8192 MB) of physical memory, but, not all that memory is available to user programs. Compute Node Linux (the kernel), the Lustre file system software, and message passing library buffers all consume memory, as does loading the executable into memory. Thus the precise memory available to an application varies, approximately 7566 MB (7.38 GB) of memory can be allocated from within an MPI program using all 4 cores per node, i.e., 1891 MB (1.83 GB) per MPI task on average. Using 1 or 2 cores per node an MPI program can allocated up to about 7566 MB (7.38 GB) or 3783 MB (3.69 GB) per task.
A user can change MPI buffer sizes by setting certain MPICH environment variables. See the man page for intro_mpi for more details.
Sometimes out-of-memory jobs may leave some nodes in an unhealthy state that would affect future jobs landed on these nodes. Users are encouraged to evaluate carefully the memory requirement vvia internal checking in their codes or by some tools. Craypat could track heap usage. And IPM also tracks memory usage.