Unable to allocate hugepages in running jobs
January 14, 2013 by Helen He (0 Comments)
User job sometimes get an error message similar to the following, usually at the start of a batch job, causing the job to abort:
MPICH2 ERROR [Rank 7436] [job id 14638087] [Sat Jan 12 04:56:54 2013] [c11-2c1s3n1] [nid04487] - MPIU_nem_gni_get_hugepages(): Unable to mmap 4194304 bytes for file /var/lib/hugetlbfs/global/pagesize-2097152/hugepagefile.MPICH.0.5841.kvs_14638087, err Cannot allocate memory
This is caused by available huge page memory being not sufficient on one or more of the allocated compute nodes. The above error happens more often with jobs using the "-ss" option for the aprun command. It is confirmed that the available hugepages are not even among the 4 NUMA nodes on a compute node.
The first workaround is to resubmit your batch job so that it launches on a different set of compute nodes. We monitor failed jobs and manually reboot the problem nodes. The second workaroud is to not use the "-ss" option in your batch script, it sometimes has negative performance impact especailly for hybrid MPI/OpenMP applications.
Several bugs regarding hugepages have been opened with Cray.