NERSCPowering Scientific Discovery Since 1974

Unable to allocate hugepages in running jobs

January 14, 2013 by Helen He (0 Comments)

 

Symptom

User job sometimes get an error message similar to the following, usually at the start of a batch job, causing the job to abort:

MPICH2 ERROR [Rank 7436] [job id 14638087] [Sat Jan 12 04:56:54 2013] [c11-2c1s3n1] [nid04487] - MPIU_nem_gni_get_hugepages(): Unable to mmap 4194304 bytes for file /var/lib/hugetlbfs/global/pagesize-2097152/hugepagefile.MPICH.0.5841.kvs_14638087, err Cannot allocate memory

This is caused by available huge page memory being not sufficient on one or more of the allocated compute nodes.  The above error happens more often with jobs using the "-ss" option for the aprun command. It is confirmed that the available hugepages are not even among the 4 NUMA nodes on a compute node.

Workaround

The first workaround is to resubmit your batch job so that it launches on a different set of compute nodes.  We monitor failed jobs and manually reboot the problem nodes. The second workaroud is to not use the "-ss" option in your batch script, it sometimes has negative performance impact especailly for hybrid MPI/OpenMP applications.

Status

Several bugs regarding hugepages have been opened with Cray.

 


Post your comment

You cannot post comments until you have logged in. Login Here.

Comments

No one has commented on this page yet.

RSS feed for comments on this page | RSS feed for all comments