blog

Unable to allocate hugepages in running jobs

Mon, 14 Jan 2013 13:48:29 -0800

Symptom

User job sometimes get an error message similar to the following, usually at the start of a batch job, causing the job to abort:

MPICH2 ERROR [Rank 7436] [job id 14638087] [Sat Jan 12 04:56:54 2013] [c11-2c1s3n1] [nid04487] - MPIU_nem_gni_get_hugepages(): Unable to mmap 4194304 bytes for file /var/lib/hugetlbfs/global/pagesize-2097152/hugepagefile.MPICH.0.5841.kvs_14638087, err Cannot allocate memory

This is caused by available huge page memory being not sufficient on one or more of the allocated compute nodes. The above error happens more often with jobs using the "-ss" option for the aprun command. It is confirmed that the available hugepages are not even among the 4 NUMA nodes on a compute node.

Workaround

The first workaround is to resubmit your batch job so that it launches on a different set of compute nodes. We monitor failed jobs and manually reboot the problem nodes. The second workaroud is to not use the "-ss" option in your batch script, it sometimes has negative performance impact especailly for hybrid MPI/OpenMP applications.

Status

Several bugs regarding hugepages have been opened with Cray.

Resolved -- Error Message: "ModuleCmd_Switch.c(172):ERROR:152: Module 'PrgEnv-‐cray' is currently not loaded”

Thu, 28 Jun 2012 10:19:30 -0700

Symptom:

User batch jobs with "#PBS -V" in the script are seeing an error message: "ModuleCmd_Switch.c(172):ERROR:152: Module 'PrgEnv-‐cray' is currently not loaded”. This is caused by overwriting the default Cray Programming Environment from Cray to PGI. This message can also be seen with "script" on the login nodes, and "nodestat" on the MOM nodes. User jobs without "#PBS -V" do not trigger this error message. Solution: ignore as harmless.

Status:

This issue has been resolved by Cray modifying the PrgEnv module load sequences.

pgi/12.4.0 has link error with OpenMP when perftools/5.3.x is loaded

Thu, 14 Jun 2012 15:04:46 -0700

Symptom:

Using the default pgi/12.4.0 compiler, when perftools/5.3.x is loaded, OpenMP codes have linking errors similar to the following:

/global/homes/y/yunhe/.craypat/a.out/13554/pgccPJtd7jk8ihft.o: In function `main':

/global/u1/y/yunhe/shared/./xthi.c:46: undefined reference to `_mp_trace_parallel_enter'

/global/u1/y/yunhe/shared/./xthi.c:52: undefined reference to `_mp_trace_parallel_begin'

/global/u1/y/yunhe/shared/./xthi.c:60: undefined reference to `_mp_trace_parallel_end'

/global/u1/y/yunhe/shared/./xthi.c:60: undefined reference to `_mp_trace_parallel_exit'

/usr/bin/ld: link errors found, deleting executable `a.out'

Workaround

Use the older pgi/12.2.0 instead for openmp codes when perftools/5.3.x is loaded.

% module swap pgi pgi/12.2.0

There is another workaround which will enable you to use pgi 12.4.0 and later versions.

Create a file called stubs.c which contains this:

void _mp_trace_loop_chunk_begin () {}
void _mp_trace_loop_chunk_end () {}
void _mp_trace_loop_enter () {}
void _mp_trace_loop_exit () {}
void _mp_trace_master_enter () {}
void _mp_trace_master_exit () {}
void _mp_trace_parallel_begin () {}
void _mp_trace_parallel_end () {}
void _mp_trace_parallel_enter () {}
void _mp_trace_parallel_exit () {}
void _mp_trace_section_begin () {}
void _mp_trace_section_end () {}
void _mp_trace_sections_enter () {}
void _mp_trace_sections_exit () {}
void _mp_trace_single_enter () {}
void _mp_trace_single_exit () {}
void _mp_trace_task_begin () {}
void _mp_trace_task_end () {}
void _mp_trace_task_enter () {}
void _mp_trace_task_exit () {}
void _mp_trace_workshare_begin () {}
void _mp_trace_workshare_end () {}
void _mp_trace_workshare_enter () {}
void _mp_trace_workshare_exit () {}

Include this file in the command line and the code will compile and link without problems:

ftn -o omphello -mp=nonuma omphello.f stubs.c

Status

A bug has been opened with Cray. It is confirmed that the bug has been fixed with pgi/12.5.0.

Job dependency broken between xfer and regular jobs

Tue, 12 Jun 2012 11:58:04 -0700

Symptom:

A job in xfer queue that is dependent on another job in the regular queue would still be held in the queue after its dependent job completes.

(Note: job dependency between regular jobs are still working.)

Workaround

Users can submit their xfer jobs to the "regualr" queue if the jobs are dependent on other regular jobs.

Status

Cray is working with Adaptive Computing on the job dependency issue between the xfer and regular jobs.

Resolved: Reports of Hanging Jobs on Hopper

Thu, 01 Mar 2012 09:56:49 -0800

Issue:

A number of users have reported intermittent large jobs hanging on Hopper. A job appears to start and then hangs shortly after producing no output. The job stops when the wall clock limit has been reached.

Status:

Cray has identified a few bad nodes in the system. After rebooting these nodes, no new hung jobs have been reported since Mar 12. A new xt-mpich2/5.4.4 has been installed and set to default, with a system wide MPI env set so that a job will be aborted if detected being hung. A kernel patch has been installed on Apr 3 to finally address the issue.

Resolved -- "cannot find -lhdf5_hl_cpp" compiler error with C++ code using hdf5

Tue, 24 Jan 2012 10:00:48 -0800

Symptom:

After the 1/18 system maintenance, C++ code compilation gets an error if the default hdf5/1.8.5.0 module is loaded: "/usr/bin/ld: cannot find -lhdf5_hl_cpp".

Workaround

Users can either do any of the following module swaps and recompile: 1) module swap xt-asyncpe xt-asyncpe/5.01; or 2) module swap hdf5 hdf5/1.8.7; or 3) if the default netcdf/4.1.1.0 is loaded, then do "module swap netcdf netcdf/4.1.3".

Status

Problem is resolved by setting newer netcdf/4.1.3 and hdf5/1.8.7 to the default versions. The underlying problem has been reported to Cray.

"module: command not found" in batch jobs

Fri, 06 Jan 2012 09:08:32 -0800

Sympotom:

Users with csh/tcsh as default login shells will get this error when trying to use bash syntax in the batch scripts. The following batch script will get the "module: command not found" error at run time.

#!/bin/bash
#PBS -q debug
#PBS -l mppwidth=24
#PBS -l walltime=00:10:00
cd $PBS_O_WORKDIR
module load acml

Workaround:

Use PBS keyword to specify the bash shell syntax for the batch script. The following batch script works successfully:

#PBS -S /bin/bash
#PBS -q debug
#PBS -l mppwidth=24
#PBS -l walltime=00:10:00
cd $PBS_O_WORKDIR
module load acml

Resolved -- Job cannot be executed

Tue, 03 Jan 2012 11:25:53 -0800

Symptom:

The problem happens mostly to interactive batch jobs, but it happens to regular batch jobs as well. It usually happens when user does ctrl-C at the "qsub -I" command, but it also happens when user does nothing.

Job just fails, showq can not locate the job (qstat can), then user gets an email similar to:

PBS Job Id: 1095290.sdb
Job Name: STDIN
Exec host: nid03934/9
Aborted by PBS Server
Job cannot be executed
See Administrator for help

This is a bug related to the connection timeout between MOM nodes and compute nodes with the Moab scheduler. NERSC is working with Cray and Adaptive Computing to resolve the issue.

Workaround:

The problem is transient. Resubmit usually works.

Status:

Problem resolved with an increased value for connection timeout setting in the MOM node configuration.

Pathscale/4.0.9 not compatible with default libraries

Fri, 14 Oct 2011 10:45:48 -0700

Symptom:

New xt-libsci/11.x.x does not have pathscale support. Other default libraries (netcdf, hdf5, fftw, petsc, etc.) have no pathscale/4.0.9 support.

Workaround:

-- The pathscale/4.0.9 module swaps the current default (xt-libsci/11.0.01) to xt-libsci/10.5.01 (no action from users needed).

-- To use pathscale/3.2.99, users need to swap xt-libsci to xt-libsci/10.5.01 first.

-- User codes need hdf5, netcdf, fftw, petsc, etc. will have to stick with pathscale/3.2.99 version for now.

Status:

NERSC is working with Cray to provide pathscale/4.0.9 support for these libraries.

Resolved -- Linking error when mixing C++ and Fortran using PGI under xt-asyncpe/4.9

Fri, 01 Jul 2011 13:49:37 -0700

Symptom

Mixed C++ and Fortran codes had the linking error using PGI under xt-asyncpe:

In function `std::uncaught_exception(void)': eh_util.c:(.text+0x317): undefined reference to `__zceh_uncaught_exception'

The linking works under xt-asyncpe/4.8 for pgi wrapper. It works with gnu compiler wrapper, and pgi native compiler under xt-asyncpe/4.9.

The workaround is to either use old xt-asyncpe/4.8, or still use xt-asyncpe/4.9, and use "ftn -v" to find out the exact link line, then remove "-lstdc++" from it and relink.

Status

Problem resolved with the new xt-asyncpe/5.01 installed and set to default on Hopper as of 09/24/11.

Resolved -- Incorrect path with bash under csh

Fri, 01 Jul 2011 13:38:43 -0700

Symptom:

With default csh or tcsh shell, entering bash (as login or non-login) messes up the path, causing qsub or ftn command either not found or uses wrong path. Unloadind and loading the PrgEnv-xxx module again may partially address the problem.

For example:

Under csh:
% echo $PATH
/usr/common/usgbin:...
% which qsub
/usr/common/nsg/bin/qsub
% which ftn
/opt/cray/xt-asyncpe/4.7/bin/ftn

Now enter bash:
% bash
..
% which qsub
/opt/torque/2.4.8-201004261413/bin/qsub
% which ftn
no ftn in ... 
% module unload PrgEnv-pgi 
% module load PrgEnv-pgi
% which qsub
/opt/torque/2.4.8-201004261413/bin/qsub
% which ftn
/opt/cray/xt-asyncpe/4.7/bin/ftn
% exit

Now enter bsh as a login shell:
% bash -l
% which qsub
/usr/common/nsg/bin/qsub
% which ftn
no ftn in ..
% module unload PrgEnv-pgi 
% module load PrgEnv-pgi
% which qsub
/usr/common/nsg/bin/qsub
% which ftn
/opt/cray/xt-asyncpe/4.7/bin/ftn
% exit

There are similar problems for using #!/bin/bash in your batch script if your default shell is csh/tcsh.

For a csh/tcsh user, running the following batch script generates error about
"module command not found" due to the module command in not in the default path.

#!/bin/bash
#PBS -l mppwidth=4
#PBS -l walltime=8:00
#PBS -V
cd $PBS_O_WORKDIR
module list

While the following script runs well:

#PBS -S /bin/bash
#PBS -l mppwidth=4
#PBS -l walltime=8:00
#PBS -V
cd $PBS_O_WORKDIR
module list

Workaround

Change your default shell to bash from NIM interface to use bash directly. Avoid invoking bash under csh/tcsh.

For batch script, use"#PBS -S /bin/bash" instead of "#!/bin/bash"

Status:

This bug has been resolved as of 9/14/2011.

Variation in application runtime

Thu, 09 Jun 2011 14:50:29 -0700

Some users have reported variation in application runtime while Hopper is running in production mode with many applications running concurrently. NERSC also sees runtime variability in some benchmarking codes. NERSC is exploring the causes of runtime variability hypothesizing it could be caused by the placement of an application across the network, contention for resources or a from a particular mix of jobs on the system.

We will put our variability numbers online and analyze some of the hypotheses described above and report back at the next NUG call.

"Unable to open kgni version file /sys/class/gemini/kgni0/version" error

Wed, 13 Apr 2011 10:15:26 -0700

Symptom:

Dynamic executables built with compiler wrappers running directly on the external login nodes are getting the following error message:

% ftn -dynamic -o testf testf.f
% ./testf 
./testf: /opt/pgi/10.9.0/linux86-64/10.9/libso/libnuma.so.1: no version information available (required by ./testf) 
Unable to open kdreg version file: No such file or directory 
Warning: Unable to open kgni version file /sys/class/gemini/kgni0/version errno 2 at line 599 in file cdm.c 
LIBDMAPP ERROR: Unable to open kgni version file /sys/class/gemini/kgni0/version errno 2 

aborting job: 
LIBDMAPP ERROR: Unable to open kgni version file 
Aborted

Workaround:

Please use the native compilers such as pgf90, pathcc, etc instead of the Cray compiler wrappers ftn, cc, or CC to build the executable. It will then run successfully on the external login nodes.

% pgf90 -o testf testf.f
% file testf
testf: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), for GNU/Linux 2.6.4, dynamically linked (uses shared libs), not stripped
% ./testf 
Test ok

Dynamic executables compiled with compiler wrappers could also run successfully on the MOM nodes (via "qsub -I -V -lmppwidth=24 -q interactive"). However, only short jobs (a few minutes) are appropriate to run on MOM node since these are shared resources.

% qsub -I -V -q interactive -lmppwidth=24
... wait for a new session ...

% cd $PBS_O_WORKDIR
% ftn -dynamic -o testf testf.f
% ./testf 
Test ok

Resolved -- Default version not shown in "module avail module_name" command

Wed, 13 Apr 2011 08:06:25 -0700

Symptom:

The default software version is not shown when "module avail module_name" is issued. For example:

% module avail pgi

---------------------------------------------- /opt/modulefiles -----------------------------------------------
pgi/10.9.0 pgi/11.0.0 pgi/11.1.0 pgi/11.2.0 pgi/9.0.4

Workaround:

Issue the command "module avail" without the specific module name, the output gives the complete list of available modules, including the default version of each module. For example:

% module avail
<snippet>

PrgEnv-pathscale/3.1.49                   pgi/10.9.0(default)
PrgEnv-pathscale/3.1.49A                  pgi/11.0.0
PrgEnv-pathscale/3.1.61(default)          pgi/11.1.0
PrgEnv-pgi/3.1.27A                        pgi/11.2.0
PrgEnv-pgi/3.1.35                         pgi/9.0.4

Status:

This problem has been fixed on June 28, 2011.

Job dependencies do not work as expected. Require additional string in job name

Fri, 08 Apr 2011 15:53:17 -0700

Symptom: submitting jobs which depend on other jobs does not work.

Work Around: An additional string needs to be added to the jobid @sdb. We are working to resolve this issue. See the example below. Submit "jobA" and "jobB" will run after "jobA" has completed.

>> cat dependency_jobA.scr
#!/bin/csh
#PBS -q debug
#PBS -l mppwidth=1
#PBS -l walltime=00:05:00
#PBS -N jobA
#PBS -j oe
cd $PBS_O_WORKDIR

echo "submitting jobB"

qsub -W depend=afterany:${PBS_JOBID}@sdb dependency_jobB.scr
sleep 60
echo "hi from jobA, ${PBS_JOBID}"

>> cat dependency_jobB.scr
#!/bin/csh
#PBS -q debug
#PBS -l mppwidth=1
#PBS -l walltime=00:02:00
#PBS -N jobB
#PBS -j oe
cd $PBS_O_WORKDIR
echo "hi from jobB, ${PBS_JOBID}"

DVS scalability issue on GPFS file system when reading an initialization file with a small IO size

Tue, 05 Apr 2011 10:14:07 -0700

Description:

It has been observed by some users that it takes a very long time at job start up for the initial IO when the input files are in GPFS file systems, such as /home, /project, or /global/scratch. The IO time for files in Lustre /scratch or /scratch2 are very fast. This is a DVS bug related to GFPS file systems.

Here are some of our testing results: for a 4KB IO size, the time it took 32 nodes on /global/scratch took over 11 min, while it took only 7 seconds on /scratch. Changing the file IO size to 4MB, it still took about 100 sec on /global/scratch.

Status:

Cray and NERSC are looking into DVS performance tuning.

Recommendation:

Put application IO initial/startup files on /scratch or /scratch2 Lustre files systems, instead of GPFS file systems.

Resolved: some OpenMP flags ignored in PGI C/C++ compiler

Tue, 29 Mar 2011 12:16:27 -0700

Description:

OpenMP flags other than -mp=nonuma are ignored with the PGI C and C++ wrapper on Hopper. The PGI Fortran wrapper behaves correctly.

Status:

This problem has been fixed with the xt-asyncpe/4.8 and later versions.

"gni_pub.h" not found in compilation

Tue, 29 Mar 2011 11:50:24 -0700

Description:

After the OS upgrade to CLE3.1UP03, codes using "gni_pub.h" are getting the "gni_pub.h not found" error at compile time. The workaround is to specify the detailed location of this include file in the CFLAGS or FCFLAGS. For example: set CFLAGS = "$CFLAGS -I/opt/cray/gni-headers/default/include".

Status:

Problem has been reported to Cray.

Resolved - Segfaults when mixing C/C++ and Fortran under PrgEnv-gnu

Thu, 24 Feb 2011 10:36:35 -0800

Description

If a code has C/C++ and Fortran routines, it does I/O in a Fortran code and it is built with the GNU compiler, the built executable can generate a segfault during runtime with the following error message:

[NID 03978] 2011-02-12 11:10:53 Apid 599991: initiated application termination

Application 59991 exit signals: Segmentation fault, Killed

Application 599991 resources: utime ~1s, stime ~0s

This happens with xt-asyncpe/4.7. Cray has provided the following workaround:

% CC -c main.cpp

% ftn -c sub.f

% CC main.o sub.o -u pthread_mutex_trylock -u pthread_mutex_destroy -u pthread_create

% CC -dynamic main.o sub.o

Status

The bug is fixed in xt-asyncpe/4.9 which was released on March 17, 2011 and installed as the default version on March 24, 2011. The workaround doesn't need to be applied any more.

Resolved - Trouble scheduling jobs on large memory nodes

Sat, 05 Feb 2011 09:34:19 -0800

We are having some trouble scheduling jobs on the large memory nodes. This weekend jobs requesting large memory nodes will be held until we can look at the issue more closely next week.

blog

Unable to allocate hugepages in running jobs

Symptom

Workaround

Status

Resolved -- Error Message: "ModuleCmd_Switch.c(172):ERROR:152: Module 'PrgEnv-­‐cray' is currently not loaded”

Symptom:

Status:

pgi/12.4.0 has link error with OpenMP when perftools/5.3.x is loaded

Symptom:

Workaround

Status

Job dependency broken between xfer and regular jobs

Symptom:

Workaround

Status

Resolved: Reports of Hanging Jobs on Hopper

Issue:

Status:

Resolved -- "cannot find -lhdf5_hl_cpp" compiler error with C++ code using hdf5

Symptom:

Workaround

Status

"module: command not found" in batch jobs

Sympotom:

Workaround:

Resolved -- Job cannot be executed

Symptom:

Workaround:

Status:

Pathscale/4.0.9 not compatible with default libraries

Symptom:

Workaround:

Status:

Resolved -- Linking error when mixing C++ and Fortran using PGI under xt-asyncpe/4.9

Symptom

Status

Resolved -- Incorrect path with bash under csh

Symptom:

Workaround

Status:

Variation in application runtime

"Unable to open kgni version file /sys/class/gemini/kgni0/version" error

Symptom:

Workaround:

Resolved -- Default version not shown in "module avail module_name" command

Symptom:

Workaround:

Status:

Job dependencies do not work as expected. Require additional string in job name

DVS scalability issue on GPFS file system when reading an initialization file with a small IO size

Description:

Status:

Recommendation:

Resolved: some OpenMP flags ignored in PGI C/C++ compiler

Description:

Status:

"gni_pub.h" not found in compilation

Description:

Status:

Resolved - Segfaults when mixing C/C++ and Fortran under PrgEnv-gnu

Description

Status

Resolved - Trouble scheduling jobs on large memory nodes

Resolved -- Error Message: "ModuleCmd_Switch.c(172):ERROR:152: Module 'PrgEnv-‐cray' is currently not loaded”