blog

Hopper OS upgrade and new SW set to default next Wed, Feb 27

Thu, 21 Feb 2013 10:29:38 -0800

1) There will be a scheduled hardware and software maintenance for Hopper next Wednesday, February 27, from 7 am to 7 pm Pacific time. This is a major OS upgrade. Most applications are highly recommended to recompile (or at least relink) after the maintenance. C++ and PGAS applications are recommended to recompile and relink. Please plan your work accordingly and check the NERSC Message of the Day (MOTD) for status update: http://www.nersc.gov/live-status/motd/.

2) After the maintenance, the following software versions will be set to default on Hopper:

-- pgi/12.9.0
-- gcc/4.7.2
-- cce/8.1.3
-- intel/13.0.1.117
-- chapel/1.6.0
-- cray-mpich2/5.6.0 (xt-mpich2/5.6.0); cray-shmem/5.6.0 (xt-shmem/5.6.0)
-- cray-libsci/12.0.0 (xt-libsci/12.0.0)
-- cray-petsc/3.3.00 (xt-petsc/3.3.00); cray-petsc-complex/3.3.00 (petsc-complex/3.3.00)
-- cray-trilinos/10.12.1.1
-- cray-tpsl/1.3.01
-- fftw/2.1.5.4
-- cray-netcdf/4.2.1.1
-- cray-hdf5/1.8.9; cray-hdf5-parallel/1.8.9
-- parallel-netcdf/1.3.1
-- perftools/6.0.1
-- papi/5.0.1
-- atp/1.6.0
-- stat/1.2.1.3
-- java/jdk1.7.0_07
-- xt-asyncpe/5.16

The availability of the above software versions were announced on Dec 19 and Jan 18.

Notice Cray has changed some of the module names from "xt-xxx" to "cray-xxx", for example: xt-libsci is now cray-libsci, and xt-mpich2 is now cray-mpich2. Also some of the Cray supported modules now have a "cray-" prefix to their module names, for example: netcdf is now cray-netcdf, and petsc is now cray-petsc, etc.

Hopper scheduled maintenance tomorrow (Sept 19) and /project outage

Tue, 18 Sep 2012 16:46:01 -0700

There will be a scheduled hardware and software maintenance for Hopper next Wednesday, Sept 19, from 6:30 am to midnight Pacific time. Please plan your work accordingly and check the NERSC Message of the Day (MOTD) for status update: http://www.nersc.gov/live-status/motd/.

The /project file system (also known as /global/project) will be unavailable from 8am Wednesday, Sept 19 until 5pm Friday, Sept 21, during and after the scheduled Hopper maintenance.

If your job depends on /project, please add the following "gres" setting in your batch script (so that your job won't start and fail during the /project outage):
#PBS -l gres=project

If your job is already queued (but not yet running), the following "qalter" command can be used to add the "gres" setting:
% qalter -l gres=project my_jobid

Please use "gres" settings in your batch scripts

Tue, 04 Sep 2012 14:03:26 -0700

We would like to encourage you to use the generic resources ("gres") setting for various file systems that your batch jobs use. This feature is currently available on Hopper and Carver. The advantage of this setting is that your jobs won't start (thus won't fail) during a scheduled file system maintenance.

The syntax for the "gres" setting is:
#PBS -l gres=filesystem1[%filesystem2%filesystem3...] (new recommendation)
or
#PBS -l gres=filesystem1:1[%filesystem2:1%filesystem3:1...] (as announced before)

Note that the "%" character means "and". Therefore, if multiple file systems are specified, the job will not start if *any* of the specified file systems are unavailable.

The file systems available for "gres" setting are:
on Hopper: scratch, scratch2, gscratch, project, and projectb
on Carver: gscratch, project, and projectb
(note: home is not a defined resource, since it is used by every job anyway).

Below are some sample "gres" lines:
#PBS -l gres=project
#PBS -l gres=scratch
#PBS -l gres=gscratch2%project
#PBS -l gres=scratch%scratch2%projectb

To add this option to a queued job (not in "running" state), the following command can be used:
% qalter -l gres=... my_jobid

We will have a scheduled project file system maintenance on Sept 19, lasting for 2 to 3 days (details TBA). If your current or future batch jobs depend on the project file system, please use the "gres=project" setting in your batch scripts (or use the qalter command for scripts already submitted). This will prevent your jobs from failing during this outage.

In general, we strongly encourage you *always* to include the "gres" directive in your batch scripts to specify the file systems needed to protect the jobs during future scheduled file system outages.

high thruput queue now available on Hopper

Thu, 30 Aug 2012 16:27:23 -0700

A new batch queue named "thruput" has been implemented on Hopper to support the increased high throughput computing needs from the user community. The queue limits for this queue are as follows:

-- max wall time is 168 hrs
-- max node count is 2 (max core count is 48)
-- max queue-able jobs per user is 500
-- max running jobs from all users in this queue is 500
-- has same priority as of the "reg_small" queue
-- charging factor is 1

max walltime for low queue is increased to 24 hrs on Hopper

Thu, 31 May 2012 15:48:53 -0700

We have increased the max walltime for the low queue on Hopper from 12 to 24 hrs.

Hopper compilers and DDT short outage next Wed, May 16

Thu, 10 May 2012 15:47:53 -0700

Due to a scheduled maintenance for the License Servers, most of the compilers (except GNU) and the DDT debugger on Hopper will not be available from 10:30 am to 12:30 pm on Wednesday, May 16.

If there are any questions or concerns, please contact "consult at nersc dot gov".

Hopper scheduled maintenance next Tues (May 8) and new software available

Tue, 01 May 2012 15:46:32 -0700

1) In conjunction with the NERSC Center Wide Outage for global file systems maintenance, there will be a scheduled hardware and software maintenance for Hopper next Tuesday, May 8, from 8 am to 6 pm Pacific time. Please plan your work accordingly and check the NERSC Message of the Day (MOTD) for status update: http://www.nersc.gov/live-status/motd/.

2) The following new software versions are available now (as non-default) on Hopper:
-- pathscale/4.0.12.1
-- intel/12.1.3.293
-- pgi/12.2.0
-- cce/8.0.3
-- xt-libsci/11.0.06
-- acml/5.1.0
-- perftools/5.3.1
-- papi/4.3.0
-- jdk/1.7.0_03

Many of the above software depend on the xt-asyncpe/5.08 instead of the default loaded xt-asyncpe/5.07 module. Please issue the following commands before testing the new software versions:
% module swap xt-asyncpe xt-asyncpe/5.08

FYI, Issuing the command "module help module_name/version" (e.g., "module help pgi/12.2.0") will display some useful software release information.

More information about the above new software versions can be found in the following CrayDoc Release Announcements:
http://docs.cray.com/books/S-9401-1202/S-9401-1202.pdf
http://docs.cray.com/books/S-9401-1203/S-9401-1203.pdf

If there are no problems reported, the above list of new software versions (along with the new xt-asyncpe/5.08) will be set to default on 5/8 at the scheduled system maintenance. The special command of "module swap xt-asyncpe ..." above will not be needed after we set the new defaults next week.

If there are any questions or comments, please contact "consult at nersc dot gov".

Cluster Compatibility Mode is now available on Hopper

Tue, 10 Apr 2012 15:43:01 -0700

We are pleased to announce a new feature on Hopper, Cray Cluster
Compatibility Mode (CCM) which allows applications that previously could
only run on Carver to run on Hopper.

Cluster Compatibility Mode (CCM) is a Cray software solution that
provides services needed to run most cluster-based independent software
vendor (ISV) applications on the Cray XE6. It supports the standard
Linux services, such as ssh, rsh, nscd, and ldap, and complete root file
systems on the compute nodes (therefore supports dynamic shared
libraries). With CCM, the applications that couldn't run on Hopper due
to the lack of TCP/IP support now can run on Hopper. The Gaussian (G09),
NAMD replica simulation, and WIEN2k codes are the applications that are
enabled on Hopper recently through CCM. Additionally, CCM enables serial
workloads to run on Hopper.

You can access CCM through the ccm_queue on Hopper. Due to the slower
processor speed on Hopper (2.1GHz) you may find applications run slower
on Hopper CCM when compared to Carver (2.7GHz), however, we expect the
queue wait time to be shorter, which we hope will give users the
incentive to try CCM. We have also created a new queue ccm_int, which
has the same queue priority as the debug/interactive queue, to
accommodate the interactive CCM workloads, eg., debugging, data
analysis. Currently the applications and tools available for CCM are
g09, namd_ccm, wien2k_ccm, matlab_ccm, ipm_ccm and ddt_ccm. For more
information about CCM and how to use it on Hopper, please visit our
website at,

http://www.nersc.gov/users/computational-systems/hopper/cluster-compatibility-mode/

If you have been limited by the lack of the TCP/IP support on Hopper
computing environment in the past, you are encouraged to try out CCM on
Hopper. CCM is a new product of Cray, and it is still in the
experimental stage. Your feedback and comments are especially welcome.

Hopper scheduled maintenance on Tue March 20 and new software available

Tue, 13 Mar 2012 17:06:33 -0700

1) There will be a scheduled hardware and software maintenance for Hopper next Tuesday, March 20, from 8 am to 4 pm Pacific time. Please plan your work accordingly and check the NERSC Message of the Day (MOTD) for status update: http://www.nersc.gov/live-status/motd/.

2) The following new software versions are now available on Hopper:
-- pgi/11.10.0
-- cce/8.0.0, 8.0.1, 8.0.2
-- gcc/4.6.2
-- pathscale/4.0.9
-- intel/12.1.1.256, 12.1.2.273
-- xt-mpich2/5.4.1, 5.4.2, 5.4.3; xt-shmem/5.4.1, 5.4.2, 5.4.3
-- perftools/5.3.0
-- papi/4.2.0
-- xt-libsci/11.0.04, 11.0.05
-- trilinos/10.8.3.0
-- petsc/3.2.00, petsc-complex/3.2.00
-- tpsl/1.2.00
-- ga/5.0.2
-- onesided/1.5.0
-- atp/1.4.1, 1.4.2
-- java/jdk1.7.0_02

Please issue the command below first when using the new software versions before they become the default:
% module swap xt-asyncpe xt-asyncpe/5.07

After the maintenance, the following versions will be set to default:
-- cce/8.0.2
-- gcc/4.6.2
-- pathscale/4.0.9
-- intel/12.1.2.273
-- perftools/5.3.0
-- papi/4.2.0
-- xt-libsci/11.0.05
-- trilinos/10.8.3.0
-- petsc/3.2.00, petsc-complex/3.2.00
-- tpsl/1.2.00
-- ga/5.0.2
-- onesided/1.5.0
-- atp/1.4.2
-- java/jdk1.7.0_02

More information about the above new software versions can be found in the following CrayDoc Release Announcements:
http://docs.cray.com/books/S-9401-1111//S-9401-1111.pdf
http://docs.cray.com/books/S-9401-1112//S-9401-1112.pdf
http://docs.cray.com/books/S-9401-1201//S-9401-1201.pdf
http://docs.cray.com/books/S-9401-1202//S-9401-1202.pdf

Purging of Hopper scratch file systems to start on Mar 14

Tue, 28 Feb 2012 09:57:21 -0800

The purging for the Hopper scratch file systems will start from Wednesday, March 14. The files in your $SCRATCH and $SCRATCH2 will be deleted ("purged") if they are older (defined by last access time) than 12 weeks once the purge starts.

Please take some time now to clean up contents, back up all important files (eg. to HPSS), and reduce your overall usage as much as you can.

Hopper scheduled maintenance and SW new default versions on Wednesday, Jan 18

Wed, 11 Jan 2012 11:52:02 -0800

1) There will be a scheduled hardware and software maintenance for Hopper next Wednesday, January 18, from 7 am to 10 pm Pacific time. This is a major OS upgrade. Users are recommended to recompile your applications after the maintenance. Please plan your work accordingly and check the NERSC Message of the Day (MOTD) for status update: http://www.nersc.gov/live-status/motd/.

2) After the maintenance, the following software versions will be set to default on Hopper:

-- pgi/11.9.0
-- cce/7.4.4
-- intel/12.0.5.220
-- chapel/1.4.0
-- xt-mpich2/5.4.0, xt-shmem/5.4.0
-- perftools/5.2.3
-- papi/4.1.4
-- xt-libsci/11.0.03
-- trilinos/10.6.4.0
-- petsc/3.1.09, petsc-complex/3.1.09
-- tpsl/1.1.01
-- fftw/2.1.5.3

The availability of the above software versions was announced on Dec 6.

If there are any questions or comments, please contact "consult at nersc dot gov".

max wall time for reg_small increased to 48 hrs

Wed, 04 Jan 2012 12:12:56 -0800

The max wall time for Hopper jobs using up to 682 nodes (16,368 cores, corresponds to the "reg_small" queue) has been increased from 36 hrs to 48 hrs. Notice that jobs should still be submitted to the "regular" queue.

New scavenger Queue

Mon, 07 Nov 2011 12:49:37 -0800

There is a new queue, called scavenger, available on Hopper and Franklin to users with a zero or negative balance in one of their repositories. This applies to both total repository balances as well as per-user balances. The queue is not available for jobs submitted against a repository with a positive balance. The charging rate for this queue is 0 and the priority on all systems is lower than the “low” queue. The Hopper scavenger queue limits are:

1-683 nodes
6 hour max wallclock
2 concurrent job max

To use the scavenger queue please specify the following line your batch script:

#PBS -q scavenger

If you have multiple repositories, you can specify the repository to use for the job by adding the following line to your batch script:

#PBS -A <repo>

large Hopper jobs now discounted 40 percent

Tue, 04 Oct 2011 12:58:59 -0700

Effective October 1 NERSC has increased the discount for large Hopper jobs from 25% to 40%. Large jobs are defined to be those using 683 or more nodes. The new queue charge rate for the reg_med, reg_big and reg_xbig queues is now .6 (and has been applied retroactively to Oct 1).

We hope that this change will allow projects with work that can run efficiently on 683 or more nodes will now have enough allocated time to permit these large runs.

Hopper scheduled maintenance and new software available

Wed, 14 Sep 2011 16:05:07 -0700

Hopper will have a scheduled maintenance on next Wed, September 21, from 8am to 4pm Pacific time. A long list of new software versions are now available (as non-default) on Hopper. Details can be found in this announcement.

Intel compiler available on Hopper

Wed, 10 Aug 2011 11:19:20 -0700

The Intel compiler suite is now available on Hopper. The current installed version is 12.0.4.191.

To use the Intel compiler:
% module swap PrgEnv-pgi PrgEnv-intel

Then use the Cray wrappers ftn, cc, and CC to compile the Fortran, C, and C++ codes, respectively.

More details of using the Intel compiler on Hopper can be found at Intels Compilers on Hopper.

Problems submitting Jobs on Hopper -- Fixed

Wed, 03 Aug 2011 10:10:53 -0700

We are experiencing problems submitting jobs to Hopper. We are looking into the issue and will get it fixed as soon as possible.

/project and /global/scratch expected available time

Wed, 13 Jul 2011 10:07:03 -0700

As announced yesterday, expected available time for /global/scratch is 5pm today (July 13), and expected available time for /project is 5pm tomorrow (July 14): http://www.nersc.gov/REST/announcements/message_text.php?id=2073

Hopper scheduled maintenance and new software available

Wed, 06 Jul 2011 15:04:53 -0700

In conjunction with the NERSC systems Outages next week (for NERSC Global File Systems maintenance) announced earlier today (see details at: http://www.nersc.gov/REST/announcements/message_text.php?id=2070), Hopper will have a scheduled maintenance on next Wed, July 13, from 8am to 1pm Pacific time.

/project and /global/scratch directories will not be available on Hopper from 8am to 1pm on Wed July 13. User jobs that need to use the /project and/or /global/scratch file system should add the "gres" directive in their batch job scripts to indicate this requirement. (see detailed instructions at: http://www.nersc.gov/REST/announcements/message_text.php?id=2071)

The following new software versions are available now (as non-default version) on Hopper, and will be set to default at the maintenance.
-- pgi/11.3.0
-- perftools/5.2.0
-- xt-mpich2/5.3.0 and xt-shmem/5.3.0
-- cce/7.4.0

Hopper batch walltime increase and new xfer queue

Wed, 29 Jun 2011 21:09:37 -0700

There are two batch queue configuration changes on Hopper:

1) User jobs using fewer than 4,096 nodes can now run for up to 36 hrs. In particular, the max wall time limits for reg_small (1-683 nodes), reg_med (684-2,048 nodes), and reg_big (2,049-4,096 nodes) jobs have been increased from 24 to 36 hrs.

2) A new "xfer" queue has been introduced. Users can use the xfer job to pre-stage input files for a large simulation and archive output files afterwards without the penalty of being charged for the large amount of compute nodes during the file transfers.

The xfer queue on Hopper is being implemented on a local batch server (hopper06), which is separate from the main batch server on the main Hopper XE. So there are some special notes about the xfer jobs, for example: their job ids are in a different format (such as 6000345.hopper06), and since xfer jobs do not use any compute nodes, the torque keyword "mppwidth" should not be specified in an xfer batch script.

The details of the batch queue configuration can be found at:
http://www.nersc.gov/users/computational-systems/hopper/running-jobs/queues-and-policies/

And the details of submitting an "xfer" job can be found at:
http://www.nersc.gov/users/computational-systems/hopper/running-jobs/batch-jobs/#xfer