NERSCPowering Scientific Discovery Since 1974

Retirement Plans

The Carver system will be retired at the end of September, 2015.  Carver users should prepare for this by migrating their workload to one of the other NERSC systems, primarily Edison.  This page is provided to assist users in that migration.

Schedule

Carver's final day of operation will be Wednesday, September 30, 2015.  The precise timeline has not yet been determined.  It is likely that the system will not be "drained"; that is, batch jobs will be allowed to run until the system is shut down.

Edison Details for Carver Users

Configuration

Edison

  • 12 Login Nodes
    • 4 quad-core 2.0 GHz Intel "Sandy Bridge" processors (16 cores total)
    • 512 GB, 1333 MHz DDR3 memory
    • 32 GB/core
  • 5,576  Compute Nodes
    • 2 12-core 2.4 GHz Intel "Ivy Bridge" processors (24 cores total)
    • 64 GB, 1600 MHz DDR3 memory
    • 2.67 GB/core

Carver

  • 4 Login Nodes
    • 2 quad-core 2.67 GHz Intel "Nehalem" processors (8 cores total)
    • 48 GB, 1066 MHz DDR3 memory
    • 6 GB/core
  • 960 "Smallmem" Compute Nodes
    • 2 quad-core 2.67 GHz Intel "Nehalem" processors (8 cores total)
    • 24 GB, 1333 MHz DDR3 memory
    • 3 GB/core
  • 160 "Bigmem" Compute Nodes
    • 2 quad-core 2.67 GHz Intel "Nehalem" processors (8 cores total)
    • 48 GB, 1066 MHz DDR3 memory
    • 6 GB/core
  • 80 "Serial" Compute Nodes
    • 2 6-core 2.67 GHz Intel "Westmere" processors (12 cores total)
    • 48 GB, 1333 MHz DDR3 memory
    • 4 GB/core
  • 2 "Xlmem" Compute Nodes
    • 4 8-core 2.00 GHz Intel "Nehalem-EX" processors (32 cores total)
    • 1 TB, 1066 MHz DDR3 memory
    • 32 GB/core

Programming

Edison

The default compiler is Intel.  Cray and GNU compilers are also available.  Edison's MPI implementation is based on MPICH.  Cray's compiler wrappers (ftn, cc, and CC) provide the necessary settings for the MPI libraries, as well as for some commonly-used math libraries.

Carver

The default compiler is PGI.  Intel and GNU compilers are also available.  Carver's MPI implementation (Open MPI) is based on LAM.  Carvers's compiler wrappers (mpif90, mpicc, and mpiCC) provide the necessary settings for the MPI libraries.

Running Jobs

Edison

Edison's batch system comprises the Moab scheduler and Torque resource manager.  Batch jobs request computational resources (cores) via the "mppwidth" directive.  Computational processes are launched onto compute nodes via the "aprun" command.  Edison provides batch queues that allow jobs to use a maximum of 131,088 cores, and other queues that allow jobs to run for a maximum walltime of 48 hours.  Edison will have a serial queue available be the end of summer, 2014.

Carver

Carver's batch system comprises the Moab scheduler and Torque resource manager.  Batch jobs request computational resources (nodes) via the "nodes" directive.  Computational processes are launched onto compute nodes via the "mpirun" command.  Carver provides batch queues that allow jobs to use a maximum of 512 cores, and other queues that allow jobs to run for a maximum walltime of 504 hours.  Additionally, Carver provides a serial queue.

File Storage

Edison

Edison provides access to the following global GPFS file systems:

  • /global/homes
  • /global/common
  • /global/scratch
  • /global/project
  • /global/projectb

Edison also provides 3 local Lustre scratch file systems:

  • /scratch1
  • /scratch2
  • /scratch3

All Edison users are allocated a scratch directory in either /scratch1 or /scratch2.  Access to /scratch3, which has twice the bandwidth of the other two, is by request only.

Carver

Carver provides access to the following global GPFS file systems:

  • /global/homes
  • /global/common
  • /global/scratch
  • /global/project
  • /global/projectb

Note that Carver has no local scratch file systems.

Software

Edison

Edison's Linux operating system is based on the SUSE distribution.  The compute nodes use a "stripped-down" version known as Extreme Scalability Mode (ESM).  In order to support a wider range of third-party applications, Cray also provides Cluster Compatiblity Mode (CCM).  NERSC provides a wide range of tools, libraries, and applications.  Hadoop will be available on Edison in the future, although no specific date has yet been determined.

Carver

Carver's Linux operation system is based on the Scientific Linux distribution, which is based on the RedHat distribution.  The compute nodes are not as full-featured as the login nodes, mostly due to the absence of development rpms and desktop-only applications.  NERSC provides a wide range of tools, libraries, and applications, including Hadoop.