About

HPSS Mass Storage

The High Performance Storage System (HPSS) is a modern, flexible, performance-oriented mass storage system. It has been used at NERSC for archival storage since 1998.

HPSS is Herarchical Storage Management (HSM) software developed by a collaboration of DOE labs, of which NERSC is a participant, and IBM.  The HSM software enables all user data to be ingested onto high performance disk arrays and automatically migrated to a very large enterprise tape subsystem for long-term retention.  The disk cache in HPSS is designed to retain 5 days worth of new data and the tape subsystem is designed to provide the most cost-effective long-term scalable data storage available.

At NERSC, the data in storage doubles almost every year. As of Feb 28 2012, we have over 14.6 petabytes of data stored in over 128 million files in our HPSS User system (archive.nersc.gov) and 11.7 petabytes of data stored in over 15 million files in our HPSS Backup system (hpss.nersc.gov). HPSS sustains an average transfer rate of more than 100 MB/s, 24 hours per day, with peaks to 450 MB/s.

NERSC has two HPSS systems:

• archive
  • Used for storing user files
  • Default system used when no argument is given on the HSI or PFTP command on NERSC production machines.
  • Better disk cache performance
  • Generally delivers higher transfer rates and lower response times.
  • Also referred to as the "HPSS User System".
• hpss
  • Used for computer system backups.
  • Initial releases of new storage software are deployed here first.
  • Original NERSC HPSS production system.
  • Also referred to as the "HPSS Backup System".

Some characteristics of the NERSC HPSS systems:

• Theoretical capacity: 200 Petabytes.
• Buffer (disk) cache: 288 Terabytes.
• Theoretical maximum throughput: 6.4 GB/sec.

Users can access NERSC's HPSS machines through a variety of clients such as hsi, htar, ftp, pftp, and grid clients.  For more information about the specific features of each HPSS client see Accessing HPSS.