NERSCPowering Scientific Discovery Since 1974

File Storage and I/O

Disk Quota Change Request Form

Hopper File Systems

Hopper has 5 user file systems which provide different degrees of storage,  performance and permanence.  The table below summarizes these file systems:

File SystemHomeLocal ScratchGlobal ScratchProject
$GSCRATCH None.  Must use
  • Global home file system shared with other NERSC systems.
  • All NERSC machines mount the same home directory.
  • GPFS filesystem.
  • Where users land when they log into the system.
  • Two 1PB Lustre local file systems. 
  • Local means the files can not be viewed on other NERSC systems.
  • Large (2 PB) GPFS file system for temporary storage.
  • Currently mounted on all NERSC systems except PDSF.
  • GPFS global file system mounted on all NERSC systems.
Default Quota
  • 40 GB
  • 1 million inodes
  • 5 TB
  • 5 million inodes
  • 20 TB
  • 2 million inodes
  • 1 TB
  • 1 million inodes
Intended Purpose
  • Shell initializations
  • Storing source code
  • Compiling codes
  • Not intended for IO intensive applications
  • Running production applications
  • I/O intensive jobs
  • Temorary storage of large files
  • Alternative file system to run production applications
  • Aid users whose workflow requires running on multiple platforms
  • Temorary storage of large files
  • Running production applications
  • Groups needing shared data access
  • Projects running on multiple NERSC machines
Peak Performance Low, ~100 MB/sec 35 GB/sec for each 80 GB/sec peak 40 GB/sec
Purged? No Yes, files older than 12 weeks are purged Yes, files older than 12 weeks are purged No

Scratch Directories

Hopper is configured with two distinct scratch file systems named /scratch and /scratch2; every user has access to a scratch directory in each file system.  These directories should always be referenced using the environment variables $SCRATCH and $SCRATCH2. Both of these file systems are available from all nodes and are tuned for high performance. You may run using both scratch file systems but are encouraged to choose one or the other for your primary work.

There is a single (large) quota (space and inode) for each user that applies to the combined contents of $SCRATCH and $SCRATCH2. If your combined usage of $SCRATCH and $SCRATCH2 exceeds your quota, you will not be able to submit batch jobs until you reduce your combined usage.

The "myquota" command (with no options) will display your current usage and quota.  NERSC sometimes grants temporary quota increases for legitimate purposes. To apply for such an increase, please use the Disk Quota Increase Form.

Note that both scratch file systems are subject to purging.  Files in your $SCRATCH and $SCRATCH2 that are older than 12 weeks (defined by last access time) are removed. Please back up your important files to HPSS. 

All of NERSC's global file systems are available on Hopper. Additionally, Hopper has 2 PB of locally attached high-performance /scratch disk space  For information on the NERSC file systems, see the link at right.

Scratch Filesystem Configuration

  Size TBAggregate Peak Performance# of Disks# IO Servers (OSSs)OSTsFile System SoftwareDisk Array Vendor
$SCRATCH 1 PB 35 GB/sec  13 26 156 Lustre LSI
$SCRATCH2 1 PB 35 GB/sec 13 26 156 Lustre LSI

SCRATCH and SCRATCH2 both have the same configuration.

    • 13 LSI 7900 disk controllers
    • Each disk controller is served by 2 I/O servers called OSSs (Object Storage Servers)
    • Each OSS host 6 OSTs (Object Storage Target) which a user can think of as a software abstraction of a physical disk
    • Fiber Channel 8 connectivity from OSSs to the LSI disk controllers
    • Infiniband connects the Lustre router nodes in the 3d torus through a QDR switch to the OSSs

In total each /scratch file system has 156 OSTs which is the lowest layer with which users need to interact.  When a file is created in /scratch it is "striped" or split across two different OSTs, which is the default.  Lustre file systems at other computing centers may set a different default based on their workload.  Striping is a technique to increase I/O performance.  Instead of writing to a single disk, striping to two disks allows the user to potentially double read and write bandwidth.  

Do Not Use /tmp explicitly

WARNING: Do not attempt to explicitly use a file system named /tmp. Your job may fail or be deleted if it writes to /tmp.  Some software tools (editors, compilers, etc.) use the location specified by the $TMPDIR environment variable to store temporary files. Additionally, Fortran codes which open files with status="scratch" will write those files into $TMPDIR. On many Unix systems, $TMPDIR is set to /tmp. NERSC has set $TMPDIR to be $SCRATCH. Please do not redefine $TMPDIR!

Related Pages

Optimizing I/O performance on the Lustre file systems

Optimizing I/O performance on the $GSCRATCH and $PROJECT file systems 

Real time $SCRATCH I/O rates

Transferring Data

Lessons Learrned from a Hero IO run on Hopper