NERSCPowering Scientific Discovery Since 1974

NERSC File Systems

Overview

NERSC file systems can be divided into two categories:  local and global.  Local file systems are only accessible on a single platform, providing best performance; global file systems are accessible on multiple platforms, simplifying data sharing between platforms. 

File systems are configured for different purposes. On each machine you have access to at least three different file system

  • Home: Permanent, relatively small storage for data like source code, shell scripts, etc. that you want to keep. This file system is not tuned for high performance from parallel jobs. Referenced by the environment variable $HOME.
  • Scratch: Large, high-performance file system. Place your large data files in this file system for capacity and capability computing. Data is purged as described below, so you must save important files elsewhere (like HPSS). Referenced by the environmental variable $SCRATCH.
  • Project: Large, permanent, medium-performance file system. Project directories are intended for sharing data within a group of researchers.

The following table shows the availability of the various file systems on each of the primary NERSC platforms.

File SystemHopperEdisonCarverGenepoolData Transfer NodesPDSF
 Global homes 
Y
Y
Y
Y
Y
 
 Global scratch 
Y
Y
Y
Y
Y
Y
 Global project
Y
Y
Y
Y
Y
Y
 Global projectb
Y
Y
Y
Y
Y
 
 Local scratch
Y
Y
 
Y
   

Summary of File System Policies

File SystemPathTypePeak PerformanceDefault
Quota
BackupsPurge Policy
Global homes  $HOME  GPFS Not For IO Jobs 40 GB
1,000,000 Inodes
Yes Not purged
Global scratch  $GSCRATCH  GPFS 80GB/Second 20 TB
4,000,000 Inodes
No Files not accessed for 12 weeks are deleted
Global project  /project/projectdirs/projectname  GPFS 40GB/Second 1 TB
1,000,000 Inodes
Yes if ≤ 5 TB quota. No if quota is > 5 TB. Not purged
Hopper local scratch $SCRATCH and $SCRATCH2  Lustre 35GB/Second Each 5 TB
5,000,000 Inodes
(Combined)
No Files not accessed for 12 weeks are deleted.
Edison local scratch $SCRATCH Lustre 168GB/Second (across 3 systems)  10 TB
5,000,000 Inodes
No Files not accessed for 12 weeks are deleted

File Systems' Intended Use

File SystemIntended UseFile Optimization
Global homes Hold source code, executables, configuration files, etc. NOT meant to hold the output from your application runs; the scratch or project file systems should be used for computational output. Optimized for small to medium sized files.
Global project Sharing data within a team or across computational platforms. Store application output files. Intended for actively used data. Optimized for high-bandwidth, large-block-size access to large files.
Scratch file systems Hopper and Edison each have large, local, parallel scratch file systems dedicated to the users of those systems.  The global scratch file system, while accessible from all systems, is primarily for use with our midrange systems (Carver, Genepool, PDSF, etc.).  The scratch file systems are intended for temporary uses such as storage of checkpoints or application result output.  If you need to retain files longer than the purge period, the files should be copied to global project or to HPPS. Optimized for high-bandwidth, large-block-size access to large files.

Summary of File System Capacities

File system Capacity
Global homes 246 TB
Global scratch 3632 TB
Global project 5150 TB
Global projectb 2620 TB
Hopper local scratch1 1117 TB
Hopper local scratch2 1106 TB
Edison local scratch1 2.1  PB
Edison local scratch2 2.1  PB
Edison local scratch3 3.2  PB

Finding your disk usage

NERSC provides the myquota command which allows you to find your current disk space and inode usage for all file systems.

bash-3.2$ myquota

---------- Space (GB) ---------      --------------- Inode ---------------
FileSystem Usage Quota InDoubt Usage Quota InDoubt
---------------- --------- --------- --------- ----------- ----------- -----------
/scratch1 0 10240 - 6546 5000000 -
/scratch2 0 10240 - 5978 5000000 -
HOME 33 40 0 673352 1000000 1595
GSCRATCH 145 20480 0 222914 4000000 0
SCRATCHTOT 0 10240 - 12524 5000000 -

The prjquota command gives you the usage for a project directory.

bash-3.2$ prjquota mpccc
                   ------ Space (GB) -------     ----------- Inode -----------
         Project    Usage    Quota   InDoubt      Usage      Quota     InDoubt  
        --------   -------  -------  -------     -------    -------    -------  
           mpccc      5818    10240       11     3755653    4000000       1462

Purge Policy

In order to provide equitable access to scarce file system resources, NERSC "purges" certain scratch file systems on a regular basis, as indicated in the above table.  Scratch file systems are intended to provide temporary storage on high-performace hardware.  Data that is not in active use should be placed in HPSS.

Users are not allowed to run commands (e.g., "touch" scripts) whose sole purpose is to update the time-of-last-access on purgeable files.

Local Scratch on Hopper

Hopper uses Lustre for its local scratch file systems. These file systems are not available on any other computer systems.  For more information, please see the File Storage Page for Hopper.

NERSC monitors I/O rates and I/O performance on the local Lustre file systems.  Users can see live and historical I/O rates.

 

Global Home Filesystem

Global home directories (or "global homes") provide a convenient means for a user to have access to source files, input files, configuration files, etc., regardless of the platform the user is logged in to. Wherever possible, you should refer to your home directory using the environment variable $HOME. Read More »

Global Scratch File System

The global scratch file system (or "global scratch") can be accessed by multiple NERSC systems. Its default quota is much larger than the Global Home quota, so it can be used to temporarily store large amounts of data. This file system is periodically purged. Read More »

Project File System

The project file system is a global file system available to all NERSC computational systems. It allows groups of NERSC users to store and share data. A directory in the project file system is available by default for each repository. Read More »

Disk Quota Increase Request

Use this form to request quota increases. You need to log in using your NIM password. Read More »

Project Directory Request Form

Use this form to request a designer project directory space in the NERSC Global File system (NGF). Here are some guidelines for completing this form: The name, login, e-mail address, and phone number should be those of the requester filling out this form. The sponsoring repository will typically be that of the requester, although this is not a requirement. Participating repositories are those that are collaborating with the sponsoring repository. This field may be left blank for a… Read More »

Sponsored Storage

To accommodate projects that need storage in the NERSC Global Filesystem (NGF) beyond what NERSC can provide in its base allocations, Principal Investigators (PIs) can request to purchase a sponsored storage allocation.  A sponsored storage allocation provides 50 TB or more in 10 TB increments in the /project file system.  The file system is capable of a bandwidth of 10 GB/sec per 1 PB of storage, and supports high bandwidth parallel I/O workloads from all the computational systems at… Read More »

Frontiers in Advanced Storage Technologies (FAST) project

  Working with vendors to develop new functionality in storage technologies generally not yet available to industry.  The NERSC project involves selecting particular technologies of interest, partnering with the vendor, assessing their hardware, and providing feedback or co-development to improve the product for use in HPC environments. The FAST project involves establishing long-term development collaboration agreements to develop the following opportunities: File system acceleration.  A… Read More »