NERSCPowering Scientific Discovery Since 1974

NERSC File Systems


NERSC file systems can be divided into two categories:  local and global. Local file systems are only accessible on a single platform, providing best performance; global file systems are accessible on multiple platforms, simplifying data sharing between platforms. 

File systems are configured for different purposes. On each machine you have access to at least three different file system

  • Home: Permanent, relatively small storage for data like source code, shell scripts, etc. that you want to keep. This file system is not tuned for high performance from parallel jobs. Referenced by the environment variable $HOME.
  • Scratch: Large, high-performance file system. Place your large data files in this file system for capacity and capability computing. Data is purged as described below, so you must save important files elsewhere (like HPSS). Referenced by the environmental variable $SCRATCH.
  • Project: Large, permanent, medium-performance file system. Project directories are intended for sharing data within a group of researchers.

The following table shows the availability of the various file systems on each of the primary NERSC platforms.

File SystemEdisonCoriGenepoolData Transfer NodesPDSF
 Global homes 
 Global project
 Global projectb
 Local scratch

Summary of File System Policies

File SystemPathTypePeak PerformanceDefault
BackupsPurge Policy
Global homes  $HOME  GPFS Not For IO Jobs 40 GB
1,000,000 Inodes
Yes Not purged
Global project  /project/projectdirs/projectname  GPFS 130GB/Second 1 TB
1,000,000 Inodes
Yes if ≤ 5 TB quota. No if quota is > 5 TB. Not purged
Edison local scratch $SCRATCH Lustre 168GB/Second (across 3 systems)  10 TB
5,000,000 Inodes
No Files not accessed for 8 weeks are deleted 
Cori local scratch $SCRATCH Lustre 700GB/Second 20 TB
10,000,000 Inodes
No Files not accessed for 12 weeks are deleted

File Systems' Intended Use

File SystemIntended UseFile Optimization
Global homes Hold source code, executables, configuration files, etc. NOT meant to hold the output from your application runs; the scratch or project file systems should be used for computational output. Optimized for small to medium sized files.
Global project Sharing data within a team or across computational platforms. Store application output files. Intended for actively used data. Optimized for high-bandwidth, large-block-size access to large files.
Scratch file systems Edison and Cori each has large, local, parallel scratch file systems dedicated to the users of those systems. The scratch file systems are intended for temporary uses such as storage of checkpoints or application result output. If you need to retain files longer than the purge period, the files should be copied to global project or to HPPS. Optimized for high-bandwidth, large-block-size access to large files.

Summary of File System Capacities

File system Capacity
Global homes 275 TB
Global project 7 PB
Global projectb 3.6 PB
Edison local scratch1 2.1  PB
Edison local scratch2 2.1  PB
Edison local scratch3 3.2  PB
Cori local scratch 30 PB

Finding your disk usage

NERSC provides the myquota command which allows you to find your current disk space and inode usage for all file systems.

bash-3.2$ myquota

---------- Space (GB) ---------      --------------- Inode ---------------
FileSystem Usage Quota InDoubt Usage Quota InDoubt
---------------- --------- --------- --------- ----------- ----------- -----------
/scratch1 0 10240 - 6546 5000000 -
/scratch2 0 10240 - 5978 5000000 -
HOME 33 40 0 673352 1000000 1595
GSCRATCH 145 20480 0 222914 4000000 0
SCRATCHTOT 0 10240 - 12524 5000000 -

The prjquota command gives you the usage for a project directory.

bash-3.2$ prjquota mpccc
                   ------ Space (GB) -------     ----------- Inode -----------
         Project    Usage    Quota   InDoubt      Usage      Quota     InDoubt  
        --------   -------  -------  -------     -------    -------    -------  
           mpccc      5818    10240       11     3755653    4000000       1462

Purge Policy

In order to provide equitable access to scarce file system resources, NERSC "purges" certain scratch file systems on a regular basis, as indicated in the above table. Scratch file systems are intended to provide temporary storage on high performance hardware. Data that is not in active use should be placed in HPSS.

Users are not allowed to run commands (e.g., "touch" scripts) whose sole purpose is to update the time-of-last-access on purgeable files.

NERSC monitors I/O rates and I/O performance on the local Lustre file systems. Users can see Completed Jobs (Click any jobid, then click Lustre I/O)


Global Home Filesystem

Global home directories (or "global homes") provide a convenient means for a user to have access to source files, input files, configuration files, etc., regardless of the platform the user is logged in to. Wherever possible, you should refer to your home directory using the environment variable $HOME. Read More »

Project File System

The project file system is a global file system available to all NERSC computational systems. It allows groups of NERSC users to store and share data. A directory in the project file system is available by default for each repository. Read More »

Disk Quota Increase Request

Use this form to request quota increases. You need to log in using your NIM password. Read More »

Project Directory Request Form

Use this form to request a designer project directory space in the NERSC Global File system (NGF). Here are some guidelines for completing this form: The name, login, e-mail address, and phone number should be those of the requester filling out this form. The sponsoring repository will typically be that of the requester, although this is not a requirement. Participating repositories are those that are collaborating with the sponsoring repository. This field may be left blank for a… Read More »

Sponsored Storage

To accommodate projects that need storage in the NERSC Global Filesystem (NGF) beyond what NERSC can provide in its base allocations, Principal Investigators (PIs) can request to purchase a sponsored storage allocation.  A sponsored storage allocation provides 50 TB or more in 10 TB increments in the /project or /projecta file systems.  The file systems are capable of a bandwidth of 10 GB/sec per 1 PB of storage, and support high bandwidth parallel I/O workloads from all the computational… Read More »

Frontiers in Advanced Storage Technologies (FAST) project

  Working with vendors to develop new functionality in storage technologies generally not yet available to industry.  The NERSC project involves selecting particular technologies of interest, partnering with the vendor, assessing their hardware, and providing feedback or co-development to improve the product for use in HPC environments. The FAST project involves establishing long-term development collaboration agreements to develop the following opportunities: File system acceleration.  A… Read More »