NERSC File Systems
Overview
NERSC file systems can be divided into two categories: local and global. Local file systems are only accessible on a single platform, providing best performance; global file systems are accessible on multiple platforms, simplifying data sharing between platforms.
File systems are configured for different purposes. On each machine you have access to at least three different file system
- Home: Permanent, relatively small storage for data like source code, shell scripts, etc. that you want to keep. This file system is not tuned for high performance from parallel jobs. Referenced by the environment variable $HOME.
- Scratch: Large, high-performance file system. Place your large data files in this file system for capacity and capability computing. Data is purged as described below, so you must save important files elsewhere (like HPSS). Referenced by the environmental variable $SCRATCH.
- Project: Large, permanent, medium-performance file system. "Project directories" are available upon request are intended for sharing data within a group of researchers.
The following table shows the availability of the various file systems on each of the primary NERSC platforms.
| File System | Hopper | Edison (Phase 1) | Carver | Genepool | Data Transfer Nodes | PDSF |
|---|---|---|---|---|---|---|
Global homes |
Y |
Y |
Y |
Y |
Y |
|
Global scratch |
Y |
Y |
Y |
Y |
Y |
|
Global project |
Y |
Y |
Y |
Y |
Y |
Y |
Global projectb |
Y |
Y |
Y |
Y |
Y |
|
Local scratch |
Y |
Y |
Y |
Summary of File System Policies
| File System | Path | Type | Default Quota | Backups | Purge Policy |
|---|---|---|---|---|---|
| Global homes | $HOME | GPFS | 40 GB 1,000,000 Inodes |
Yes | Not purged |
| Global scratch | $GSCRATCH | GPFS | 20 TB 2,000,000 Inodes |
No | Files not accessed for 12 weeks are deleted |
| Global project | /project/projectdirs/projectname | GPFS | 4 TB 4,000,000 Inodes |
Yes if under 5 TB quota. No if quota is ≥ 5 TB. | Not purged |
| Hopper local scratch | $SCRATCH and $SCRATCH2 | Lustre | 5 TB 5,000,000 Inodes (Combined) |
No | Files not accessed for 12 weeks are deleted. |
| Edison (Phase 1) local scratch | $SCRATCH | Lustre | 5 TB 5,000,000 Inodes |
No |
File Systems' Intended Use
| File System | Intended Use | File Optimization |
|---|---|---|
| Global homes | Hold source code, executables, configuration files, etc. NOT meant to hold the output from your application runs; the scratch or project file systems should be used for computational output. | Optimized for small to medium sized files. |
| Global project | Sharing data within a team or across computational platforms. Store application output files. Intended for actively used data. | Optimized for high-bandwidth, large-block-size access to large files. |
| Scratch file systems | Hopper and Edison each have large, local, parallel scratch file systems dedicated to the users of those systems. The global scratch file system, while accessible from all systems, is primarily for use with our midrange systems (Carver, Genepool, PDSF, etc.). The scratch file systems are intended for temporary uses such as storage of checkpoints or application result output. If you need to retain files longer than the purge period, the files should be copied to global project or to HPPS. | Optimized for high-bandwidth, large-block-size access to large files. |
Summary of File System Capacities
| File system | Capacity |
| Global homes | 246 TB |
| Global scratch | 1092 TB |
| Global project | 3965 TB |
| Global projectb | 2620 TB |
| Hopper local scratch1 | 1117 TB |
| Hopper local scratch2 | 1106 TB |
| Edison (Phase 1) local scratch1 | 1.62 PB |
Finding your disk usage
NERSC provides the myquota command which allows you to find your current disk space and inode usage for all file systems.
bash-3.2$ myquota
==== Space (GB) ====
FileSystem Used InDoubt Quota
---------- ---- ------- -----
HOME 10 0 40
GSCRATCH 6 3 20480
==== Inodes ====
FileSystem Used InDoubt Quota
---------- ---- ------- -----
HOME 89468 39 1000000
GSCRATCH 20732 757 2000000
The prjquota command gives you the usage for a project directory.
bash-3.2$ prjquota mpccc
------ Space (GB) ------- ----------- Inode -----------
Project Usage Quota InDoubt Usage Quota InDoubt
-------- ------- ------- ------- ------- ------- -------
mpccc 5818 10240 11 3755653 4000000 1462
Purge Policy
In order to provide equitable access to scarce file system resources, NERSC "purges" certain scratch file systems on a regular basis, as indicated in the above table. Scratch file systems are intended to provide temporary storage on high-performace hardware. Data that is not in active use should be placed in HPSS.
Users are not allowed to run commands (e.g., "touch" scripts) whose sole purpose is to update the time-of-last-access on purgeable files.
Local Scratch on Hopper
Hopper uses Lustre for its local scratch file systems. These file systems are not available on any other computer systems. For more information, please see the File Storage Page for Hopper.
NERSC monitors I/O rates and I/O performance on the local Lustre file systems. Users can see live and historical I/O rates.
Global Home Filesystem
Global home directories (or "global homes") provide a convenient means for a user to have access to source files, input files, configuration files, etc., regardless of the platform the user is logged in to. Wherever possible, you should refer to your home directory using the environment variable $HOME. Read More »
Global Scratch File System
The global scratch file system (or "global scratch") can be accessed by multiple NERSC systems. Its default quota is much larger than the Global Home quota, so it can be used to temporarily store large amounts of data. This file system is periodically purged. Read More »
Project Filesystem
The Project Directory is a global file system available to all NERSC computational systems. It allows groups of NERSC users to store and share data. The Project Directory is available by request. Read More »
Disk Quota Increase Request Form
Use this form to request quota increases. You need to log in using your NIM password. Read More »
Project Directory Request Form
Use this form to request project directory space in the NERSC Global File system (NGF). Here are some guidelines for completing this form: The name, login, e-mail address, and phone number should be those of the requestor filling out this form. The sponsoring repository will typically be that of the requestor, although this is not a requirement. Participating repositories are those that are collaborating with the sponsoring repository. This field may be left blank for a single-repository… Read More »
