NERSCPowering Scientific Discovery Since 1974

Cori File Systems

 Disk Quota Change Request Form

Cori File Systems

The Cori system has 5 different user-accessible file systems; they provide different levels of disk storage and I/O performance.  The table below describes these systems.

File SystemHomeScratchProjectGlobal CommonBurst Buffer
Environment
Variable
$HOME $SCRATCH None.  Must use  /project/projectdirs/ None. Must use /global/ common/software Varies; accessible from within the compute job. 
Description
  • Global home file system shared with other NERSC systems.
  • All NERSC machines mount the same home directory.
  • GPFS filesystem.
  • Where users land when they log into the system.
  • 28 PB of total storage disk space.
  • Available on both Cori and Edison.
  • GPFS global file system mounted on all NERSC systems.
  • GPFS global file system mounted on all NERSC systems.
  • Mounted read-only on the computes to enable client-side caching for faster reads.
  • 1.8PB of SSDs that sit within the HSN. 
  • Users may request an allocation for the duration of their compute job, or for a short-term persistent reservation. 
  • Cray Datawarp product creates an XFS from the SSDs on-demand. 
  • See this page for how to use the Burst Buffer. 
Default Quota
  • 40 GB
  • 1 million inodes^
  • 20 TB*
  • 10 million inodes
  • 1 TB
  • 1 million inodes
  • 10 GB
  • 1 million inodes
  •  50TB; larger reservations available upon request. 
Intended Purpose
  • Shell initializations
  • Not intended for IO intensive applications
  • Running production applications
  • I/O intensive jobs
  • Temporary storage of large files
  • Running production applications
  • Groups needing shared data access
  • Projects running on multiple NERSC machines
  • Installation of software stacks
  • Storing source code
  • Compiling codes
  •  I/O intensive jobs
  • High IOPs jobs 
  • Random or small reads/writes
Peak Performance Low, ~100 MB/sec >700 GB/sec  150 GB/sec    >1.7 TB/sec
Purged? No Yes, files older than 12 weeks since last access are purged. No No  Data is deleted at the end of the reservation. 

*) Cori quota is set on /global/cscratch1. The /global/cscratch1 file system is subject to purging.

^) An inode is a data structure on a filesystem that stores all the information about a file except its name and its actual data. Each file is associated with an inode. Exhaustion of the inodes will prohibit the creation of additional files even if sufficient disk space exists. 

Lustre Scratch Directory

Cori has one scratch file system named /global/cscratch1. The file system has 30 PB disk space and >700 GB/sec IO bandwidth. 

The /global/cscratch1 file system should always be referenced using the environment variable $SCRATCH (which expands to /global/cscratch1/sd/YourUserName). The scratch file system is available from all nodes (login, MOM, and compute nodes) and is tuned for high performance. We recommend that you run your jobs, especially data intensive ones, from the burst buffer or the scratch file system.  

All users have 20 TB of quota for the scratch file system. If your $SCRATCH usage exceeds your quota, you will not be able to submit batch jobs until you reduce your usage.  The batch job submit filter checks the usage of the /global/cscratch1.

The "myquota" command will display your current usage and quota.  NERSC sometimes grants temporary quota increases for legitimate purposes. To apply for such an increase, please use the Disk Quota Increase Form.

The scratch file system is subject to purging. Files in your $SCRATCH directory that are older than 12 weeks (defined by last access time) are removed. Please make sure to back up your important files (e.g. to HPSS).   Instructions for HPSS are here.

To restore deleted or purged files, please refer to Backup/Restore on Home directory or Projects directory. 

Scratch Filesystem Configuration

 $SCRATCH Size PBAggregate Peak Performance# of Disks# IO Servers (OSSs)OSTsFile System SoftwareDisk Array Vendor

/global/cscratch1

30 744 GB/sec  10168 248 248 Lustre Cray

The table shows the Cori scratch file system configuraiton. The scratch file system has 248 OSTs, the lowest I/O layer with which users need to interact. 

Do Not Use /tmp explicitly

WARNING: Do not attempt to explicitly use a file system named /tmp. Some software tools (editors, compilers, etc.) use the location specified by the $TMPDIR environment variable to store temporary files. Additionally, Fortran codes which open files with status="scratch" will write those files into $TMPDIR. On many Unix systems, $TMPDIR is set to /tmp. NERSC has set $TMPDIR to be $SCRATCH. Please do not redefine $TMPDIR!