File Storage and I/O
Edison File Systems
The Edison system has 4 different file systems; they provide different levels of disk storage, I/O performance and file permanence. The table below describes the various Edsion file systems.
|File System||Home||Local Scratch||Global Scratch||Project|
||$GSCRATCH||None. Must use
|Peak Performance||Low, ~100 MB/sec||36 GB/sec||15 GB/sec||15 GB/sec|
|Purged?||No||Yes, we will purge. To be implemented on Edison.||Yes, files older than 8 weeks are purged||No|
Edison Phase 1 has a scratch file system named /scratch1. All users have access to this scratch directory; it should always be referenced using the environment variable $SCRATCH (which expands to /scratch1/scratchdirs/YourUsername on Edison). The scratch file system is available from all nodes and is tuned for high performance. We recommend that you run your jobs, especially data intensive ones, from this scratch file system.
There is a 5 TB of quota (space and inode) for the scratch file system. If your $SCRATCH usage exceeds your quota, you will not be able to submit batch jobs until you reduce your usage.
The "myquota" command (with no options) will display your current usage and quota. NERSC sometimes grants temporary quota increases for legitimate purposes. To apply for such an increase, please use the Disk Quota Increase Form.
The scratch file system is subject to purging. Files in your $SCRATCH directory that are older than 12 weeks (defined by last access time) are removed. Please make sure to back up your important files (e.g. to HPSS). Instructions for HPSS are here.
Scratch Filesystem Configuration
|Size TB||Aggregate Peak Performance||# of Disks||# IO Servers (OSSs)||OSTs||File System Software||Disk Array Vendor|
|$SCRATCH||1.62 PB||36 GB/sec||9||18||72||Lustre||Cray|
The /scratch file system has 72 OSTs which is the lowest I/O layer with which users need to interact. When a file is created in /scratch it is "striped" or split across two different OSTs, which is the default. Lustre file systems at other computing centers may set a different default based on their workload. Striping is a technique to increase I/O performance. Instead of writing to a single disk, striping to two disks allows the user to potentially double read and write bandwidth.
Do Not Use /tmp explicitly
WARNING: Do not attempt to explicitly use a file system named /tmp. Your job may fail or be deleted if it writes to /tmp. Some software tools (editors, compilers, etc.) use the location specified by the $TMPDIR environment variable to store temporary files. Additionally, Fortran codes which open files with status="scratch" will write those files into $TMPDIR. On many Unix systems, $TMPDIR is set to /tmp. NERSC has set $TMPDIR to be $SCRATCH. Please do not redefine $TMPDIR!