File Storage and I/O
Disk Quota Change Request Form
Edison File Systems
The Edison system has 5 different file systems mounted which provide different levels of disk storage, I/O performance and file permanence. The table below describes the various Edsion file systems:
| File System | Home | Local Scratch | Global Scratch | Project |
|---|---|---|---|---|
| Environment Variable Definition |
$HOME | $SCRATCH |
$GSCRATCH | None. Must use /project/projectdirs/ |
| Description |
|
|
|
|
| Default Quota |
|
|
|
|
| Intended Purpose |
|
|
|
|
| Peak Performance | Low, ~100 MB/sec | 36 GB/sec | 15 GB/sec | 15 GB/sec |
| Purged? | No | Yes, we will purge. To be implemented on Edison. | Yes, files older than 8 weeks are purged | No |
Scratch Directory
Edison P1 has a scratch file systems named /scratch1. Each user has access to the scratch directory that should always be referenced using the environment variable $SCRATCH (the environment variable SCRATCH expands to /scratch1/scratchdirs/YourUsername on Edison). The scratch file system is available from all nodes and is tuned for high performance. You are recommended to run your jobs, especially for the data intensive ones, on this scratch file system.
There is a 5 TB of quota (space and inode) for the scratch file system. If your usage of the $SCRATCH exceeds your quota, you will not be able to submit batch jobs until you reduce your usage.
The "myquota" command (with no options) will display your current usage and quota. NERSC sometimes grants temporary quota increases for legitimate purposes. To apply for such an increase, please use the Disk Quota Increase Form.
The scratch file system is subject to purge. Files in your $SCRATCH directory that are older than 12 weeks (defined by last access time) are removed. Please make sure to back up your important files (e.g. to HPSS). Instructions for HPSS are here.
Scratch Filesystem Configuration
| Size TB | Aggregate Peak Performance | # of Disks | # IO Servers (OSSs) | OSTs | File System Software | Disk Array Vendor | |
|---|---|---|---|---|---|---|---|
| $SCRATCH | 1.62 PB | 36 GB/sec | 9 | 18 | 72 | Lustre | Cray |
The /scratch file system has 72 OSTs which is the lowest layer with which users need to interact. When a file is created in /scratch it is "striped" or split across two different OSTs, which is the default. Lustre file systems at other computing centers may set a different default based on their workload. Striping is a technique to increase I/O performance. Instead of writing to a single disk, striping to two disks allows the user to potentially double read and write bandwidth.
Do Not Use /tmp explicitly
WARNING: Do not attempt to explicitly use a file system named /tmp. Your job may fail or be deleted if it writes to /tmp. Some software tools (editors, compilers, etc.) use the location specified by the $TMPDIR environment variable to store temporary files. Additionally, Fortran codes which open files with status="scratch" will write those files into $TMPDIR. On many Unix systems, $TMPDIR is set to /tmp. NERSC has set $TMPDIR to be $SCRATCH. Please do not redefine $TMPDIR!
