NERSCPowering Scientific Discovery Since 1974

Project File System


The project file system is a global file system available on all NERSC computational systems. It allows sharing of data between users, systems, and/or (via science gateways) the "outside world".  Default project directories are provided to every MPP project.  Additional project directories can be provided upon request.


No, files in project directories are not subject to purging. 


No managed backups or project directories are done by NERSC. Instead, users can recover files lost in the last seven days by using the snapshot functionality described below on this page.

All NERSC users should back up important files to HPSS on a regular basis.  Ultimately, it is your responsibility to protect from data loss.

Access Control

There must be a project directory administrator associated with each project directory. This user must have a NIM role of PI, PI Proxy, or Project Manager.

Access control for project directories is based on Unix groups. The project directory administrator is responsible for managing membership in the associated group. Instructions for doing this are in the NIM Users Guide for PIs.


Default project directory quotas are 1 TB and 1,000,000 inodes. If your directory needs more than that, fill out the Disk Quota Increase Form.

To check your current usage and quota in a project directory, use the prjquota command, specifying the name of the project directory. For example, if you have access to a project directory named "bigsci":

% prjquota bigsci
------ Space (GB) ------- ----------- Inode -----------
Project Usage Quota InDoubt Usage Quota InDoubt
-------- ------- ------- ------- ------- ------- -------
bigsci 1455 3072 0 307423 500000 20

In the above example, the project directory "bigsci" has used about 1.5 TB of its 3 TB block quota, and about 307000 inodes out of its 500000 inode quota.


The system has a peak aggregate bandwidth of 130 GB/sec bandwidth for streaming I/O, although actual performance for user applications will depend on a variety of factors. Because NGF is a distributed network filesystem, performance typically will be less than that of filesystems that are local to a specific compute platform. This is usually an issue only for applications whose overall performance is sensitive to I/O performance.


Project directories are created in /project/projectdirs. The name of a "default" project directory is the same as its associated MPP repository.  There is also a Unix group with the same name; all members of the repository are also members of the group.  Access to the project directory is controlled by membership in this group.

Occasionally there are cases where the above model is too limiting. For example, some large projects might want a project directory to be accessible by members of multiple repositories. Also, some long-term projects outlive the specific repositories that constitute them. In these cases, a project directory administrator may request the creation of a "designer" project directory with a specific name. This will result in the creation of a new Unix group with that name, consisting solely of the project directory administrator, followed by the creation of the project directory itself. The project directory administrator must then use NIM to add users to the newly-created Unix group (this is a very simple operation). Only these users will be able to access the project directory. If you are a PI or a PI Proxy, you can request a designer project directory in NIM. Search for the repository name you wish this designer project directory to be attached to (the user who can access this directory do not need to be members of this repository, but the directory must be attached to a repository where you are a PI or PI Proxy). Scroll to the bottom of the "Project Information" tab and you will see a link that says "Request a custom project directory".


Project directories use a snapshot capability to provide users a seven-day history of their project directories. This allows you to restore lost files, for example. Each project directory contains a ".snapshots" entry at /project/projectdirs/<repo_name>/.snapshots. This entry is not a typical directory. It is invisible; that is, the "ls" command will not display it, even with the "-a" option. Nor will the "find" command be able to find it. However, its contents are visible. This is best illustrated with an example.

% ls -al
total 24
drwxrwxr-x 2 dpturner dpturner 4096 Dec 15  2013 .
drwxrwxr-x 5 dpturner dpturner 4096 Dec 20  2013 ..
-rw-rw-r-- 1 dpturner dpturner  115 Dec 15  2013 chkenv.pbs
-rw-rw-r-- 1 dpturner dpturner  182 Dec 15  2013 chkenv_mpi.pbs

In the above directory listing, note that ".snapshots" is not visible.  Next, I "accidentally" remove a file:

% rm chkenv.pbs
% ls -al
total 20
drwxrwxr-x 2 dpturner dpturner 4096 Feb  9 16:44 .
drwxrwxr-x 5 dpturner dpturner 4096 Dec 20  2013 ..
-rw-rw-r-- 1 dpturner dpturner  182 Dec 15  2013 chkenv_mpi.pbs

Realizing my error, I look in .snapshots:

% ls -F .snapshots
2015-02-01/  2015-02-02/  2015-02-03/  2015-02-04/  2015-02-05/  2015-02-06/  2015-02-07/  2015-02-08/

Today is February 9 2015, so I look in the entry for February 8:

% date
Mon Feb  9 16:45:18 PST 2015
% ls -al .snapshots/2015-02-08
total 17
drwxrwxr-x  2 dpturner dpturner 4096 Dec 15  2013 .
dr-xr-xr-x 10 root     root     4096 Feb  8 22:03 ..
-rw-rw-r--  1 dpturner dpturner  115 Dec 15  2013 chkenv.pbs
-rw-rw-r--  1 dpturner dpturner  182 Dec 15  2013 chkenv_mpi.pbs

Next, I copy (using the "-p" option to preserve the original file's time stamp) yesterday's version of the missing file back to its original location:

% cp -p .snapshots/2015-02-08/chkenv.pbs .
% ls -al
total 20
drwxrwxr-x 2 dpturner dpturner 4096 Feb  9 16:48 .
drwxrwxr-x 5 dpturner dpturner 4096 Dec 20  2013 ..
-rw-rw-r-- 1 dpturner dpturner  115 Dec 15  2013 chkenv.pbs
-rw-rw-r-- 1 dpturner dpturner  182 Dec 15  2013 chkenv_mpi.pbs

Note that you cannot delete or create anything in a snapshot; you can only copy items out of a snapshot.


Project directories will remain in existence as long as the owning project is active. Projects typically "end" at the end of a NERSC Allocation Year. This happens when the PI chooses not to renew the project, or DOE chooses not to provide an allocation for a renewal request. In either case, the following steps will occur following the termination of the project:

  1. Day -365 - The start of the new Allocation Year and no Project renewal
    The data in the project directory will remain available on the project file system until the start of the next Allocation Year. After that time, we will start the archiving process to move the data off of the project file system into the HPSS tape archive.
  2. Day 0 - The start of the following Allocation Year
    Users of the affected project directory will be notified that the project directory will be archived, and then removed from the file system, in 90 days.
  3. Day 30
    The project directory will become read-only.  Users will still be able to retrieve data from the project directory, but will not be able to store data into the directory.
  4. Day 60
    The full pathname to the project directory will be modified.  This will most likely cause automated scripts to fail.
  5. Day 90
    User access to the directory will be terminated. The directory will then be archived in HPSS, under ownership of the PI, and subsequently removed from the file system.