NERSCPowering Scientific Discovery Since 1974

Setting up File Permissions

Recommendations for Setting up "Shared" Directories

This section shows the recommended commands for setting up shared directories.  Please read the full page to get an understanding of what these commands are doing.  These recommendations are based on the common use-case at the JGI for user/group read/write and global read access.

Creating a New Shared Directory

dmj@genepool04:~$ umask 002                    # or have this set in .bashrc.ext
dmj@genepool04:~$ mkdir sharedDirectory        # make shared directory
dmj@genepool04:~$ chgrp genome sharedDirectory # change group ownership
dmj@genepool04:~$ chmod g+s sharedDirectory    # set setgid bit
dmj@genepool04:~$ ls -ld sharedDirectory       # check permissions
drwxrwsr-x 2 dmj genome 4096 Nov 30 14:34 sharedDirectory
dmj@genepool04:~$ 

Retrofitting an Existing Directory Structure

dmj@genepool04:~$ chgrp -R genome existingDirectory
dmj@genepool04:~$ chmod -R g+rwX existingDirectory
dmj@genepool04:~$ find existingDirectory -type d -exec chmod g+s {}\;

Note that unless you own all files in the existing directory structure, these commands are likely to have some errors for the files you don't own.  Please ask your group leader to file a ticket at http://help.nersc.gov to request that an administrator help with this operation.

Setting file and group permissions

Many people at the JGI want to make sure that their files are available to read and manipulated by other people in their groups.  Every user at the JGI is part of the unix filegroup "genome", but there are a variety of other groups for group-specific work, e.g. compgen, rqc_user, gat, gbp, etc.  This page will show how to view and change file permissions, and will also show how to setup permissions to properly share data with your colleagues.

The key to making files accessible is to others is to manipulate the file permissions so that your colleagues are able to read (and sometimes write to) the files.  Details on how to do this for a file or directory can be found in "Basic UNIX File Permissions" and "Making Files and Directories Accessible".  An important concept for the JGI is "group" ownership and permissions to files and directories.  To properly setup directories for sharing data with colleagues, see the 'Making Group Settings "Stick"' section.

Basic UNIX File Permissions

Each file and directory has "permissions" associated with them; each file or directory has both user and group ownership.  If you create a file or directory, that file will be owned by you, and, by default, will be owned by your default group.  To see owership and permissions of a file or directory, use the ls command:

dmj@genepool04:~$ mkdir testDirectory          # create a directory
dmj@genepool04:~$ touch testDirectory/file1    # create a file
dmj@genepool04:~$ ls -l testDirectory          # see files in testDirectory
total 0
-rw------- 1 dmj dmj 0 Nov 30 13:17 file1
dmj@genepool04:~$ ls -ld testDirectory         # Note the "-ld" flags
drwx------ 2 dmj dmj 4096 Nov 30 13:17 testDirectory
dmj@genepool04:~$ 

The output of "ls -l" shows much more information about a file or directory than "ls".  The first column shows the file permssions -- more on that in a minute.  The third and fourth columns show the user and group ownership of the file (both "dmj" in this case).  The file permisions column has ten characters of information contained within it (often called file permission "bits").  The first bit shows the file type ("-" for file, "d" for directory, "l" for link).  The remaining 9 bits represent the file permissions for user, group, and "others", three bits for each, most simply described as one for read ("r"), one for write ("w"), and one for execute ("x").  A dash ("-") is displayed when a permission is disallowed.  So, in the example above, where the user bits are "rw-" and the group and other bits are "---", user dmj can read and write to file1, but not group members or others.

A note about directories: the execute bit must be set in order for a user to traverse through a directory.  The read bit must be set for a user to read the contents of a directory; thus to allow a user to access files in a directory, both the read ("r"), and execute ("x") bits must be set for the appropriate user, group, or other permissions set.

Making Files and Directories Accessible

To make your file and directories accessible, you need to first evaluate:

  1. Who should be able access the files?  (people belonging to a particular filegroup? all people?)
  2. How much access should they have?  No access?  Read? Write?

Since you are the only user who has access to your personal filegroup (named the same as your username), there is little value in setting the group permission bits unless you also change the group to a more shared filegroup.  To change group ownership of a file or directory, use the chgrp command:

dmj@genepool04:~$ chgrp genome testDirectory    # change group for a single file/dir
dmj@genepool04:~$ chgrp -R genome testDirectory # recursively change groups
dmj@genepool04:~$ ls -l testDirectory
total 0
-rw------- 1 dmj genome 0 Nov 30 13:17 file1    # now the group is genome!
dmj@genepool04:~$ 

 Now, to make files group accessible, use chmod:

dmj@genepool04:~$ chmod -R g+rX testDirectory # recursively set r/x bits
dmj@genepool04:~$ ls -l testDirectory
total 0
-rw-r----- 1 dmj genome 0 Nov 30 13:17 file1  # group readable
dmj@genepool04:~$ ls -ld testDirectory
drwxr-x--- 2 dmj genome 4096 Nov 30 13:17 testDirectory # group read/exec
dmj@genepool04:~$ 

The "-R" flag for chgrp and chmod makes those commands work recursively, meaning it will traverse through all files and directories contained within the specified path.  The "g+rX" means add the read and execute bits for group permissions.  You can set your user permisions with "u+..." and other with "o+...".  A short-cut to set all three groups at once is to use "a" for all.  E.g. chmod a+rwx sets the read, write, and execute bits for user, group, and others.  In the example above, I also used a capital "X"; the capitals instruct chmod to copy the relavent bits from user.  This way,  chmod g+X ... means exec bits will only get set if they are already set for the user.  This is extremely useful when recursely setting permissions to make sure that execute bits are only set on directories and executable files.  For more information, please see the chmod man page (man chmod).

Making everything accessible by default

The umask command can be used to change the file permissions of files and directories you create.  If you enter "umask 002", then all files and directories will be created with user and group read and write permissions (+execute for directories); also the files will be world readable.  The meaning of the numbers after umask are steeped in the octal representation of the file permission bits, and if you are interested, you can learn more about that on http://en.wikipedia.org/wiki/Filesystem_permissions#Octal_notation

dmj@genepool04:~$ umask 002                 # set umask
dmj@genepool04:~$ touch testDirectory/file2 # create file with new umask
dmj@genepool04:~$ ls -l testDirectory/
total 0
-rw-r----- 1 dmj genome 0 Nov 30 13:17 file1
-rw-rw-r-- 1 dmj dmj    0 Nov 30 14:13 file2 #u/g r/w, other read
dmj@genepool04:~$ 

To make this new umask permanent, you can add "umask 002" to your .bashrc.ext or .tcshrc.ext login file (depending on if you use bash or tcsh respectively).  Note that "umask 007" limits permissions to user/group permissions and would block "others".

Make Group Settings "Stick"

For directories where many people will be creating files, and all want to make sure that the group-ownership permissions remain constant (e.g. if you want the "genome" group to own everything that is ever created within a particular path).  You can use the group setgid-bit for this purpose:

dmj@genepool04:~$ chmod g+s testDirectory   # add group setgid bit
dmj@genepool04:~$ touch testDirectory/file3 # create a new file
dmj@genepool04:~$ ls -ld testDirectory      # check directory permissions
drwxr-s--- 2 dmj genome 4096 Nov 30 14:26 testDirectory  # note the "s"
dmj@genepool04:~$ ls -l testDirectory       # check file permissions
total 0
-rw-r----- 1 dmj genome 0 Nov 30 13:17 file1 # file1 was chgrp'd earlier
-rw-rw-r-- 1 dmj dmj    0 Nov 30 14:13 file2 # file2 wasn't
-rw-rw-r-- 1 dmj genome 0 Nov 30 14:26 file3 # file3 has genome because of the setgid bit
dmj@genepool04:~$ 

 In this example, I modified the testDirectory to have the group setgid bit.  This means that all files and directories created in that directory will carry new group ownership I set earlier.  Furthermore, directories created within will also have the setgid bit set, so all new files in those directories will also carry over the group ownership.  This is a very effective way of make entire file-hierarchies accessible to groups.