Secure File Transfers

• 1. Introduction to Transferring Files from and to NERSC
• 2. SCP and SFTP
• 3. File Transfers using GridFTP
• 4. File Transfers using BBCP/BBFTP
• 5. File Transfers using HSI/HTAR

1. Introduction to Transferring Files from and to NERSC

There are a number of ways to transfer files. Small transfers can be done simply with scp or sftp. (Unencrypted ftp connections are not allowed into NERSC.) Larger file transfers will benefit from using more advanced and higher-performing file transfer methods like GridFTP or BBCP. If transferring files in and out of HPSS mass storage, the like HSI and HTAR tools offer good performance.

2. SCP and SFTP

SCP and SFTP use the SSH authentication and encryption methods to transfer files. Because packets are ecrypted and decrypted, performance may not be acceptable for files of many GBs or TBs. For small files, SCP and SFTP are fast and convenient.

An example scp session used to copy a file from Franklin to a local machine:

% scp user_name@franklin.nersc.gov:myfile.txt .
user_name's password: (password is entered here)
% ls -l myfile.txt*
-rw-------    1 user_name  aaa         2368 Aug 11 17:06 myfile.txt

3. File Transfers using GridFTP

NERSC provides a variety of Grid Services, including file transfers using GridFTP. GridFTP achieves good transfer rates and can be very useful for transferring files in non-interactive enviroments.

4. File Transfers using BBCP/BBFTP

BBCP and BBFTP offer the secure authentication of SCP without the data encryption overhead. They are also multi-streamed, easy to use, and can achieve good transfer rates. See File Transfers using bbcp.

5. File Transfers using HSI/HTAR

An often suggested and simple way of transferring data is to use our NERSC HPSS Mass Storage as a staging area. For example, you could first archive your data from a NERSC machine to HPSS. You could do this via HSI or HTAR. Then use HSI or HTAR to download the data to your local machine from HPSS. We have different HSI and HTAR versions ready to be download and easily installed on different platforms. Although it involves an extra step, there is an added benefit of backing up your data, also HPSS is specifically designed with higher performance.