NERSCPowering Scientific Discovery Since 1974

Accessing HPSS

Once you have successfully generated an HPSS token you can access NERSC's HPSS in the different ways listed below.  HSI and HTAR are usually the best ways to transfer data in and out of HPSS but other methods are also included.

Access MethodWhen to use this methodFeaturesLimitations
HSI When a full-featured unix-like interface is desired high performance (parallel); unix-like user interface; firewall mode client is specific to HPSS version and might not work at other sites;
HTAR When you have large collections of smaller (~10MB or less) files high performance (parallel); creates a tar file directly in HPSS along with an index file;  more efficient for large collections of file same client limitations as HSI; also does not have firewall mode so using on a remote site with a firewall will require modification of firewall rules
pftp When ftp interface is desired along with higher performance high performance (parallel); FTP-compatible syntax; limited command set; can't use all HPSS features (e.g., chown, chmod, etc.)
ftp When convenient for smaller transfers ubiquitous; free utilities such as ncftp, curl and Perl Net::FTP add flexibility for scripting same limitations as pftp along with low performance (non-parallel)
gridFTP When you have a grid credential and are transferring to/from a gridFTP-enabled sites high performance (parallel) requires grid credential;
Globus Online To do reliable file transfer between grid-enabled sites with your browser high performance (parallel); robust; no grid credential required - your NERSC login will generate one for you. requires grid-enabled endpoints; requires GlobusOnline account

NERSC HPSS cannot be accessed by SSH.

Access from NERSC platforms

HSI, HTAR and PFTP are available on NERSC platforms.   To access NERSC HPSS with HSI from the command line on a NERSC system you simply type "hsi" followed by the hsi command you want to execute:

hsi command

The HPSS system used for backups is hpss.nersc.gov and should be specified as such when using the HSI utility at NERSC.

HPSS can be accessed interactively and may also be used in batch scripts.

HSI, HTAR, PFTP and some FTP clients will look for a file name ".netrc" in your home directory.  The file should have stanzas for each system that provide your login username and password.  This will enable automated authentication of access to HPSS.  You will not be prompted for a username/password pair.  A sample file showing entries for the HPSS backups system (hpss.nersc.gov) is provided below.

# comment
machine hpss
login franky
password 02S&feVYA!UMR_aG1jaw....Bx22w%%wp((ubVDfIn7FG2W50jSg==

machine hpss.nersc.gov
login franky
password 02S&feVYA!UMR_aG1jaw....Bx22w%%wp((ubVDfIn7FG2W50jSg==

The password argument is your HPSSS generated authentication token.  Ensure that the only permissions enabled on your .netrc file are for the owner (i.e., mode 600 is appropriate).

Access from outside NERSC

The primary HPSS system for users is known as "archive.nersc.gov".  All new users and most current users utilize the archive system for their archival storage need.

The NERSC HPSS system used by NERSC staff for backups is called "hpss.nersc.gov" and is sometimes referred to as the "regent" system.  This system is used primarily for system backups although it does contain some older user data.

archive.nersc.gov can be reached using the hsi and htar utilities and ftp clients.  The HSI and HTAR utilities are available for download and use by NERSC users.  If you do not have HSI/HTAR behind a firewall you may see a significant performance improvement by disabling firewall mode that is on by default.  To disable firewall mode see the README document that comes with your downloaded HSI bundle.