Once you have successfully generated an HPSS token you can access NERSC's HPSS in the different ways listed below. HSI and HTAR are usually the best ways to transfer data in and out of HPSS but other methods are also included.
|Access Method||When to use this method||Features||Limitations|
|HSI||When a full-featured unix-like interface is desired||high performance (parallel); unix-like user interface; firewall mode||client is specific to HPSS version and might not work at other sites;|
|HTAR||When you have large collections of smaller (~10MB or less) files||high performance (parallel); creates a tar file directly in HPSS along with an index file; more efficient for large collections of file||same client limitations as HSI; also does not have firewall mode so using on a remote site with a firewall will require modification of firewall rules|
|pftp||When ftp interface is desired along with higher performance||high performance (parallel); FTP-compatible syntax;||limited command set; can't use all HPSS features (e.g., chown, chmod, etc.)|
|ftp||When convenient for smaller transfers||ubiquitous; free utilities such as ncftp, curl and Perl Net::FTP add flexibility for scripting||same limitations as pftp along with low performance (non-parallel)|
|gridFTP||When you have a grid credential and are transferring to/from a gridFTP-enabled sites||high performance (parallel)||requires grid credential;|
|Globus Online||To do reliable file transfer between grid-enabled sites with your browser||high performance (parallel); robust; no grid credential required - your NERSC login will generate one for you.||requires grid-enabled endpoints; requires GlobusOnline account|
NERSC HPSS cannot be accessed by SSH.
Access from NERSC platforms
HSI, HTAR and PFTP are available on NERSC platforms. To access NERSC HPSS with HSI from the command line on a NERSC system you simply type "hsi" followed by the hsi command you want to execute:
The HPSS system used for backups is hpss.nersc.gov and should be specified as such when using the HSI utility at NERSC.
HPSS can be accessed interactively and may also be used in batch scripts.
HSI, HTAR, PFTP and some FTP clients will look for a file name ".netrc" in your home directory. The file should have stanzas for each system that provide your login username and password. This will enable automated authentication of access to HPSS. You will not be prompted for a username/password pair. A sample file showing entries for the HPSS backups system (hpss.nersc.gov) is provided below.
The password argument is your HPSSS generated authentication token. Ensure that the only permissions enabled on your .netrc file are for the owner (i.e., mode 600 is appropriate).
Access from outside NERSC
The primary HPSS system for users is known as "archive.nersc.gov". All new users and most current users utilize the archive system for their archival storage need.
The NERSC HPSS system used by NERSC staff for backups is called "hpss.nersc.gov" and is sometimes referred to as the "regent" system. This system is used primarily for system backups although it does contain some older user data.
archive.nersc.gov can be reached using the hsi and htar utilities and ftp clients. The HSI and HTAR utilities are available for download and use by NERSC users. If you do not have HSI/HTAR behind a firewall you may see a significant performance improvement by disabling firewall mode that is on by default. To disable firewall mode see the README document that comes with your downloaded HSI bundle.