NERSCPowering Scientific Discovery Since 1974

Accessing HPSS

Once you have successfully generated an HPSS token you can access NERSC's HPSS in the different ways listed below.  Hsi and htar are usually the best ways to transfer data in and out of HPSS but other methods are also included. You can click on each access method for more details on how to use them.

Access MethodWhen to use this methodFeaturesLimitations
hsi When a full-featured unix-like interface is desired high performance (parallel); unix-like user interface; firewall mode client is specific to HPSS version and might not work at other sites;
htar When you have large collections of smaller (~10MB or less) files high performance (parallel); creates a tar file directly in HPSS along with an index file;  more efficient for large collections of file same client limitations as HSI; also does not have firewall mode so using on a remote site with a firewall will require modification of firewall rules
globus To do reliable file transfer between grid-enabled sites with your browser high performance (parallel); robust; no grid credential required - your NERSC login will generate one for you. requires grid-enabled endpoints; requires GlobusOnline account
gridFTP When you have a grid credential and are transferring to/from a gridFTP-enabled sites high performance (parallel) requires grid credential;
pftp When ftp interface is desired along with higher performance high performance (parallel); FTP-compatible syntax; limited command set; can't use all HPSS features (e.g., chown, chmod, etc.)
ftp When convenient for smaller transfers ubiquitous; free utilities such as ncftp, curl and Perl Net::FTP add flexibility for scripting same limitations as pftp along with low performance (non-parallel)

HPSS cannot be accessed by SSH.

Access from NERSC platforms

Hsi, htar, and pftp are available by command line on NERSC platforms. HPSS can be accessed interactively (by typing "hsi") and can also be used in batch scripts.

Hsi, htar, pftp, and some ftp clients will look for a file name ".netrc" in your home directory. This will enable automated authentication of access to HPSS. You will not be prompted for a username/password pair. A sample file showing entries is provided below.

# comment
machine archive.nersc.gov
login usgtest
password 02V02zwoA3kI5sZ2VysafaFyZABi8K7Tz+iJj4jJ99EdyMjFMZcUyw==

The password argument is your HPSS generated authentication token. Ensure that the only permissions enabled on your .netrc file are for the owner (i.e., mode 600 is appropriate).

Session Limits

Users are limited to 15 concurrent sessions. This number can be temporarily reduced if a user is impacting system usability for others

Access from Outside NERSC

HPSS can be accessed from outside NERSC via globus, gridFTP, pftp, and ftp. For moving large amounts of data to / from HPSS to a remote site, we recommend a two-step process. Use globus to transfer the data between sites (at NERSC you can stage the data in your scratch directory) and hsi or htar within NERSC to store/retrieve the data from HPSS. If you have questions about this or if this won't work for your setup, please contact NERSC consulting.

Sleepers

Sleepers are only available using the hsi or pftp clients from NERSC production machines.

When scheduled maintenance or unexpected events necessitate taking HPSS down, "sleepers" are enabled. This causes all jobs attempting to use HPSS to wait. Usually this causes no problems for these jobs as they will resume safely when sleepers are removed. However, users may wish to test for HPSS system availability and take alternate action if sleepers are enabled.

Testing for sleepers can be accomplished by using the "hpss_avail" utility which is available on all NERSC systems. This utility takes a single argument which may be "archive" or "help". The utility returns its result in the predefined shell variable "status" ($? in some shells).