Once you have successfully generated an HPSS token you can access NERSC's HPSS in the different ways listed below. Hsi and htar are usually the best ways to transfer data in and out of HPSS but other methods are also included. You can click on each access method for more details on how to use them.
|Access Method||When to use this method||Features||Limitations|
|hsi||When a full-featured unix-like interface is desired||high performance (parallel); unix-like user interface; firewall mode||client is specific to HPSS version and might not work at other sites;|
|htar||When you have large collections of smaller (~10MB or less) files||high performance (parallel); creates a tar file directly in HPSS along with an index file; more efficient for large collections of file||same client limitations as HSI; also does not have firewall mode so using on a remote site with a firewall will require modification of firewall rules|
|globus||To do reliable file transfer between grid-enabled sites with your browser||high performance (parallel); robust; no grid credential required - your NERSC login will generate one for you.||requires grid-enabled endpoints; requires GlobusOnline account|
|gridFTP||When you have a grid credential and are transferring to/from a gridFTP-enabled sites||high performance (parallel)||requires grid credential;|
|pftp||When ftp interface is desired along with higher performance||high performance (parallel); FTP-compatible syntax;||limited command set; can't use all HPSS features (e.g., chown, chmod, etc.)|
|ftp||When convenient for smaller transfers||ubiquitous; free utilities such as ncftp, curl and Perl Net::FTP add flexibility for scripting||same limitations as pftp along with low performance (non-parallel)|
HPSS cannot be accessed by SSH.
Access from NERSC platforms
Hsi, htar, and pftp are available by command line on NERSC platforms. HPSS can be accessed interactively (by typing "hsi") and can also be used in batch scripts.
Hsi, htar, pftp, and some ftp clients will look for a file name ".netrc" in your home directory. This will enable automated authentication of access to HPSS. You will not be prompted for a username/password pair. A sample file showing entries is provided below.
The password argument is your HPSS generated authentication token. Ensure that the only permissions enabled on your .netrc file are for the owner (i.e., mode 600 is appropriate).
Users are limited to 15 concurrent sessions. This number can be temporarily reduced if a user is impacting system usability for others
Access from Outside NERSC
HPSS can be accessed from outside NERSC via globus, gridFTP, pftp, and ftp. For moving large amounts of data to / from HPSS to a remote site, we recommend a two-step process. Use globus to transfer the data between sites (at NERSC you can stage the data in your scratch directory) and hsi or htar within NERSC to store/retrieve the data from HPSS. If you have questions about this or if this won't work for your setup, please contact NERSC consulting.
Sleepers are only available using the hsi or pftp clients from NERSC production machines.
When scheduled maintenance or unexpected events necessitate taking HPSS down, "sleepers" are enabled. This causes all jobs attempting to use HPSS to wait. Usually this causes no problems for these jobs as they will resume safely when sleepers are removed. However, users may wish to test for HPSS system availability and take alternate action if sleepers are enabled.
Testing for sleepers can be accomplished by using the "hpss_avail" utility which is available on all NERSC systems. This utility takes a single argument which may be "archive" or "help". The utility returns its result in the predefined shell variable "status" ($? in some shells).