HSI Manual: Chapter 9

Chapter 9: Tricks, Tips and Helpful Hints

9.1 Using HSI to Backup Filesystems

9.2 Wild Card Tricks

9.3 File Integrity and Backup

9.4 Using Recursion for Project Management

9.5 Frequently Used Requests

9.6 Listing Very Large Directories

9.7 Retrieving Large Files

9.8 Operating System Considerations for One-Liners

9.1 Using HSI to Backup Filesystems

A common method of backing up large filesystems on UNIX workstations is to Use the `tar' utility in conjunction with HSI to create a backup file on HPSS, by piping the output of TAR to HSI. For example, suppose you wish to backup the filesystem "/usr". Enter the following:

cd/ (changes working directory to one level before "usr")
tar cf - usr | HSI save -:usr.tar

To retrieve the file segments and recreate the original directory contents, enter

HSI get -:usr.tar | tar xf -

[ PAGE TOP ]

9.2 Wildcard Tricks

Example: Using ? as a Wildcard Filter

The * (asterisk) is the "wildest" of the wildcards: it can stand for any character in any string of any length. The ? (question mark) wildcard character, however, can stand for a pattern of only one character. It can be used to filter names not only by their content but also by their length.

The following example shows you how to delete files with names of two characters in the directory `~/backup' and any file that has a two-character name and starts with 'f` in the subdirectory /original/94nov.

rm ~/backup/?? /original/94nov/f?

Example: Using a Wildcard filter to refer to a single node

Using wildcards can often reduce the amount of typing in a command when only a single nodename is desired; for example, assume that the following files exist on HPSS:

TestData-RunNumber3-10July1993
TestData-RunNumber4-10July1993
TestData-RunNumber5-10July1993

Any of the following commands can be used to refer to the second of these files:

get datafile : TestData-RunNumber4-10July1993
get datafile: *4*93
get datafile : *4-*

[ PAGE TOP ]

9.3 File Integrity and Backup

HPSS stores data on a hierarchy of disk and tape storage devices, and manages the movement of files within this hierarchy on the basis of how recently files are read or written. However, no "backups" of data are made. It is your responsibility to safeguard your critical files!

[ PAGE TOP ]

9.4 Using Recursion for Project Management

You can use the recursion features available in HSI to simplify the management of projects. The GET, LS, SAVE, STORE and RM commands all support recursion by specifying `-R'. For example, suppose your project consists of the directory tree:


MyProject          (root)
   Source
      Include
         h1.h
         h2.h
         h3.h
      CFiles
         file1.c
         file2.c
         file3.c
   Binaries
      Programs
         myproject
      Relocatables
         file1.o
         file2.o
         file3.o

then the command

put -R MyProject

can be used to save the directory "MyProject" and all of its files and subdirectories, automatically creating subdirectories as required.

Conversely, the command

get -R MyProject

can be used to fetch the directory and all of its files and subdirectories, automatically creating local subdirectories as required.

Note: The use of recursive requests can cause problems, because the resources needed to satisfy such requests cannot always be determined in advance. For this reason, caution is advised in using it to store or retrieve very large directory structures or file collections; this is especially true in situations where failure or incomplete transfer can cause other problems, such as in the staging of crucial files to or from expensive batch jobs.

[ PAGE TOP ]

9.5 Frequently Used Requests

If you have a set of HSI keywords that you often use to establish your HPSS keyset environment, you should consider setting the values and keeping them as a keyset. See KEEP Command and ADOPT Command in Chapter 7.

Note: This method is safer, in general, than using shell or environment variables to save path values, due to potential confusion over when and by what agencies shell or environment variables get resolved.

[ PAGE TOP ]

9.6 Listing Very Large Directories

It may sometimes be necessary to list HPSS directories containing a very large number of nodes. Normally, the entire set of directory entry names is first read into memory and sorted to allow for listing by columns, rows, etc. If this fails due to memory exhaustion or it simply takes too long, the "-O" option may be specified on the LS command to cause the directory entries to be listed in the order that they are read, and not attempt to sort them. This is also the fastest method of listing a directory.

[ PAGE TOP ]

9.7 Retrieving Large Files

Users having trouble retrieving files over 25 GB in size may wish to use the -S option for the get or mget commands. The -S option was added to allow users to disable staging files from tape, specifically for this type of situation. The syntax is:

get -S [other options] local : hpss [local : hpss ...]

The usage message that is displayed by typing "get" with no options, will also explain the -S flag.

[ PAGE TOP ]

9.8 Operating System Considerations for One-Liners

Special considerations apply to HSI one-liner requests which contain certain characters, including wild card characters. Some operating systems and some command languages have defined certain characters, such as the asterisk (*) and the double quote("), to have special meanings, so procedures should be followed if such characters are to be passed to HSI without local interpretation. UNIX-based systems support wildcards, but in varying ways, depending upon the command shell in use, so you should know the effect these systems will have on your results.

If wildcard pattern-matching is to be done by HSI only, the wildcard path in a one-liner should always be enclosed in single or double quotes or the wildcard characters should be preceded by the escape character (\). When wild path is "quoted" or "escaped" in this way, local wildcard processing will not be invoked for the quoted string; the string will be passed without substitution to HSI.

Conversely, an "unprotected" path will be processed by the shell and the wildcard pattern will be resolved in terms of local files, and then passed on to HPSS. For example, to preserve a wildpath so that only HSI will process it enter

hsi get 'fil*'

These techniques do not apply if HSI is used in interactive mode instead of single-line mode. They also do not apply to HSI commands read from IN files.

Note: Redirecting Standard Input (STDIN) by way of a here-doc may also prevent the shell from interpreting and/or expanding wild-card characters. This technique is useful in scripts, and where HSI one-liners will not suffice, and it is documented in the man pages for the Unix Shell, csh.

Be aware when using locally supported wildcard capabilities that HPSS restrictions on legal file name characters and legal file name length will still apply. You should be wary when using wildcards in HSI paths in command files.

Note: The ordering of characters differs among operating systems, and does not necessarily follow the ASCII collating sequence. In general, HSI will try to resolve the meanings of "special" characters, before it assembles paths. This means that

wild card characters, such as "*" and "?"

characters that have reserved meanings within HSI, such as " ' " (single-quote) and "-" (hypen)

bracket pairs "[ ]" and "{ }" (often denoting character sets and intervals)

will be handled before the alphanumeric and other legal path characters. Therefore, the results of commands containing wild card characters may differ across operating systems.

See Operating System Considerations for UNIX-Based Systems in Section 4 for more operating system-specific information.

[ PAGE TOP ]