NERSCPowering Scientific Discovery Since 1974

pftp

On the NERSC computational systems both the ftp and pftp commands are aliases to a special, high-performance FTP client that will transfer data to and from the NERSC HPSS systems in parallel for high bandwidth transfers.  This pftp client also supports an extended command set not typically available on standard ftp clients.  The pftp client software is not available for download.

Note that the pftp client supports the standard internet File Transfer Protocol and can be used to transfer files to and from remote non-HPSS systems.  In such a situation, the transfers will not be parallel and only the command supported by that remote FTP server will work (the HPSS specific command will not work).

Access Example

To access the archive  storage system using the parallel client from a NERSC computational platform you must specify the system name on the command line:

% pftp archive.nersc.gov

or

% ftp archive.nersc.gov

Likewise, to access the hpss backup system (not all users have accounts on this system) type:

% pftp hpss.nersc.gov

or

% ftp hpss.nersc.gov

As with the other HPSS clients (hsi and htar) pftp will look for your HPSS token in the .netrc file of your local home directory for authentication to the HPSS server.  If you have not generated an HPSS token pftp will prompt you for your NIM password, generate a token and store it in your .netrc file for you.  See the HPSS Passwords page for details.

Extended Command Set

The pftp client has a rich extended command set for use when connected to the NERSC HPSS system.  To see a list of all the commands, type "help" and to get a short description of a specific command, type "help <command>":

%pftp archive
...
ftp> help
Commands may be abbreviated.  Commands are:

!                       macdef                  quit
$                       mchgrp                  quote
account                 mchmod                  recv
append                  mdelete                 reget
ascii                   mdir                    rstatus
autoparallel            mget                    rhelp
bell                    mkdir                   rename
binary                  mlfget                  reset
bye                     mlfput                  restart
case                    mls                     rmdir
cd                      mode                    runique
cdup                    modtime                 safe
chgrp                   mput                    send
chmod                   multinode               setpwidth
clear                   newer                   setpblocksize
close                   nmap                    getremtuningparms
cr                      nlist                   gettuningparms
delete                  ntrans                  setsockbufsize
debug                   open                    setxferbufsize
dir                     p64bit                  site
disconnect              passive                 size
form                    pdata                   status
get                     pdatapush               struct
getprot                 pmover                  system
glob                    private                 sunique
hash                    prompt                  tenex
help                    protect                 term
features                proxy                   time_read
hfeatures               sendport                time_write
idle                    mpget                   tput
image                   mpput                   trace
lcd                     pappend                 type
lfappend                pget                    user
lfget                   pput                    umask
lfput                   put                     verbose
ls                      pwd                     ?

ftp> help chmod
chmod           change file permissions of remote file

A subset of commonly used extended commands are listed in the fillowing table:

Desired Action:Command to Execute:
change group id of a file chgrp groupname hpssfile
change permissions of a file chmod mode hpssfile
store a local file into HPSS in parallel pput localfile [hpssfile]
get an HPSS file in parallel pget hpssfile [localfile]
retrieve multiple files from HPSS in parallel mpget hpssfile1 hpssfile2 ...
store multiple local files into HPSS in parallel mpput localfile1 localfile2 ...
get file from HPSS if it is newer thatn local file newer hpssfile [localfile]

See the pftp man page for details.

Sleepers

The pftp clients on the NERSC computational systems will detect if the HPSS system you are trying to connect to is temporairly unavailable.  If so, the session will be suspended until the system is available again.  When running in a batch script, you may not want the command to suspend.  You can modify your script to detect this condition by following the instructions on the sleepers page.

Usage Examples

The following example shows how you can use pftp in a batch script by placing the commands in a file and having pftp read the command from standard input:

% cat send_files
mkdir testdir
cd testdir
pput file1.tar
pput file2 myfile2
ls -l
quit

% pftp archive < send_files
Using ...
ftp> mkdir testdir
ftp> cd testdir
ftp> pput file1.tar
All Mover transfer information received, beginning data transmission.
ftp> pput file2 myfile2
All Mover transfer information received, beginning data transmission.
ftp> ls -l
-rw-r-----   1 joeuser  ccc         93104 Apr  6 12:17 file1.tar
-rw-r-----   1 joeuser  ccc         59752 Apr  6 12:17 myfile2
ftp> quit

%

Another way to do this is by using HEREIS documents:

% cat putfiles
#!/bin/bash

pftp archive <<EOF
mkdir testdir
cd testdir
pput file1.tar
pput file2 myfile2
ls -l
quit
EOF

% ./putfiles
Using ...
ftp> mkdir testdir
ftp> cd testdir
ftp> pput file1.tar
All Mover transfer information received, beginning data transmission.
ftp> pput file2 myfile2
All Mover transfer information received, beginning data transmission.
ftp> ls -l
-rw-r-----   1 joeuser  ccc         93104 Apr  6 12:22 file1.tar
-rw-r-----   1 joeuser  ccc         59752 Apr  6 12:22 myfile2
ftp> quit

%