File Transfers using bbcp

The bbcp utility is a faster alternative to scp . It is available on all NERSC systems except Tesla and Turing. You need to download and install the command on your local machine to use it to transfer files to and from NERSC. Both the download and documentation are available from SLAC.

There is no man page for bbcp on the NERSC systems; use bbcp --help instead.

The basic syntax is quite similar to that of scp:

% bbcp -option(s) sourcespec targetspec 

There are three machines involved in a bbcp file transfer: the agent - the machine on which the bbcp command is run, the source - the machine on which the file(s) to be transferred currently reside, and the target - the machine to which the file(s) will be transferred. The agent is typically the same machine as either the source or the target, but it does not have to be.

NOTE: It is currently necessary to add this option to the bbcp command if either Jacquard or Davinci is the source system:

-S "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp"

-T "ssh -x -a -oFallBackToRsh=no %I -l %U %H /usr/common/usg/bin/bbcp"

Sometimes the "-z" option (use reverse connection protocol, i.e., sink to source) can resolve transfer problems caused by firewall issues.

Note: While the transfer is working, bbcp won't report anything unless you use the -v option.

Peformance of bbcp

We have done several tests of the performance of bbcp between Jacquard and Davinci, involving one 11.5 GByte file. scp always gives pretty much the same level of performance for both puts and gets on any given agent, regardless of which system is the target and which the source, about 20 MBytes/sec on Jacquard and 23 Mbytes/sec on Davinci.

The performance of bbcp with no optimization arguments depends on the source and target systems rather than the agent. Transfers with the same source and target perform the same regardless of the agent on which bbcp is run.

If Jacquard is the source system and Davinci the target, the default bbcp attains 115-120 MBytes/sec. If Davinci is the source system and Jacquard the target, the default bbcp attains 43-46 MBytes/sec.

We investigated two performance options to increase the default performance, -s for number of parallel network streams (the default is four), and -w for size of disk I/O buffers (the default size is 64k).

When the number of streams on the Davinci source Jacquard target transfer was reduced to 1 and the buffer size doubled to 128k (-s 1 -w 128k, the performance of the file transfer more than doubled to almost that of the Jacquard source Davinci target transfer, 115 MBytes/sec.

Neither of these options was able to improve on the default performance of the the Jacquard source Davinci target transfer.