BBCP is a file transfer tool, similar to scp and ftp, which is recommended to transfer files at high speed. Indeed, scp and ftp cannot reach the full speed available to most modern supercomputers. BBCP fixes some of the problems of the two aforementionned tools and is recommended to transfer large files toward or from our sites.
bbcp shares the syntax of scp and is therefore rather straightforward to use.
The basic syntax is as follow:
[name@server $] bbcp [options] file [user@]server:relative_path
This command will transfer a file from the local host to the specified server. Note that the path is relative to the $HOME directory on the remote host. You do not need to specify the user if it is the same as the one on the local host.
Like with scp, you will need to input your credentials to start the transfer.
If you want to download a file from a remote host to your local host, use the following syntax
[name@server $] bbcp [options] [user@]remote_host:path/to/file local/relative/path
For example, if you want to download the results.dat file, located on server.calculquebec.ca in your remote $HOME directory to your local $HOME directory, you could use the command
[nom@serveur $] bbcp email@example.com:results.dat /home/user/
Reaching higher speed (-s et -w)
To reach optimal transfer speed, you must use the -s et -w options.
The -s option sets the number of streams that are opened at the same time, and the -w option sets the size, in kilobytes, of the TCP window to use.
Best speed are typically obtained with 10 to 12 concurrent streams and a window size of 1200 kb. The options we recommend are therefore
[name@server $] bbcp -s 12 -w 1200K file firstname.lastname@example.org:/path/to/target
Force existing files overwriting (-f)
The default behavior of BBCP is to stop transfer if the target file already exists on the target location. If you want to overwrite an existing file, use the -f option:
[name@server $] bbcp -f file email@example.com:path/to/file
Progress report (-P number)
With the -P option, BBCP will show a progress report every number seconds. For example
[name@server $] bbcp -P 2 file firstname.lastname@example.org:path/to/target
will show a progress report every two seconds. Note: Do not use rates smaller than 1.
Reversed connection (-z)
If you need to communicate with a server that is blocked by a firewall, you can attempt to reverse the connection with the -z option. This will request that the remote server initiates the connection with your local host. For example, with the command
[name@server $] bbcp -z file email@example.com:/home/user/
the host server.calculquebec.ca will start the transfer, which will go around its own firewall.
Resume a partial transfer (-a and -k)
If a transfer fails, you can resume it from the point where it ended. To do so, two options are available: -a and -k.
The -a option continues the transfer of partially transfered files. To use -a, the -k option must have been used for the initial transfer.
With the -k option, BBCP keeps partially transfered files, allowing to resume the transfer at a later time. The default behavior without this option is to delete partially transfered files.
Therefore, if you use the first of the following commands, you will be able to use the second one if the first transfer fails.
[nom@serveur $] bbcp -k file firstname.lastname@example.org:/home/user/ [nom@serveur $] bbcp -a -k file email@example.com:/home/user/
Recursive copy (-R)
The -R option allows to recursively copy a directory and its content.
For more details, you may read the official BBCP documentation.