How to compress and transfer folders from one server to another over network

Question

Do we have any option to transfer folder from one server to another, that is faster and compressible and bandwidth limit settable method? The servers run on Linux.

I tried tar + ssh but I couldn't set bandwidth limit to 3 or 4 MBPS likewise.

Any other service does it?

Answer 1

You can use rsync with option --bwlimit= to limit bandwidth and -z to compress file data during the transfer. Using SSH as a transport for example.

Answer 2

bsdtar cf - . | xz | pv -L 3m | ssh host 'cd /there && xz -d | bsdtar xpSf -'

(here using bsdtar as other ones generally don't store all the file metadata).

xz is probably the best compression you can get (but is very CPU intensive), pv for rate limiting (note that it's 3 mega (2²⁰) bytes per second).

Add the --numeric-owner option to the second bsdtar if it's for backup and you want to preserve the uids/gids.

That would be more efficient (in terms of bandwidth and resource on both servers) than using rsync (with xz compression) for a full transfer, but if you need to resume a transfer or sync folders that have files in common, that's where rsync comes handy (but read the rsync man page carefully to select which options you need to preserve the meta data that you want to preserve).

Answer 3

To fine-tune network throughput there are also tools such as cstream or mbuffer.

In addition to -t num to throttle data throughput to num bytes per second, cstream, for example, not only has a -b num option to set the block size used for read/write, but also a -B num option to buffer an input block up to num bytes before writing.

As an alternative to ssh and rsync you could try nc (netcat) or ncat, especially if the data stream is not going to go over the internet. ncat is part of the Nmap Project and has SSL support though.

# netcat + tar + cstream example to throttle network throughput
# requires netcat on both servers
# cf. http://superuser.com/questions/291803/best-way-to-copy-millions-of-files-between-2-servers/
# cstream -T 10 -t 4m </dev/urandom >/dev/null   # test

nc -l -p 2342 | tar -C /target/dir -xzf -                       # destination box
tar -cz /source/dir | cstream -T 30 -t 4m | nc Target_Box 2342  # source box