Does compression option -z with rsync speed up backup

Question

In rsync, --compress or -z will compress file data during the transfer.

If I understand correctly, it compresses files before transfer and then decompress them after transfer. Does the time reduced during transfer due to compression outweight the time for compression and decompression?

Does the answer to the question depend on if I backup to an external HDD via usb (2.0 or 3.0), or to a server by SSH over the Internet?

Answer 1

It's a general question. Does compression and decompression at endpoints improve the effective bandwidth of a link?

The effective (perceived) bandwith of a link doing compression and decompression at endpoints is a function of:

1. how fast you can compress (your CPU speed)
2. your network's actual bandwidth

The function is described with this 3D graph, which you might want to consult for your particular situation:

The graph originates with the Compression Tools Compared 2005 article by http://www.linuxjournal.com/ .

Answer 2

If you have a very slow connection (think GPRS) you definitely want to compress you data as much as possible, otherwise your connection will slow things down.

If you have a very slow CPU and a fast connection (like an embedded network device) you usually do not want to compress your data, otherwise your CPU will slow things down.

Answer 3

tl;dr Over slow transfer links, compress, otherwise don't. Below is a compression speed test, a link to a bandwidth conversion tool and some info.

Using compression with rsync will only speed things up if the intermediate link is "slow enough", i.e. if the machine in one end is able to produce a compressed data stream quick enough to saturate the communication link.

So, what's the slowest link at which I should use compression to gain anything?

The following is a very unscientific test, which will show how quickly gzip can produce data, and what that means for whether you should compress your network bulk transfers in general.

The input data will change the outcome of the test greatly. I'm using an uncompressed (!) regular file on my computer which may be representative of the type of data I usually transfer over networks. Using /dev/zero (producing unlimited zeroes) would be misleading as a stream of zeroes would be very easy to compress, and using /dev/random would be misleading for the opposite reason. So instead I use a tar file of my $HOME/local directory, which contains software I've installed in my $HOME. The file is uncompressed in itself, but contains a mix of binary files, small compressed files and source/text files, and would I compress it with default setting for gzip it would shrink by 67% from 64 MiB to 22 MiB.

$ gzip -c local.tar | dd of=/dev/null
43092+4 records in
43093+1 records out
22063854 bytes transferred in 2.819 secs (7825741 bytes/sec)

I do this a few times to get a feeling for what the average might be, and it comes to about 7800000 bytes/s.

Then I use a network bandwidth calculator (sorry, link is dead and I haven't found a good replacement) to see what this converts into. In this particular case, it happens to be just under the capacity of a "100Mb Ethernet" wired link, just faster than a "VDSL Download" internet uplink, slightly quicker than a "802.11[a/g]" wireless link, and somewhere in-between "Bluetooth v3.0" (slower) and "USB 2.0" (faster).

This means that if I'm using compression over anything faster than that, compression will likely slow down the transferring of the file.

rsync might not be using the exact same libraries as gzip to do compression, but the above would give you a bit of a hint at least.

rsync does more than compression though, as you know, and the real speed increase comes from only transferring [bits of] files that have changed.

In my own experience, using compression with rsync has become less and less beneficial over the last 10 years or so, as the bandwidth of the networks have increased (where I am).

For doing incremental backups, I would definitely recommend investigating the --link-dest option (this has nothing to do with what's transferred, only with how things are stored at the target). Also, if you're doing it over SSH, don't use compression if your SSH connection is already compressed, and only compress SSH connections (tunnels etc.) that are over slow links, for the same reasons as above.

Answer 4

Yes, the speed of the connection determines if the speeds things up. It will be overhead only for USB backup, because not the disks inflates the data but the process that writes the data. So the same machine that reads and deflated it, has to inflate and write it too. Rsync is still two processes I think but your memory to hand data from one process to the other is fast enough and the cpu need more time compressing it (while reading it into the same memory that later hands it over :).

Compression only helps when you have a sender and a receiver rsync and some slower network in between. 1Gbit might be already fast enough when you have a local NAS for instance, 10Gbit is already raw SATA speed. So compression is only needed when you have 100Mbit or less connectivity and it only makes sense when the data compressed is compressible.

I am think rsync might notice that it does not run on two machines but one and skips compression but not sure.

Answer 5

Depends on how compressible is your data and the processing power of your source and destination. A full disk backup in my experience will compress to about 30-50% of its original size, so it might be worth to give it a shot. Otherwise, don't bother with compression. It might be worth to test your compression rate with pigz -c <your file> | wc -c and compare the returned size with your original size.

Answer 6

I'd recommend to use --compress --compress-choice=lz4. lz4 has compression speeds of over 500MB/s with fairly good compression ratios, so it will not cause any overhead even on very fast network links, while still getting a large benefit for compressible files.

The only disadvantage is that it only works on new rsync versions, so no debian oldstable.

In general, you can tell when compression is the bottleneck by the rsync process on the sender side taking 100% cpu.

Answer 7

rsync since version 3.2.0 supports more than zlib:

--compress, -z
       turn on compression
--compress-choice=STR, --zc=STR
       choose compression from lz4 (fastest), zstd, zlibx, zlib (slowest), none
--compress-level=NUM, --lz=NUM
       zlib and zstd can be tuned with compression level
       zlib from 1 (lowest) to 9 (highest), default 6
       zstd from -131072 to 22, default 3
       For lz4 compression there are no levels, so the value is always 0.

Compression speeds up rsync if the the compression both manages to produce a smaller amount of data to transfer and does the compressing fast enough not to create a new bottleneck for the transfer. There are now many options to try, most faster than typical Internet connection speeds and some even faster than mass storage speeds (lz4 and zstd can be used in e.g. ZFS)

Answer 8

Here is linked question "Should I use rsync compression over a gigabit LAN?": https://serverfault.com/questions/613709/should-i-use-rsync-compression-over-a-gigabit-lan

Based on practice in common environment with LAN 1Gbps copying from Server to NAS a single file size of 100GB can see that with default rsync compression type the speed varies about 10-15MB/s . And without compression it's 50-100MB/s.

Although there is new compression lz4 added to rsync. Yet to test it myself. https://serverfault.com/a/1089914/537333 (Note it must be supported on both sides i.e. Server and NAS)

Answer 9

rsync over network uses ssh by default. ssh cannot saturate fast networks, it is a long-standing problem, see https://www.psc.edu/hpn-ssh-home/hpn-ssh-faq/ for more information.

That's why compression is a requirement for network transfers with rsync over ssh.

Both rsync and ssh can do compression. Enabling both wastes CPU cycles on the second compression in ssh without achieving any better compression.

rsync skips compressing already compressed files (see --skip-compress option). rsync compresses each file on its own better than ssh compresses the entire uncompressed stream of rsync.

For fastest network file transfer the best practice is to enable rsync compression and explicitly disable ssh compression:

rsync --compress -e'ssh -oCompression=no' ...