32

In rsync, -z will compress file data during the transfer.

If I understand correctly, -z compress files before transfer and then decompress them after transfer. Does the time reduced during transfer due to compression outweight the time for compression and decompression?

Does the answer to the question depend on if I backup to an external hdd via usb (2.0 or 3.0), or to a server by ssh over the internet?

  • Also remember if the compressed file does not differ much in size from the original file, this could be a huge overhead. – heemayl Mar 7 '15 at 11:24
  • 1
    To elaborate on what heemayl says, if the content is largely material that is already in a compressed format (jpeg, mpeg, distro packages, etc) compression is much less effective. I notice in man rsync that there is in fact a list of file suffixes that will not be compressed even with -z (see --skip-compress). – goldilocks Mar 7 '15 at 12:09
41

It's a general question. Does compression and decompression at endpoints improve the effective bandwidth of a link?

The effective (perceived) bandwith of a link doing compression and decompression at endpoints is a function of:

  1. how fast you can compress (your CPU speed)
  2. your network's actual bandwidth

The function is described with this 3D graph, which you might want to consult for your particular situation:

enter image description here

The graph originates with the Compression Tools Compared 2005 article by http://www.linuxjournal.com/ .

12

If you have a very slow connection (think GPRS) you definitely want to compress you data as much as possible, otherwise your connection will slow things down.

If you have a very slow CPU and a fast connection (like an embedded network device) you usually do not want to compress your data, otherwise your CPU will slow things down.

2

Yes, the speed of the connection determines if the speeds things up. It will be overhead only for USB backup, because not the disks inflates the data but the process that writes the data. So the same machine that reads and deflated it, has to inflate and write it too. Rsync is still two processes I think but your memory to hand data from one process to the other is fast enough and the cpu need more time compressing it (while reading it into the same memory that later hands it over :).

Compression only helps when you have a sender and a receiver rsync and some slower network in between. 1Gbit might be already fast enough when you have a local NAS for instance, 10Gbit is already raw SATA speed. So compression is only needed when you have 100Mbit or less connectivity and it only makes sense when the data compressed is compressible.

I am think rsync might notice that it does not run on two machines but one and skips compression but not sure.

2

Depends on how compressible is your data and the processing power of your source and destination. A full disk backup in my experience will compress to about 30-50% of its original size, so it might be worth to give it a shot. Otherwise, don't bother with compression. It might be worth to test your compression rate with pigz -c <your file> | wc -c and compare the returned size with your original size.

1

tl;dr Over slow transfer links, compress, otherwise don't. Below is a compression speed test, a link to a bandwidth conversion tool and some info.

Using compression with rsync will only speed things up if the intermediate link is "slow enough", i.e. if the machine in one end is able to produce a compressed data stream quick enough to saturate the communication link.

So, what's the slowest link at which I should use compression to gain anything?

The following is a very unscientific test, which will show how quickly gzip can produce data, and what that means for whether you should compress your network bulk transfers in general.

The input data will change the outcome of the test greatly. I'm using an uncompressed (!) regular file on my computer which may be representative of the type of data I usually transfer over networks. Using /dev/zero (producing unlimited zeroes) would be misleading as a stream of zeroes would be very easy to compress, and using /dev/random would be misleading for the opposite reason. So instead I use a tar file of my $HOME/local directory, which contains software I've installed in my $HOME. The file is uncompressed in itself, but contains a mix of binary files, small compressed files and source/text files, and would I compress it with default setting for gzip it would shrink by 67% from 64 MiB to 22 MiB.

$ gzip -c local.tar | dd of=/dev/null
43092+4 records in
43093+1 records out
22063854 bytes transferred in 2.819 secs (7825741 bytes/sec)

I do this a few times to get a feeling for what the average might be, and it comes to about 7800000 bytes/s.

Then I use a network bandwidth calculator to see what this converts into. In this particular case, it happens to be just under the capacity of a "100Mb Ethernet" wired link, just faster than a "VDSL Download" internet uplink, slightly quicker than a "802.11[a/g]" wireless link, and somewhere in-between "Bluetooth v3.0" (slower) and "USB 2.0" (faster).

This means that if I'm using compression over anything faster than that, compression will likely slow down the transferring of the file.

rsync might not be using the exact same libraries as gzip to do compression, but the above would give you a bit of a hint at least.

rsync does more than compression though, as you know, and the real speed increase comes from only transferring [bits of] files that have changed.

In my own experience, using compression with rsync has become less and less beneficial over the last 10 years or so, as the bandwidth of the networks have increased (where I am).

For doing incremental backups, I would definitely recommend investigating the --link-dest option (this has nothing to do with what's transferred, only with how things are stored at the target). Also, if you're doing it over SSH, don't use compression if your SSH connection is already compressed, and only compress SSH connections (tunnels etc.) that are over slow links, for the same reasons as above.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.