87

In rsync, --compress or -z will compress file data during the transfer.

If I understand correctly, it compresses files before transfer and then decompress them after transfer. Does the time reduced during transfer due to compression outweight the time for compression and decompression?

Does the answer to the question depend on if I backup to an external HDD via usb (2.0 or 3.0), or to a server by SSH over the Internet?

6
  • 1
    Also remember if the compressed file does not differ much in size from the original file, this could be a huge overhead.
    – heemayl
    Commented Mar 7, 2015 at 11:24
  • 3
    To elaborate on what heemayl says, if the content is largely material that is already in a compressed format (jpeg, mpeg, distro packages, etc) compression is much less effective. I notice in man rsync that there is in fact a list of file suffixes that will not be compressed even with -z (see --skip-compress).
    – goldilocks
    Commented Mar 7, 2015 at 12:09
  • 2
    I found it quite optimal to use rsync -z --compress-level=1 over a 5MB/s or faster link. Higher compression levels max-out on a single cpu thread of rsync.
    – macieksk
    Commented Jan 25, 2020 at 13:57
  • 3
    For anyone who is wondering whether -z helps when copying a file from a laptop to an external drive connected to the laptop: No it won't help. It will only make it worse. See this answer for explaination. Commented Mar 28, 2020 at 19:27
  • 1
    See also Should I use rsync compression over a gigabit LAN? Commented Jul 12, 2022 at 18:32

9 Answers 9

104

It's a general question. Does compression and decompression at endpoints improve the effective bandwidth of a link?

The effective (perceived) bandwith of a link doing compression and decompression at endpoints is a function of:

  1. how fast you can compress (your CPU speed)
  2. your network's actual bandwidth

The function is described with this 3D graph, which you might want to consult for your particular situation:

enter image description here

The graph originates with the Compression Tools Compared 2005 article by http://www.linuxjournal.com/ .

8
  • 19
    Your type of data is also a major factor (factor #3 missing from the list). The linked article uses a typical mix of data. Yours might not be typical. If you are syncing 100% ZIP files (or any pre-compressed data) you probably don't want compression. If you are syncing 100% text files you might be faster to compress even if your network is fast and your CPU is slowish. Weigh all 3 factors. Commented Aug 23, 2019 at 2:07
  • 2
    This is a great answer. That chart helps me thinking about this in ways I hadn't before. Commented Jun 8, 2020 at 17:33
  • is this statement correct? -> "I use a 56k modem and have c64 cpu, therefore I should activate the rsync -z flag."
    – kiltek
    Commented Dec 8, 2022 at 16:05
  • @kiltek Commodore 64 chips are really, really slow though.. maybe try benchmarking and report back here? I, too, would like to know. Commented Apr 25, 2023 at 4:28
  • 2
    @Lou "It really depends" and it's not that hard to grok. Compression takes CPU time. CPU time that could have been spent sending uncompressed data. Spending that CPU time on compression is basically only worth it if both the link is slower-ish and your CPU's not too slow. Commented Nov 23, 2023 at 22:17
21

If you have a very slow connection (think GPRS) you definitely want to compress you data as much as possible, otherwise your connection will slow things down.

If you have a very slow CPU and a fast connection (like an embedded network device) you usually do not want to compress your data, otherwise your CPU will slow things down.

15

tl;dr Over slow transfer links, compress, otherwise don't. Below is a compression speed test, a link to a bandwidth conversion tool and some info.

Using compression with rsync will only speed things up if the intermediate link is "slow enough", i.e. if the machine in one end is able to produce a compressed data stream quick enough to saturate the communication link.

So, what's the slowest link at which I should use compression to gain anything?

The following is a very unscientific test, which will show how quickly gzip can produce data, and what that means for whether you should compress your network bulk transfers in general.

The input data will change the outcome of the test greatly. I'm using an uncompressed (!) regular file on my computer which may be representative of the type of data I usually transfer over networks. Using /dev/zero (producing unlimited zeroes) would be misleading as a stream of zeroes would be very easy to compress, and using /dev/random would be misleading for the opposite reason. So instead I use a tar file of my $HOME/local directory, which contains software I've installed in my $HOME. The file is uncompressed in itself, but contains a mix of binary files, small compressed files and source/text files, and would I compress it with default setting for gzip it would shrink by 67% from 64 MiB to 22 MiB.

$ gzip -c local.tar | dd of=/dev/null
43092+4 records in
43093+1 records out
22063854 bytes transferred in 2.819 secs (7825741 bytes/sec)

I do this a few times to get a feeling for what the average might be, and it comes to about 7800000 bytes/s.

Then I use a network bandwidth calculator (sorry, link is dead and I haven't found a good replacement) to see what this converts into. In this particular case, it happens to be just under the capacity of a "100Mb Ethernet" wired link, just faster than a "VDSL Download" internet uplink, slightly quicker than a "802.11[a/g]" wireless link, and somewhere in-between "Bluetooth v3.0" (slower) and "USB 2.0" (faster).

This means that if I'm using compression over anything faster than that, compression will likely slow down the transferring of the file.

rsync might not be using the exact same libraries as gzip to do compression, but the above would give you a bit of a hint at least.

rsync does more than compression though, as you know, and the real speed increase comes from only transferring [bits of] files that have changed.

In my own experience, using compression with rsync has become less and less beneficial over the last 10 years or so, as the bandwidth of the networks have increased (where I am).

For doing incremental backups, I would definitely recommend investigating the --link-dest option (this has nothing to do with what's transferred, only with how things are stored at the target). Also, if you're doing it over SSH, don't use compression if your SSH connection is already compressed, and only compress SSH connections (tunnels etc.) that are over slow links, for the same reasons as above.

2
  • rsync over network uses ssh by default. ssh cannot saturate fast networks, it is a long-standing problem. That's why compression is a requirement for network transfers with rsync over ssh. See unix.stackexchange.com/a/746573/29923 Commented May 21, 2023 at 23:32
  • @MaximEgorushkin The point I make in my answer is that if your system can't compress data fast enough (or if the destination system can't decompress the data fast enough), then compression will slow down the transfer. Likewise, compression will make no difference if the disks on either system are too busy and can't offer the throughput needed to read or write the data. Your argument is valid if you assume CPU and disk are not an issue and that these can handle the processing of data. Using a modified SSH implementation may be a further option if you support it with state-of-the-art hardware.
    – Kusalananda
    Commented May 22, 2023 at 5:57
10

Yes, the speed of the connection determines if the speeds things up. It will be overhead only for USB backup, because not the disks inflates the data but the process that writes the data. So the same machine that reads and deflated it, has to inflate and write it too. Rsync is still two processes I think but your memory to hand data from one process to the other is fast enough and the cpu need more time compressing it (while reading it into the same memory that later hands it over :).

Compression only helps when you have a sender and a receiver rsync and some slower network in between. 1Gbit might be already fast enough when you have a local NAS for instance, 10Gbit is already raw SATA speed. So compression is only needed when you have 100Mbit or less connectivity and it only makes sense when the data compressed is compressible.

I am think rsync might notice that it does not run on two machines but one and skips compression but not sure.

1
  • 5
    +1. This is a very good point for someone thinking that copying a file to an external drive attached to their laptop would be faster using compression. (No, it won't be.) Commented Mar 28, 2020 at 19:24
5

Depends on how compressible is your data and the processing power of your source and destination. A full disk backup in my experience will compress to about 30-50% of its original size, so it might be worth to give it a shot. Otherwise, don't bother with compression. It might be worth to test your compression rate with pigz -c <your file> | wc -c and compare the returned size with your original size.

1
  • 1
    This is an important point. If you're sending audio or video that's already been compressed, it's not going to do much other than slow things down.
    – W Biggs
    Commented Jan 5, 2021 at 0:55
2

I'd recommend to use --compress --compress-choice=lz4. lz4 has compression speeds of over 500MB/s with fairly good compression ratios, so it will not cause any overhead even on very fast network links, while still getting a large benefit for compressible files.

The only disadvantage is that it only works on new rsync versions, so no debian oldstable.

In general, you can tell when compression is the bottleneck by the rsync process on the sender side taking 100% cpu.

2

rsync since version 3.2.0 supports more than zlib:

--compress, -z
       turn on compression
--compress-choice=STR, --zc=STR
       choose compression from lz4 (fastest), zstd, zlibx, zlib (slowest), none
--compress-level=NUM, --lz=NUM
       zlib and zstd can be tuned with compression level
       zlib from 1 (lowest) to 9 (highest), default 6
       zstd from -131072 to 22, default 3
       For lz4 compression there are no levels, so the value is always 0.

Compression speeds up rsync if the the compression both manages to produce a smaller amount of data to transfer and does the compressing fast enough not to create a new bottleneck for the transfer. There are now many options to try, most faster than typical Internet connection speeds and some even faster than mass storage speeds (lz4 and zstd can be used in e.g. ZFS)

1
  • 2
    This may be true but it doesn't address the questions being asked, which are (a), "Does compression with rsync speed up backup?" and (b) "Does the answer to the question depend on if I backup to an external HDD […] or to a server […] over the Internet?" Commented Jan 13, 2022 at 14:13
0

Here is linked question "Should I use rsync compression over a gigabit LAN?": https://serverfault.com/questions/613709/should-i-use-rsync-compression-over-a-gigabit-lan

Based on practice in common environment with LAN 1Gbps copying from Server to NAS a single file size of 100GB can see that with default rsync compression type the speed varies about 10-15MB/s . And without compression it's 50-100MB/s.

Although there is new compression lz4 added to rsync. Yet to test it myself. https://serverfault.com/a/1089914/537333 (Note it must be supported on both sides i.e. Server and NAS)

0

rsync over network uses ssh by default. ssh cannot saturate fast networks, it is a long-standing problem, see https://www.psc.edu/hpn-ssh-home/hpn-ssh-faq/ for more information.

That's why compression is a requirement for network transfers with rsync over ssh.

Both rsync and ssh can do compression. Enabling both wastes CPU cycles on the second compression in ssh without achieving any better compression.

rsync skips compressing already compressed files (see --skip-compress option). rsync compresses each file on its own better than ssh compresses the entire uncompressed stream of rsync.

For fastest network file transfer the best practice is to enable rsync compression and explicitly disable ssh compression:

rsync --compress -e'ssh -oCompression=no' ...

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .