Why is uncompressed tar much faster than rsync?

Question

I'm currently looking for a way to improve my system backup performance and realized the following during some tests:

Backing up my Ubuntu system from an SSD to an HDD (both ext4) using TAR without compression is much faster than rsyncing the same content from SSD to HDD.

In detail:

• TAR takes 1h 15min and produces a 429G large file
• rsync takes 5h and produces a 406G large folder

Both should copy the same data as they're using ignore files with the same content slightly adjusted to both tools.

I'm not sure why the final TAR is actually larger than the rsynced folder but I don't really care about that ATM.

What I'm really interested in is why TAR is so much faster & if I could somehow improve rsync (or some other file copy tool) to gain a similar performance?

I don't really want to use TAR for my backup strategy as decompressing such large archives or only extracting single files takes "ages" which would be problematic when I really need access to them.

I realize that I could improve rsync performance drastically by always copying to the same target folder & therefore getting incremental copies but that's explicitly not what I'm looking for as I'd like to always have multiple backups of different days.

---

Update, additional info

Tested alternative "copy via TAR"

I also tried "copying via TAR" (as described here or here) which is a bit slower than rsync, so the bottleneck seems to be the write speed.

Commands used

I've used the following commands which lead to the results above:

• tar -X "tar-excludes.txt" -cvf "/media/backup/full" "/"
• rsync -aAXWvh --stats --info=progress2 --exclude-from "rsync-excludes.txt" --log-file="log.txt" "/" "/media/backup/full"

Files

I'm backing up my whole OS (with some exclusions) so the backup includes all kinds of files. Some large files as well as surely many small files.

Device details

The host is an ~ 8 year old Intel NUC D34010WYKH

The source drive is an internal SSD and the destination drive is an external HDD connected via USB 3.0. Both drives are formatted with ext4.

Answer 1

The various cpio and tar file formats are simple sequences of file headers followed by file data. Writing out a new file header is simply appending a record to the output file. Writing out file data is simply appending more records to the output file.

That's the only thing that ever happens: records get appended to the output file. Usually these records are batched into 10KiB or 5KiB (or sometimes even 1MiB) blocks, too.

This is a very efficient thing to do. If the output file is an actual tape device this is just append writes, sequential output, to the current position on the tape. This comes as no surprise. These utilities were designed to archive files to tape, with its good sequential I/O characteristics, and terrible random-access I/O characteristics.

(Adding on compression doesn't change this, either. The compression utilities are designed to use sequential I/O, too.)

Even if this is a file on a disc volume, each appended batch of records is basically three actions: twiddling the disc volume's free space maps to get another block, twiddling the file i-node to include that new block at the end of the file (which filesystems can make cheap by using extents and suitable allocation policies), and writing out the block. With the common filesystem driver optimization of speculatively pre-allocating runs of contiguous data blocks when it detects sequential append write patterns, this can be done very cheaply indeed.

Whereas an rsync-style backup is making an entire tree in the disc volume, which involves creating directory entries, updating B-trees and whatnot, allocating i-nodes, making hard links, and updating any journals, in addition to doing at the individual file level what the cpio/tar archive does: twiddling the disc volume's free space maps, twiddling an i-nodes's block allocations, and writing out file data blocks.

Writing out an archive using only sequential append operations is very efficient for tape, and can be made reasonably efficient for a single archive file stored on a disc volume. Writing out lots of individual files involves inherently doing a lot more.

The obvious things that you are trading for this efficiency are of course ease of making in-line modifications to an archive, good random access reading of the archive, and smart incremental backup facilities.

In the 1980s, Rahul Dhesi created an archive format that was mostly serial, that enabled in-line updates to be made to existing archives using a modicum of random access I/O to go back and overwrite the headers for superseded files. Its tradeoffs were that file headers and data for superseded files had to be purged by rewriting the whole archive, every so often, and of course the greater space requirements for holding old not-yet-purged versions of files.

Answer 2

TAR takes 1h 15min and produces a 429G large file

rsync takes 5h and produces a 406G large folder

Peering into my crystal ball I can make some deductions from this - there's lots of small files and significant latency between the source and destination devices. It would have been helpful if you'd had a look at these factors and included what you found in your question along with the atual comands you are running to create the backups.

Tar is so much faster because:

• the data traffic is only going in one direction, it can (potentially) saturate the connection - OTOH rsync needs to retrieve data synchronously from both ends
• tar writes a single stream so there is no impact from file creation

I realize that I could improve rsync performance drastically by always copying to the same target folder & therefore getting incremental copies but that's explicitly not what I'm looking for as I'd like to always have multiple backups of different days

Assuming that the source and destination are block devices attached to the same host (agan, not specified) maybe you should try overlay filesystems.

Answer 3

this is / dir of a centos (irrevelent here)

bin  boot  dev  etc  home  lib  lost+found  media  mnt  opt  proc  root  sbin  selinux  srv  sys  tmp  usr  var

most likely you don't want to copy /dev, /proc and /sys, depending on your need you don't want /media either.

so instead of using rsync / $DEST (I assumed $DEST is on another host).

you can run

rsync /bin /boot /etc /lib /root /sbin /selinux $DEST &
sleep 300
rsync /home $DEST &
sleep 300
rsync /opt $DEST &
...
wait

if all your data are in /home, you can go further down

rsync /home/dir1 $DEST &
sleep 300
rsync /home/dir2 $DEST &
...

You will have to adjust $DEST, or use excluding option in rsync

let's pretend you have 1 000 000 files, rsync (one or many) still have to check 1M files stats (in source part) and 1M files stats (on dest part), do some compression etc ...

As mentionned in comment, I had to synchronize, twice a day directories with 100 M files, rsync would last 14h to 16h, with above strategy (and a bit of trial and errors), I manage to go down to 4h to 5h, using 20 rsync (15 of them being short live one)