I'm currently looking for a way to improve my system backup performance and realized the following during some tests:
Backing up my Ubuntu system from an SSD to an HDD (both ext4) using TAR without compression is much faster than rsyncing the same content from SSD to HDD.
In detail:
TAR
takes1h 15min
and produces a 429G large filersync
takes5h
and produces a 406G large folder
Both should copy the same data as they're using ignore files with the same content slightly adjusted to both tools.
I'm not sure why the final TAR is actually larger than the rsynced folder but I don't really care about that ATM.
What I'm really interested in is why TAR is so much faster & if I could somehow improve rsync (or some other file copy tool) to gain a similar performance?
I don't really want to use TAR for my backup strategy as decompressing such large archives or only extracting single files takes "ages" which would be problematic when I really need access to them.
I realize that I could improve rsync performance drastically by always copying to the same target folder & therefore getting incremental copies but that's explicitly not what I'm looking for as I'd like to always have multiple backups of different days.
Update, additional info
Tested alternative "copy via TAR"
I also tried "copying via TAR" (as described here or here) which is a bit slower than rsync, so the bottleneck seems to be the write speed.
Commands used
I've used the following commands which lead to the results above:
tar -X "tar-excludes.txt" -cvf "/media/backup/full" "/"
rsync -aAXWvh --stats --info=progress2 --exclude-from "rsync-excludes.txt" --log-file="log.txt" "/" "/media/backup/full"
Files
I'm backing up my whole OS (with some exclusions) so the backup includes all kinds of files. Some large files as well as surely many small files.
Device details
The host is an ~ 8 year old Intel NUC D34010WYKH
The source drive is an internal SSD and the destination drive is an external HDD connected via USB 3.0. Both drives are formatted with ext4
.
rsync
or plaincp
in my case./
. Are you suggesting to start a dedicatedrsync
process for each folder in/
that I'm interested in? E.g./bin
,/boot
,/etc
,/home
, ...?