Speed up / avoid calculation of rsync file lists

Question

I use rsync 3.1.1 to keep in sync two discs, one of which is on the network and mounted as a samba share on /mnt/ROUTER_WD_2TB/. This is the source, with a speed of 30-40MB/s maximum due to network limitations. The other, the destination, is mounted locally (peak 110MB/s) and is /mnt/BACKUP_HITACHI_2TB/.

I use the following rsync command:

rsync  -haAXi --quiet --append-verify --delete /mnt/ROUTER_WD_2TB/* /mnt/BACKUP_HITACHI_2TB/.

The disks contain many files, most of them small.

The problem is that rsync takes quite a long time (10-20m) before starting to move any files, I guess because it has to compute file lists for a very large number of small files. During this period, the newtwork utilization sits at a low 200-500KB/s, while when transferring files the speed is about 40MB/s.

It usually happens that rsync takes about 15m to finally find something it has to copy, then takes 5 seconds to copy it, then continues checking for some other files to copy for another 5 minutes. All in all, a 5 second file copy ends up lasting 20 minutes!

Is there any measure I could take to avoid this long period prior to copying the files, other than excluding folders? Any kind of "cache" I can implement so that rsync does not have to rebuild all the file list from scratch?

Answer 1

As far as rsync is concerned you're copying between two local file trees, so it disables most of its optimisations (including its delta algorithm for which it is famous). If you can run an rsync on your remote server (so you get a true client-server scenario across your network) you'll gain a significant amount of traction.

Nevertheless here are other options to consider

1. Copying without worrying about deleting old files. This may allow you to perform a faster transfer more often, and relegate the slower cleanup to say, just once a day:

   cp -au /mnt/ROUTER_WD_2TB/. /mnt/BACKUP_HITACHI_2TB/
   

2. Use a variation of rsync's default deletion algorithm to avoid building a full list of files before the transfer:

   rsync -haAXi --quiet --delete --delete-during /mnt/ROUTER_WD_2TB/ /mnt/BACKUP_HITACHI_2TB/
   

3. Split the top-level directories into separate tasks and run them in parallel. You may find that if you're disk IO-bound then this won't really help, and for spinning platters it almost certainly will make things worse.

   for d in /mnt/ROUTER_WD_2TB/*
   do
       rsync -haAXi --quiet --delete --delete-during "$d" /mnt/BACKUP_HITACHI_2TB/ >"/tmp/rsync.${d/*\/}.log" 2>&1 &
   done
   wait
   cat /tmp/rsync.*.log
   rm -f /tmp/rsync.*.log
   

If none of these suggestions helps then it would be worth adding another --verbose to rsync to see what it is doing. I suspect it's rattling through all the unchanged files, and if you have enough files this simply takes a long time.