11

I use rsync 3.1.1 to keep in sync two discs, one of which is on the network and mounted as a samba share on /mnt/ROUTER_WD_2TB/. This is the source, with a speed of 30-40MB/s maximum due to network limitations. The other, the destination, is mounted locally (peak 110MB/s) and is /mnt/BACKUP_HITACHI_2TB/.

I use the following rsync command:

rsync  -haAXi --quiet --append-verify --delete /mnt/ROUTER_WD_2TB/* /mnt/BACKUP_HITACHI_2TB/.

The disks contain many files, most of them small.

The problem is that rsync takes quite a long time (10-20m) before starting to move any files, I guess because it has to compute file lists for a very large number of small files. During this period, the newtwork utilization sits at a low 200-500KB/s, while when transferring files the speed is about 40MB/s.

It usually happens that rsync takes about 15m to finally find something it has to copy, then takes 5 seconds to copy it, then continues checking for some other files to copy for another 5 minutes. All in all, a 5 second file copy ends up lasting 20 minutes!

Is there any measure I could take to avoid this long period prior to copying the files, other than excluding folders? Any kind of "cache" I can implement so that rsync does not have to rebuild all the file list from scratch?

  • 1
    Have you tried a solution from this thread? unix.stackexchange.com/questions/189878/… – UVV Apr 5 '16 at 7:45
  • @UVV I have not done that. i may do that, however 1) rsync already does not seem to use a full core in my case, for some reason (= I'm unsure if I'm CPU limited or if the limitation lies elsewhere) and 2) This is only on a dual-core CPU, so even if the improvement was 2X, there would still be a large margin for improvement. – AF7 Apr 5 '16 at 18:41
  • Yeah this seems crazy to me, why is rsync so slow at this? Even with "--size-only" rsync seems to take forever to build its incremental file list. Why? I can manually check the file sizes and determine which ones have changed much faster! What is it wasting so much time doing, and how do I turn these fancy features off? EDIT: Ah, never mind, I had the 'c' flag set by accident. Without the checksums it is very fast at this. – Ben Farmer Mar 8 '18 at 13:04
3

As far as rsync is concerned you're copying between two local file trees, so it disables most of its optimisations (including its delta algorithm for which it is famous). If you can run an rsync on your remote server (so you get a true client-server scenario across your network) you'll gain a significant amount of traction.

Nevertheless here are other options to consider

  1. Copying without worrying about deleting old files. This may allow you to perform a faster transfer more often, and relegate the slower cleanup to say, just once a day:

    cp -au /mnt/ROUTER_WD_2TB/. /mnt/BACKUP_HITACHI_2TB/
    
  2. Use a variation of rsync's default deletion algorithm to avoid building a full list of files before the transfer:

    rsync -haAXi --quiet --delete --delete-during /mnt/ROUTER_WD_2TB/ /mnt/BACKUP_HITACHI_2TB/
    
  3. Split the top-level directories into separate tasks and run them in parallel. You may find that if you're disk IO-bound then this won't really help, and for spinning platters it almost certainly will make things worse.

    for d in /mnt/ROUTER_WD_2TB/*
    do
        rsync -haAXi --quiet --delete --delete-during "$d" /mnt/BACKUP_HITACHI_2TB/ >"/tmp/rsync.${d/*\/}.log" 2>&1 &
    done
    wait
    cat /tmp/rsync.*.log
    rm -f /tmp/rsync.*.log
    

If none of these suggestions helps then it would be worth adding another --verbose to rsync to see what it is doing. I suspect it's rattling through all the unchanged files, and if you have enough files this simply takes a long time.

  • --delete --delete-during didn't do it for me — that still sent an incremental file list — but --delete --delete-before went straight to: building file list, which seems like major progress. I think. – mlissner Apr 19 '18 at 18:20
  • No, that's backwards, I'm afraid. If you use --delete-before, thenrsync performs two passes across the file tree: one for deletions and the next for copying. – roaima Apr 19 '18 at 18:34
  • Thanks. In that case...any way to avoid building the incremental file list? – mlissner Apr 19 '18 at 22:22
  • @mlissner it depends on several different factors. The biggest one by far is whether you're doing local-to-local or local-to-remote copying. – roaima Apr 19 '18 at 22:48
  • It's all local to local for me, though one of the locals is a USB disk? – mlissner Apr 20 '18 at 0:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.