I use rsync 3.1.1 to keep in sync two discs, one of which is on the network and mounted as a samba share on /mnt/ROUTER_WD_2TB/. This is the source, with a speed of 30-40MB/s maximum due to network limitations. The other, the destination, is mounted locally (peak 110MB/s) and is /mnt/BACKUP_HITACHI_2TB/.

I use the following rsync command:

rsync  -haAXi --quiet --append-verify --delete /mnt/ROUTER_WD_2TB/* /mnt/BACKUP_HITACHI_2TB/.

The disks contain many files, most of them small.

The problem is that rsync takes quite a long time (10-20m) before starting to move any files, I guess because it has to compute file lists for a very large number of small files. During this period, the newtwork utilization sits at a low 200-500KB/s, while when transferring files the speed is about 40MB/s.

It usually happens that rsync takes about 15m to finally find something it has to copy, then takes 5 seconds to copy it, then continues checking for some other files to copy for another 5 minutes. All in all, a 5 second file copy ends up lasting 20 minutes!

Is there any measure I could take to avoid this long period prior to copying the files, other than excluding folders? Any kind of "cache" I can implement so that rsync does not have to rebuild all the file list from scratch?

  • 1
    Have you tried a solution from this thread? unix.stackexchange.com/questions/189878/…
    – UVV
    Apr 5, 2016 at 7:45
  • @UVV I have not done that. i may do that, however 1) rsync already does not seem to use a full core in my case, for some reason (= I'm unsure if I'm CPU limited or if the limitation lies elsewhere) and 2) This is only on a dual-core CPU, so even if the improvement was 2X, there would still be a large margin for improvement.
    – AF7
    Apr 5, 2016 at 18:41
  • Yeah this seems crazy to me, why is rsync so slow at this? Even with "--size-only" rsync seems to take forever to build its incremental file list. Why? I can manually check the file sizes and determine which ones have changed much faster! What is it wasting so much time doing, and how do I turn these fancy features off? EDIT: Ah, never mind, I had the 'c' flag set by accident. Without the checksums it is very fast at this.
    – Ben Farmer
    Mar 8, 2018 at 13:04

1 Answer 1


As far as rsync is concerned you're copying between two local file trees, so it disables most of its optimisations (including its delta algorithm for which it is famous). If you can run an rsync on your remote server (so you get a true client-server scenario across your network) you'll gain a significant amount of traction.

Nevertheless here are other options to consider

  1. Copying without worrying about deleting old files. This may allow you to perform a faster transfer more often, and relegate the slower cleanup to say, just once a day:

    cp -au /mnt/ROUTER_WD_2TB/. /mnt/BACKUP_HITACHI_2TB/
  2. Use a variation of rsync's default deletion algorithm to avoid building a full list of files before the transfer:

    rsync -haAXi --quiet --delete --delete-during /mnt/ROUTER_WD_2TB/ /mnt/BACKUP_HITACHI_2TB/
  3. Split the top-level directories into separate tasks and run them in parallel. You may find that if you're disk IO-bound then this won't really help, and for spinning platters it almost certainly will make things worse.

    for d in /mnt/ROUTER_WD_2TB/*
        rsync -haAXi --quiet --delete --delete-during "$d" /mnt/BACKUP_HITACHI_2TB/ >"/tmp/rsync.${d/*\/}.log" 2>&1 &
    cat /tmp/rsync.*.log
    rm -f /tmp/rsync.*.log

If none of these suggestions helps then it would be worth adding another --verbose to rsync to see what it is doing. I suspect it's rattling through all the unchanged files, and if you have enough files this simply takes a long time.

  • --delete --delete-during didn't do it for me — that still sent an incremental file list — but --delete --delete-before went straight to: building file list, which seems like major progress. I think.
    – mlissner
    Apr 19, 2018 at 18:20
  • No, that's backwards, I'm afraid. If you use --delete-before, thenrsync performs two passes across the file tree: one for deletions and the next for copying. Apr 19, 2018 at 18:34
  • Thanks. In that case...any way to avoid building the incremental file list?
    – mlissner
    Apr 19, 2018 at 22:22
  • @mlissner it depends on several different factors. The biggest one by far is whether you're doing local-to-local or local-to-remote copying. Apr 19, 2018 at 22:48
  • It's all local to local for me, though one of the locals is a USB disk?
    – mlissner
    Apr 20, 2018 at 0:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .