1

I am syncing two drives using rsync (cygwin) using the following command rsync -a -H /cygdrive/e/ /cygdrive/d/ í have about 3 tb of data and probably about 1 million files (including) hard links. The first time it took about 1-2 days to finish if í remember correctly. If í rerun this command it takes a long time even if nothing is updated. It take a couple of hours---12 hours. Shouldn't this take a lot less time?

  • How much RAM do you have? I guess cygwin environment is much slower than real Linux but even Linux needs around 4 GB of RAM per 1 TB of storage to handle lots of small files with high performance. If cygwin causes even 50% overhead compared to in-kernel implementation, you should have at least 16 GB of effective RAM cache for the job. If you have any less, the task will take lots of time due re-reading HDD all again (I'm assuming you do not have 6 TB of SSD). – Mikko Rantalainen Jul 2 at 10:25
  • I had 12 GB at the time. I don't remember if the operation was maxing out my available RAM. Nowadays I use either Linux or Windows Subsystem for Linux (Ubuntu) without any problems. – rjss Jul 2 at 13:57
  • rsync does not need the RAM itself but the system needs RAM to keep full directory hierarchy in RAM. Otherwise rerun will need to scan the full directory hierarchy again and that is really slow with HDD because directory scanning requires nearly random access pattern to storage and HDDs are really bad at that. If you do single-shot transfers, this does not matter. However, you specifically asked about rerun taking a lot of time so this applies. – Mikko Rantalainen Jul 3 at 12:55
1

You're copying between two locations on the same system. This switches off the delta algorithm and reverts rsync to being a fancy implementation of cp -a.

The -H flag causes rsync to have to keep a record of every file with more then one hard link so that it can reconstruct the hard links in the target. This increases memory use (empirically I'd say by quite a lot), which in turn can put heavier demand on your system.

Finally, Cygwin rsync is considerably slower than a native UNIX/Linux implementation. I've not determined whether this is because it's using a compatibility layer on top of Windows or because the filesystem is slower or even because of something else entirely. But it is slower for an otherwise identical set of files on the same hardware.

0

This is not a full answer but I think it could be really valuable for some people stuck in the same situation. I ended up doing my rsync from Linux and it worked without problems. If you are having this issue try at least booting to Ubuntu/CentOS or your prefered Linux distro even from a CD and see if it works.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.