2

I'm new to rsync. I have a Linux machine with an external harddrive (ext3) and a Windows machine with an NTFS external harddrive. I connected the NTFS harddrive to the Linux machine and did a mirror copy from the ext3 drive. After, I copied some new files to the ext3 drive and then I connected the NTFS back to the Windows machine and from cygwin ran:

rsync -Pratluvz user@linux_machine:/path/to/data /cygdrive/f/

I expected the sync to be really fast since the data is the same except for the most recent addition. A typical line looks like this:

Documents/path/to/file.ext
   2688670 100%  801.24kB/s    0:00:03 (xfer#1, to-check=2477/12876)

It's taken almost as long to do the initial sync as the copy itself. However, when I stop the sync and resume, it begins where it left off. Is the first sync special in that it takes an inventory and writes to a file? Will later syncs be faster? (I toggled the 'z' parameter with no effect, and removed the 'c' parameter, which helped.)

EDIT: It turns out that I had included a '/' on the end of the source path, which caused all the contents of the data folder to be copied, not the folder itself. I halted the sync, removed the '/', and it functioned as originally expected.

|improve this question
  • Rsync does not use any kind of metadata, it will always start by building a list o all files and compare source and destination. If you are not using the -c flag it will skip files that have the same size and modification time on source and destination, if you are using the -c flag it will be much slower as it will do a checksum on every file. So you mention that it works better without 'c', but there is still a problem with it copying files that have not changed? – megahallon Jan 19 '15 at 21:23
  • I don't know if it's copying the files. It gives a progress indicator (shown above), but I didn't know if that was the progress of reading the file, comparing the files, copying, etc. It's actually still running now. – MrUser Jan 19 '15 at 22:05
  • The progress above is when its copying files. If you run it twice (with P flag) in succession, first time you should see "sending incremental file list" and then a bunch of "xfer" messages, second time you only see "sending incremental file list". – megahallon Jan 19 '15 at 22:13
  • By the way, z flag will enable compression which is pointless for a local copy and will slow it down considerably. – megahallon Jan 19 '15 at 22:16
2

Rsync only refrains from copying a file if it thinks the copy in the destination tree is identical to the copy in the target tree. By default, it assumes that the files are identical if they have the same size and the same modification time. You can pass the option -c to make rsync calculate a checksum instead (which is a lot slower since it has to read the whole file in the destination tree). You can pass the option --size-only to make rsync skip files whose size is the same on both sides, which is dangerous since files whose content has changed but where the new version has the same size as the old ones will be skipped.

I suspect that the first time you made the copy, you didn't preserve the timestamps (e.g. you used cp -r rather than cp -rp or cp -a). If that's what happened, rsync has no way to know that the files have the same content. Once the run of rsync -a is over, the files will have the same timestamtps, so rsync will know not to copy them again.

It's also possible that the timestamps aren't the same because the source filesystem and the destination filesystem use different granularity for timestamps (e.g. second vs. microsecond, so a file has a modification time like 2014-12-10 08:54:32.123465 on the source side but 2014-12-10 08:54:32 = 2014-12-10 08:54:32.000000 on the destination side). If that's the case and rsync doesn't detect it automatically, pass the option --modify-window, e.g. --modify-window=1 to allow a 1-second tolerance in timestamps.

|improve this answer
  • I used Midnight Commander to copy the files, so I'm not sure how the timestamps are preserved, but I'm assuming you're right. I will let it finish then run rsync again, and if it still tries to copy everything I'll try the --modify-window suggestion. Thanks. – MrUser Jan 19 '15 at 23:23
  • @MrUser Compare the output of ls -l or (for sub-second resolution) ls -l --full-time – Gilles 'SO- stop being evil' Jan 19 '15 at 23:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.