Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I want to make a fresh new copy of a large number of files from one local drive to another.

I've read that rsync does a checksum comparison of files when sending them to a remote machine over a network.

  1. Will rsync make the comparison when copying the files between two local drives?

  2. If it does do a verification - is it a safe bet? Or is it better to do a byte by byte comparison?

share|improve this question
up vote 49 down vote accepted

rsync always uses checksums to verify that a file was transferred correctly. If the destination file already exists, rsync may skip updating the file if the modification time and size match the source file, but if rsync decides that data need to be transferred, checksums are always used on the data transferred between the sending and receiving rsync processes. This verifies that the data received are the same as the data sent with high probability, without the heavy overheard of a byte-level comparison over the network.

Once the file data are received, rsync writes the data to the file and trusts that if the kernel indicates a successful write, the data were written without corruption to disk. rsync does not reread the data and compare against the known checksum as an additional check.

As for the verification itself, for protocol 30 and beyond (first supported in 3.0.0), rsync uses MD5. For older protocols, the checksum used is MD4.

While long considered obsolete for secure cryptographic hashes, MD5 and MD4 remain adequate for checking file corruption.

Source: the man page and eyeballing the rsync source code to verify.

share|improve this answer
I hate to burst everyone’s bubble but rsync only does check sum verification if the -c flag is added! – user30825 Jan 21 '13 at 21:32
@clint No, the answer is correct. From the man page's explanation of the -c flag: "Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking a whole-file checksum that is generated as the file is transferred, but that automatic after-the-transfer verification has nothing to do with this option's before-the-transfer "Does this file need to be updated?" check." – Michael Mrozek Jan 21 '13 at 21:41
@Kyle Jones - so that means when rsync exits, the file is completely received? This is different from scp which just sends the data and doesn't worry about if it is fully received before exiting, right? – David Doria Mar 18 '14 at 14:07
This answer does not make it clear if it actually verifies the file after a copy. If the checksum is computed as the file is being received, then it is not a post-copy checksum and you cannot be sure that the file is written correctly. You would then need to perform an additional comparison. – Andre Miller Mar 24 '15 at 21:26
@AndreMiller Thanks for the comment. I've updated the answer to address that issue. – Kyle Jones Mar 24 '15 at 22:02

rsync does not do the post-copy verification for local file copies. You can verify that it does not by using rsync to copy a large file to a slow (i.e. USB) drive, and then copying the same file with cp, i.e.:

time rsync bigfile /mnt/usb/bigfile

time cp bigfile /mnt/usb/bigfile

Both commands take about the same amount of time, therefore rsync cannot possibly be doing the checksum—since that would involve re-reading the destination file off the slow disk.

The man page is unfortunately misleading about this. I also verified this with strace—after the copy is complete, rsync issues no read() calls on the destination file, so it cannot be checksumming it. One more you can verify it is with something like iotop: you see rsync doing read and write simultaneously (copying from source to destination), then it exits. If it were verifying integrity, there would be a read-only phase.

share|improve this answer
"The man page is unfortunately misleading about this. I also verified this with strace" Did you strace the remote, running rsync process or the local one? There are two... one runs on the destination, even when you use ssh. – user129070 May 6 '13 at 19:20
There is no post-copy verification for any copies, local or remote. You run rsync -c again if you want to force it to check. – psusi May 6 '13 at 23:50

rsync makes a checksum comparison before copying (in some cases), to avoid copying what's already there. The point of the checksum comparison is not to verify that the copy was successful. That's the job of the underlying infrastructure: the filesystem drivers, the disk drivers, the network drivers, etc. Individual applications such as rsync don't need to bother with this madness. All rsync needs to do (and does!) is to check the return values of system calls to make sure there was no error.

share|improve this answer
This seems to contradict the accepted answer... – djule5 Jan 13 at 6:45
@djule5 In what way? The accepted answer seems to mostly be about how rsync checks transferred files, but the question, and my answer, are about local copies. – Gilles Jan 13 at 10:16
Ok, well in that context I agree it makes more sense. So "The point of the checksum comparison is not to verify that the copy was successful" is true only for local copies; and "checksums are always used on the data transferred between the sending and receiving rsync processes" is true only for transferred copies. I find the accepted answer misleading in regard to the question and believe your answer should be the accepted one (just my 2 cents). – djule5 Jan 13 at 18:23

I just use my own checking with my own script rather than rsync. Pre-Post-Secure-Copy checks a file integrity before transfer and after. If it fails deletes the remote file.

This is a very raw script and needs some modifications.


Hope this helps.

share|improve this answer
Please include the script here, or at least describe in detail the relevant steps so people can recreate your effort once link-rot sets in. You can of course leave the link as a pointer to more detailed information. – Anthon Apr 26 at 6:23
Can't it's more than 4296 characters – VeggieVampire Apr 30 at 0:53

protected by Community May 7 '13 at 2:47

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.