5

UPDATE: Reading this again, I see that it was not a good question. I think I should've asked something like how reliable is rsync -aP --dry-run /origin /dest?. Of course that the most reliable won't be the fastest way of checking data integrity.

As the answers gently explain what rsync does with these options (and some failure scenarios) and tell from what diff does, I'll keep this question as it is.


The problem is: I copied a 340 GB folder and I wanted to check if the copy was successful. The copy was made using rsync -aP /origin /dest from one ext4 partition to another ext4 partition.

I know that using diff -r /origin /dest to compare the origin and destination directories would take a very (very very) long time. Then I thought I could use rsync -aP --dry-run /origin /dest to compare the directories... And it took 5 seconds(!). (I used time before the command to measure it.)

Since rsync -aP showed only the "sending incremental file list" message, and nothing else (it'd list differing files or files that were not copied), I conclude that it is the fastest and most reliable way to check whether two directories are identical. (As long as rsync -a option was used, files and directories are also identical in terms of owner, permissions and modification time... -P implies --progress.)

Am I right? Is rsync -aP --dry-run /origin /dest the fastest and most reliable way to verify whether directories are identical?

7

If you use diff, it always reads the file contents to pick up any differences between their contents.

Rsync, by default, will not read the contents of files if the filename, modification time and file size between the two are the same. If these are the same, it assumes the file contents are the same and doesn't bother reading them. This will make it orders of magnitude faster on directories that are already the same (or nearly the same).

In most cases this behaviour is not a problem, as it would be very rare for a file to have changed both before and after syncing, both within the same 1-second window in order to have the same modification time, and keep the same file size. And even if that did happen, it would be rare that that would matter much.

You can force rsync to always read the file contents regardless of modification time (or size) by using the --ignore-times command-line option.

  • "(...) very rare for a file to have changed both before and after syncing, both within the same 1-second window in order to have the same modification time, and keep the same file size". So using rsync -aP --dry-run is somehow reliable. It is not the most reliable, but reliable and extremely fast. – yuric Jul 28 '16 at 20:13
  • 2
    @YuriC, modifications within the same second on purpose should be rare, so as long as you trust the system to not corrupt data, checking timestamps should be ok. But to check for silent data corruption, you'd need to reread all the data. – ilkkachu Jul 29 '16 at 7:30
  • @ilkkachu, but what is the probability of data corruption result in a identical file size? I think this is the point for considering rsync -aP --dry-run somehow reliable or not. – yuric Aug 5 '16 at 13:50
  • @YuriC, by 'silent corruption', I mean a bit getting flipped during transmission without it being noticed during the copy (be it because of bad memory, cosmic rays, physical imperfections, or whatever). A bit error within the metadata would be likely show up (or crash the copying program), but a silent change in the data could be just copied along. Granted, it's not that probable with "only" a few hundred GB, but it depends on what you think of as the threat and how paranoid you are about it. (The title said also "most reliable", so...) – ilkkachu Aug 5 '16 at 15:38
3

Speed and reliability are often opposite requirements. diff (or cmp) would compare the files bit-by-bit, which would give the most reliable answer. (Use diff -q if you only want to know if the files are the same.)

To make it faster, you'd have to skip on either reading the files in full, and count on timestamps; or skip on the data transferred, which probably isn't a huge issue within one system since disk I/O will likely be the factor bounding the speed.

If the files were on different systems, and you have md5sum or similar, you could do something like

cd /somewhere ; find . -type f | xargs -d '\n' md5sum > /tmp/checksums

on both ends and then compare the output files. That way you would only need to transfer the hashes, not the whole data. (or sha512sum if you want a stronger check). rsync -c also apparently uses MD5 internally.


The choice between just checking timestamps and reading the whole data also depends on what the assumed source of errors is. Checking metadata will tell if the copy process terminated before copying everything, but it will not check the actual data copied. To detect silent data corruption, the data would absolutely need to be read in full, and either compare the full data or just checksums. With rsync, rsync -i -c will output a list of files whose checksums did not match, and copy them again.

  • No, rsync -a --dry-run does list differing files or files that were not copied. Then, if the command doesn't list anything, the directories are the same in terms of file name, file size and modification time. But you're right that this is not the most reliable way, as long as it doesn't read the files in full. – yuric Jul 28 '16 at 19:09
  • Correction: rsync -aP --dry-run is the right command for listing differing files or files that were not copied. Because -P implies --progress. I'll edit the question. – yuric Jul 28 '16 at 19:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.