11

I just rsync-ed 2,000,000 files (3TB) from one RAID to another.

I want to make sure my data is intact.

rsync -c takes a really long time.

diff doesn't show me what it's doing.

Is there an alternative that's (a) faster, and (b) will show me progress while it's comparing?

(I'm on Mac, and brew search diff gives me apgdiff colordiff diffstat diffutils fmdiff libxdiff open-vcdiff podiff rfcdiff vbindiff bsdiff diffpdf diffuse dwdiff kdiff3 ndiff perceptualdiff rdiff-backup tkdiff wdiff ... would one of these do the job?)

  • Duplicate superuser.com/questions/708001/… – spuder Jan 28 '14 at 4:44
  • I'm also confused as to why rsync copied the data at around 150MB/s, yet diff compares at only 60MB/s ... ? – Dan Jan 28 '14 at 6:19
  • The copy using rsync is faster b/c rsync by default does not use checksums to compare files, it looks at size and date info. When you use rsync -c all the files need to have their checksums' calculated which is a burdensome task, hence why it's not the default. – slm Jan 28 '14 at 7:37
  • Yes, but diff doesn't copy ... it just reads both files; while rsync, to copy, must read each byte, then write it. This was an rsync from scratch, so it was copying every file. – Dan Jan 28 '14 at 8:06
6

edit for correction & option clarity - I forgot '--brief'

diff -rs --brief "$dir1" "$dir2" 

-r, --recursive              recursively compare any subdirectories found
-s, --report-identical-files report when two files are the same
-q, --brief                  report only when files differ
--speed-large-files      assume large files and many scattered small changes

and add other options to taste, depending on what you are comparing:

-i, --ignore-case            ignore case differences in file contents
-b, --ignore-space-change    ignore changes in the amount of white space
-B, --ignore-blank-lines     ignore changes whose lines are all blank
--strip-trailing-cr      strip trailing carriage return on input
--ignore-file-name-case  ignore case when comparing file names

diff -rs will read every byte of the original and copy, and report files that are the same.

The diff output format is defined by POSIX, so it is pretty portable. You may want to add something like:

| tee diff-out.1 | grep -v -Ee 'Files .* and .* are identical'

You could use chksums or hashes, but then you have to keep them sync'd with the file trees, so you would be back to reading every byte of every file anyway.

EDIT - too long to be a comment, in response to:

files over 10GB are not verifying

You may want to try this diff option: --speed-large-files

It is possible that the diff you are using is not coping well with very large files (bigger than system memory, for instance), and is thus reporting differences between files that are actually the same.

I had thought there was a -h option or a 'bdiff' that did better on large files, but I cannot find one in Fedora. I believe that the --speed-large-files options is a successor to a '-h' "half-hearted compare" option.

A different approach would be to repeat the rsync command you used, with '-vin' (verbose, itemize, no_run). This would report any differences that rsync finds - and there should not be any.

To move some files, you're looking at a script something like:

if [ cmp -s "$dir1/$path" "$dir2/$path" ] ; then 
target="$dir2/verified/$path"
mkdir -p $(basename "$target")
mv  "$dir2/$path" "$target"
fi

but I don't recommend doing that. The underlying question is "how can I be sure that rsync copied a file hierarchy correctly?" and if you can demonstrate to yourself that rsync is working well, with diff or some other tool, then you can just rely on rsync, rather than working around it.

rsync -vin will compare based on whatever other options you give it. I thought it defaulted to checksum, but you are right, -c or --checksum is required for that.

The diff utility is really intended for files of lines of text, but it should report 'identical' under -s for binary files.

The --brief should suppress any file content output - my apologies for overlooking it earlier - it was semi-buried in an ugly script.

  • Is there a way to get it to mv every file it finds to a "verified" folder at the root of the drive, preserving the full path? E.g., if /disk1/a/b/c/file1 is identical to /disk2/a/b/c/file1, then move it to /disk1/verified/a/b/c/file1. Then I could end up with only the badly copied files. (So far LOTS of files over 10GB are not verifying, which is scary.) – Dan Jan 28 '14 at 6:03
  • You may want to try this option: --speed-large-files – D McKeon Jan 28 '14 at 6:42
  • What does that do exactly? – Dan Jan 28 '14 at 7:05
  • If I run rsync -vin -- does that do a byte-by-byte or checksum comparison? I thought rsync only compared size/date unless you add -c. And from what I've read the speed large files seems to only make a difference with non-binary files ... or am I wrong? – Dan Jan 28 '14 at 7:12
  • diff gives me results in the form of "Files __ and ___ differ" ... and I'm running that through sed -e "s/Files /cp -afv /" -e "s/ and / /" -e "s/ differ$//" to try and generate a script for re-copying the bad files. But diff's output is unquoted, so it doesn't work. Can I get it to give me quoted paths? – Dan Jan 28 '14 at 17:51
2

Here is diff with progress report:

diff -rqs dir1 dir2 | pv -l -s filecount > logfile

You will need pv (pipe viewer): http://www.ivarch.com/programs/pv.shtml

Explanation:

  • diff -r compare directory and subdirectories recursively.
  • diff -q print only filenames of files differ. do not print actual differences.
  • diff -s print also filenames of files that do not differ. this is important for the progress information.
  • pv -l report progress based on line count.
  • pv -s count estimate time to complete based on count.
  • The redirect to logfile is for pretty output. Otherwise output from diff will mix with status line from pv.

to get file count use the following command:

find dir1 -type f | wc -l

Filter the logfile for files that are different:

grep -v "^Files .* identical$" logfile

This variation will print files that are different in real time while also logging everything in logfile:

diff -rqs dir1 dir2 | pv -l -s filecount | 
    tee logfile | grep -v "^Files .* identical$"

Alternatively you can log only files that are different:

diff -rqs dir1 dir2 | pv -l -s filecount | 
    grep -v "^Files .* identical$" > logfile

Note: the above commands will report progress based on file count. This works best if there are many smallish files. If you have a few huge files then you will not have much fun with this.

Sadly I do not know of a way to report progress based on bytes compared.


If you can find your peace just comparing the metadata (and not the actual content of the files) then you can use rsync. This will be considerably faster.

For more details:

0

I would look at using some sort of hash application to check data integrity. I know that many duplicate file finding utilities use hashes to identify duplicate/non-duplicates. Seems to me that this is an investigation that might be worthwhile.

0

You can use rdiff-backup for that. Install it on both servers and it will do smart comparisons of checksums and sync what is not yet there.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.