I have a home file server that I use Ubuntu on.

Recently, one of my drives filled up so I got another and threw it in there.

I have a very large folder, the directory is about 1.7 T in size and contains a decent amount of files.

I used GCP to COPY the files from the old drive to the new one and it seems to have worked fine.

I want to now validate the new directory on the new drive against the original directory on the old drive before I delete the data from the old drive to free up space. I understand that I can do a CRC check to do this.

How, specifically, can I do this?

  • 2
    You might like to check there are no symbolic links or special files in your original folder with find /origfolder ! -type f ! -type d -ls. There should be no output. – meuh Sep 29 '16 at 12:26
  • 1
    (And also if any files are hard linked find /origfolder -type f -links +1 -ls. This is unlikely if the files are your own, and not from some downloaded software package, for example). – meuh Sep 29 '16 at 12:37

I’d simply use the diff command:

diff -rq --no-dereference /path/to/old/drive/ /path/to/new/drive/

This reads and compares every file in the directory trees and reports any differences. The -r flag compares the directories recursively while the -q flag just prints a message to screen when files differ – as opposed to printing the actual differences (as it does for text files). The --no-dereference flag may be useful if there are symbolic links that differ, e.g., in one directory, a symbolic link, and in its corresponding directory, a copy of the file that was linked to.

If the diff command prints no output, that means the directory trees are indeed identical; you can run echo $? to verify that its exit status is 0, indicating that both sets of files are the same.

I don’t think computing CRCs or checksums is particularly beneficial in this case. It would make more sense if the two sets of files were on different systems and each system could compute the checksums for their own set of files so only the checksums need to be sent over the network. Another common reason for computing checksums is to keep a copy of the checksums for future use.

  • Does diff work with any kind of file type? Like mkv/ogg for videos? – Peter A Sep 29 '16 at 9:28
  • @PeterA That's correct. Diff works with all file types. It can provide more information about the specific differences between text files while for non-text files, it simply reports if there is any difference. BTW, I edited my answer with an extra sentence on the expected output of the command. – Anthony G - justice for Monica Sep 29 '16 at 10:49
  • Thanks @meuh I've incorporated that into my answer -- and upvoted yours. – Anthony G - justice for Monica Sep 29 '16 at 13:09

rsync is often used to copy files instead of gcp, but it can also be used to verify a copy, however it was made. Simply do

rsync -niaHc /origfolder/ /copyfolder

Be careful to end the first folder name (the source) with a /. The options are

  • -n do not copy (make no changes)
  • -i itemise the differences
  • -a preserve (i.e. compare since we have -n) permissions, ownerships, symbolic links, etc. and recurse down directories
  • -H preserve hard links
  • -c compare checksums

The output shows a code detailing the differences for each file or directory that differs. There is no output if they are the same. The code has columns YXcstpoguax where each character is a dot . if that aspect of the comparison is ok, or a letter:

Y is type of update: 
   < sent (not appropriate in this case)
   > need to copy 
   c missing file or directory
   h is hard link
   . no update
   * and rest of line is a message, eg *deleting
X file type: f file  d dir  L symlink  D device S special file
c checksum differs. + new item  " " same
s size differs
t timestamp differs
p permissions differ
o owner differ
g group differ
u (not used)
a acl differ
x extended attributes differ

For example,

.d..t...... a/b/                    directory timestamp differs
cL+++++++++ a/b/d -> /nosuch2       symbolic link missing
cS+++++++++ a/b/f                   special file missing (a/b/f is a fifo)
>f..t...... a/b/ff                  file timestamp differs
hf          a/b/xx1 => a/b/xx       files should be a hard linked
cLc.t...... a/b/z -> /tmp/hi2       symbolic link to different name
cd+++++++++ a/c/                    directory missing
>f+++++++++ a/c/i.10                missing file needs to be copied

See man rsync under --itemize-changes for more details. If you have differences in the 3rd c or 4th s columns, then you have serious data corruption. Other flags such as different permissions, owner or timestamps may be less important to you. If all the files are marked as "missing" then you have probably not given the right directories to compare. If you are sure, running rsync without the -n flag will "fix" the differences.

  • When I try this option, I get a lot of output, and it is not clear to me what it means. – Peter A Oct 4 '16 at 1:16
  • I've added some examples and their meaning. See man rsync. – meuh Oct 4 '16 at 6:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.