11

This question already has an answer here:

I copied thousands of files into an exFAT MicroSD card.

The number of files and bytes is identical, but how do I know whether the data is corrupt or not?

It would be good if the JackPal Android Terminal also supports the command.

marked as duplicate by Thomas, Thomas Dickey, Mr Shunz, Anthony Geoghegan, msp9011 Jan 21 at 13:39

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    Please explain the -1 downvote. Thanks. – neverMind9 Jan 19 at 19:24
  • 5
    Not the downvoter, but this is a pretty basic and googleable question. Showing some prior effort would have likely been better received. Downvotes aren't the end of the world though, don't let it get to you. Shrug and move on. – Jared Smith Jan 19 at 23:03
  • 1
    @JaredSmith harsh judgement. It was posted along with an answer, that's clear evidence of "some" effort IMO. – sourcejedi Jan 19 at 23:30
  • 1
    @sourcejedi didn't realize it was a self-answer. At any rate, I wasn't the downvoter. – Jared Smith Jan 19 at 23:32
  • 1
    @JaredSmith The issue is not that I take downvotes personally but StackExchange bots could automatically impose an un-liftable 6 month ban* on my account without prior warning, which happened to me on SuperUser and on Meta.StackExchange.com too. Downvotes endanger my account. “Stack Exchange cannot lift question bans by request.” – yes, they technically could. This is just a nasty way of camouflaging it. After 6 months, one more troll-downvoted question restults into a lifetime ban from that particular StackExchange network. – neverMind9 Jan 25 at 20:00
18

Using MD5 sums is a good way, but the canonical way to use it is:

  1. cd to the directory of the source files and issue:

    md5sum * >/path/to/the/checksumfile.md5
    

If you have directories with many levels, you can use shopt -s globstar and replace * by **/*.

Notice that the file specs in the MD5 file are exactly as provided in the command line (relative paths unless your pattern starts with a /).

  1. cd to the directory of the copied files and issue:

    md5sum -c /path/to/the/checksumfile.md5
    

With -c, md5sum reads the file specs in the provided MD5 file, compute the MD5 of these files, and compares them to the values from the MD5 file (which is why the file specs are usually better left relative, so you can re-use the MD5 file on files in various directories).

Using MD5 sum this ways immediately tells you about MD5 differences, and also about missing files.

  • 6
    if the number of files goes into thousands, letting the shell do the wildcard expansion may cause troubles. Using find -exec is safer. – peterph Jan 19 at 23:37
  • 1
    @peterph Either that or a loop such as for or while. Shell built-ins are immune to Argument list too long error – Sergiy Kolodyazhnyy Jan 20 at 3:32
  • Good remarks. OTOH xargs --showlimits says the max command line is around 2MB (2086956 bytes). ls ./**/*.mp3` on my 21GB music collection (3600 files, kept as Genre/Artist/Album/Title.mp3, so rather long paths) is 290K bytes. So this is not too likely to happen. – xenoid Jan 20 at 9:15
11

Unmount, eject, and remount the device. Then use

diff -r source destination

In case you used rsync to do the copy, rsync -n -c might be very convenient, and it is nearly as good as diff. It doesn't do a bit-for-bit comparison though; it uses an MD5 checksum.


There are some similar answers with other details at: Verifying a large directory after copy from one hard drive to another

  • I like that. Unfortunately, Android Terminal lacks diff. Good nevertheless. – neverMind9 Jan 19 at 20:48
  • 2
    @neverMind9 Install Termux for diff,rsync,etc on Android. – user1133275 Jan 19 at 22:26
  • @user1133275 Actually, the similar alternatives cmp and comm are included in JackPal's terminal. – neverMind9 Jan 20 at 1:19
  • 1
    I tend to prefer diff -rq ... to suppress the diff output for any text files that may be present. I want to know if there are differences, not necessarily what the differences are. – smitelli Jan 20 at 3:45
4
rsync -rc original-dir/ copied-dir/

-c causes rsync to compare files by MD5 checksum (without it, it normally uses only the timestamp and size for quicker comparisons).

This will also cause rsync to copy whatever it sees different or missing from the destination. To avoid that, you can also use -n and -i. The former ensures that rsync doesn't do any change and only compares, and the latter causes it to display the differences that it sees.

For example, I have the following dirs:

$ find dir1/ dir2/
dir1/ dir2/
dir1/
dir1/d
dir1/d/a
dir1/d/b
dir1/c
dir2/
dir2/d
dir2/d/a
dir2/d/b

And this:

$ rsync -rcni dir1/ dir2/
>f+++++++++ c
>fc.T...... d/b

Tells me, by way of all those +s, that file c does not exist in dir2, and file d/b does, but is different (indicated by the c in the first column). The T says that it's time would be updated (had we not used -n).

The format of -i's output is described in the manpage for rsync. You can man rsync and get to the part that explains that output by typing /--itemize-changes$ (and hitting Enter).

4

It is possible to generate hashsums for individual files and output them into one text file, of which the MD5 hash can be generated. For that text file, you can use any hash function you like because this hash list's size is not large enough to cause any noticeable performance difference when using harder hash functions such as sha512sum.
I use cksum due to it's universal availability (sum and crc32 are not included in JackPal's Android Terminal) and maximum speeds. It is not a cryptographic, secure algorithm like sha512sum, but any hash function is good enough to verify data integrity in an offline environment. However, if you wish all file hashes to have the same length (i.e. 32), use md5sum, the fastest universally supported secure hash algorithm (although it is older, it is much faster than any sha algorithm and will do it's job).

Run these commands on both source and destination:

cksum /path/to/folder/* | tee -a hash.files.txt |cut -f 1 -d " " >>hash.list.txt #extracts pure hashsum string only for the output, to hide the different file path.
md5sum hash.list.txt

…or with a single command:

cksum /path/to/folder/* | tee -a hash.files.txt | cut -f 1 -d " " | tee -a hash.list.txt | sort | md5sum

The name of the hashsum list files (hash.list.txt and hash.files.txt in my example) can be anything you specify. Generating two files to be able to identify damaged files (the first file contains the file names as well, the second file is for comparison).

sort because sh and bash implement alphabetical sorting slightly differently. sort compensates for it.

2

Along with the other fine answers above, I would like to also recommend considering hashdeep, from http://md5deep.sourceforge.net/. It has a good sized userbase in the scientific community, where they frequently have to do this type of thing with terabytes of data scattered across thousands of directories.

  • Thanks for the link. – neverMind9 Jan 23 at 2:34

Not the answer you're looking for? Browse other questions tagged or ask your own question.