I copied thousands of files into an exFAT MicroSD card.
The number of files and bytes is identical, but how do I know whether the data is corrupt or not?
It would be good if the JackPal Android Terminal also supports the command.
Unmount, eject, and remount the device. Then use
diff -r source destination
In case you used
rsync to do the copy,
rsync -n -c might be very convenient, and it is nearly as good as
diff. It doesn't do a bit-for-bit comparison though; it uses an MD5 checksum.
There are some similar answers with other details at: Verifying a large directory after copy from one hard drive to another
Using MD5 sums is a good way, but the canonical way to use it is:
cd to the directory of the source files and issue:
md5sum * >/path/to/the/checksumfile.md5
If you have directories with many levels, you can use
shopt -s globstar and replace
Notice that the file specs in the MD5 file are exactly as provided in the command line (relative paths unless your pattern starts with a
cd to the directory of the copied files and issue:
md5sum -c /path/to/the/checksumfile.md5
md5sum reads the file specs in the provided MD5 file, compute the MD5 of these files, and compares them to the values from the MD5 file (which is why the file specs are usually better left relative, so you can re-use the MD5 file on files in various directories).
Using MD5 sum this ways immediately tells you about MD5 differences, and also about missing files.
rsync -rc original-dir/ copied-dir/
-c causes rsync to compare files by MD5 checksum (without it, it normally uses only the timestamp and size for quicker comparisons).
This will also cause rsync to copy whatever it sees different or missing from the destination. To avoid that, you can also use
-i. The former ensures that rsync doesn't do any change and only compares, and the latter causes it to display the differences that it sees.
For example, I have the following dirs:
$ find dir1/ dir2/ dir1/ dir2/ dir1/ dir1/d dir1/d/a dir1/d/b dir1/c dir2/ dir2/d dir2/d/a dir2/d/b
$ rsync -rcni dir1/ dir2/ >f+++++++++ c >fc.T...... d/b
Tells me, by way of all those
+s, that file
c does not exist in
dir2, and file
d/b does, but is different (indicated by the
c in the first column). The
T says that it's time would be updated (had we not used
The format of
-i's output is described in the manpage for rsync. You can
man rsync and get to the part that explains that output by typing
/--itemize-changes$ (and hitting Enter).
It is possible to generate hashsums for individual files and output them into one text file, of which the MD5 hash can be generated. For that text file, you can use any hash function you like because this hash list's size is not large enough to cause any noticeable performance difference when using harder hash functions such as
cksum due to it's universal availability (
crc32 are not included in JackPal's Android Terminal) and maximum speeds. It is not a cryptographic, secure algorithm like
sha512sum, but any hash function is good enough to verify data integrity in an offline environment. However, if you wish all file hashes to have the same length (i.e. 32), use
md5sum, the fastest universally supported secure hash algorithm (although it is older, it is much faster than any sha algorithm and will do it's job).
cksum /path/to/folder/* | tee -a hash.files.txt |cut -f 1 -d " " >>hash.list.txt #extracts pure hashsum string only for the output, to hide the different file path. md5sum hash.list.txt
…or with a single command:
cksum /path/to/folder/* | tee -a hash.files.txt | cut -f 1 -d " " | tee -a hash.list.txt | sort | md5sum
The name of the hashsum list files (hash.list.txt and hash.files.txt in my example) can be anything you specify. Generating two files to be able to identify damaged files (the first file contains the file names as well, the second file is for comparison).
bash implement alphabetical sorting slightly differently.
sort compensates for it.
Along with the other fine answers above, I would like to also recommend considering hashdeep, from http://md5deep.sourceforge.net/. It has a good sized userbase in the scientific community, where they frequently have to do this type of thing with terabytes of data scattered across thousands of directories.