Need help to create a checksum file of a directory on linux

Question

I backup my data to an external hard drive regularly, and in the past, some of the files ended up getting corrupted and I haven't noticed it until it was too late. So to prevent this, I would like to create an md5 checksum for all my files on the external hard drive. Now as time goes on, new files might end up getting added and older files might end up getting deleted. I'm currently running Linux Mint and I've been able to create a text file with the md5 sum of all the files on my external hard drive using the following command:

find '/path/to/backup/' -type f -exec md5sum {} \;>> /path/to/checksum.md5

Now I got this from a website a long time ago and I'm not very familiar with what everything means, and if someone could explain what each part of that command does, I would be grateful. That being said, the command above will always rehash the entire hard drive, and as the hard drive grows in size, it will take longer and longer to complete. I would therefore like to find a way to simply append new files on the hard drive without having to start hashing the entire hard drive all over again. In the past, I used the following command to do it, and it seemed to work just fine, until recently:

find 'path/to/backup' -type f -cnewer 'path/to/old/checksum/' -exec md5sum {} + > path/to/appended/checksum/

sort -k 2 -u /path/to/old/checksum path/to/appended/checksum > path/to/new/checksum.md5

For some reason, the above no longer finds the new added files and the appended checksum file has no entry in it. Anyone knows how to fix this or perhaps has a better way to accomplish what I'm trying to do?

Also is there a way to somehow scan the backup directory with the find command, omit any directories that are already in the md5 file and only display the files that have been added after the last checksum?

Answer 1

The parameter after -cnewer could be your problem:

• find is supposed to find files newer than the filedate of the file given after -cnewer.
• Seems you put a directory. So the first part should be:

  find 'path/to/backup' -type f -cnewer '/path/to/checksum.md5'
  

The second part is the exec function.

• From man find you get:

  -exec command {} ; executes the command (in your case: md5sum) where the string {} is replaced by the current file name. The specified command is run once for each matched file.

  -exec command {} + runs the specified command on the selected files, but the command line is built by appending each selected file name at the end; the total number of invocations of the command will be much less than the number of matched files. If find encounters an error, this can sometimes cause an immediate exit, so some pending commands may not be run at all.

• So a problem could be that there are too many files, or during while find, there is an error.

I hence recommend:

# Delete file /path/to/appended_checksum.md5 as we 'add' text to it.
rm /path/to/appended_checksum.md5
find 'path/to/backup' -type f -cnewer '/path/to/checksum.md5' -exec md5sum {} \;>> '/path/to/appended_checksum.md5'

sort -k 2 -u '/path/to/checksum.md5' '/path/to/appended_checksum.md5' > path/to/new_checksum.md5