0

I backup my data to an external hard drive regularly, and in the past, some of the files ended up getting corrupted and I haven't noticed it until it was too late. So to prevent this, I would like to create an md5 checksum for all my files on the external hard drive. Now as time goes on, new files might end up getting added and older files might end up getting deleted. I'm currently running Linux Mint and I've been able to create a text file with the md5 sum of all the files on my external hard drive using the following command:

find '/path/to/backup/' -type f -exec md5sum {} \;>> /path/to/checksum.md5

Now I got this from a website a long time ago and I'm not very familiar with what everything means, and if someone could explain what each part of that command does, I would be grateful. That being said, the command above will always rehash the entire hard drive, and as the hard drive grows in size, it will take longer and longer to complete. I would therefore like to find a way to simply append new files on the hard drive without having to start hashing the entire hard drive all over again. In the past, I used the following command to do it, and it seemed to work just fine, until recently:

find 'path/to/backup' -type f -cnewer 'path/to/old/checksum/' -exec md5sum {} + > path/to/appended/checksum/

sort -k 2 -u /path/to/old/checksum path/to/appended/checksum > path/to/new/checksum.md5

For some reason, the above no longer finds the new added files and the appended checksum file has no entry in it. Anyone knows how to fix this or perhaps has a better way to accomplish what I'm trying to do?

Also is there a way to somehow scan the backup directory with the find command, omit any directories that are already in the md5 file and only display the files that have been added after the last checksum?

4
  • What is the output of find 'path/to/backup' -type f -cnewer 'path/to/old/checksum/'? Feb 26 '20 at 18:46
  • 1
    Consider also revision control, and backup the revision control repositories. Feb 26 '20 at 18:50
  • The terminal gives me no output to find 'path/to/backup' -type f -cnewer 'path/to/old/checksum/'. I just press enter and it does nothing, just goes back to a new input line.
    – John
    Feb 26 '20 at 20:08
  • I was also wondering how exactly does the find command check for newer files? Does it look at the time the last edit on the checksum file was made and then uses that value to looks for files that were created after that? Or does it go through the checksum file and adds any entries that are not in there?
    – John
    Mar 3 '20 at 17:56
0

The parameter after -cnewer could be your problem:

  • find is supposed to find files newer than the filedate of the file given after -cnewer.
  • Seems you put a directory. So the first part should be:
    find 'path/to/backup' -type f -cnewer '/path/to/checksum.md5'
    

The second part is the exec function.

  • From man find you get:

    -exec command {} ; executes the command (in your case: md5sum) where the string {} is replaced by the current file name. The specified command is run once for each matched file.

    -exec command {} + runs the specified command on the selected files, but the command line is built by appending each selected file name at the end; the total number of invocations of the command will be much less than the number of matched files. If find encounters an error, this can sometimes cause an immediate exit, so some pending commands may not be run at all.

  • So a problem could be that there are too many files, or during while find, there is an error.

I hence recommend:

# Delete file /path/to/appended_checksum.md5 as we 'add' text to it.
rm /path/to/appended_checksum.md5
find 'path/to/backup' -type f -cnewer '/path/to/checksum.md5' -exec md5sum {} \;>> '/path/to/appended_checksum.md5'

sort -k 2 -u '/path/to/checksum.md5' '/path/to/appended_checksum.md5' > path/to/new_checksum.md5

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.