How do I easily update list of md5sums?

Question

Sometimes I use an unreliable medium (flash) to store a good deal of data. To at least recognise bit flips I store a file with the md5sums alongside. This file is usually created by a variation of find -type f -exec "{}" \; >MD5SUM. Later I copy some more files on it and now I would like to add the checksums of the new files without having to recalculate the old ones. Sadly, the time of some the machines I use are screwed, so using find -newer <file> -exec md5sum "{}"\; >>MD5SUM is not an option. Basically I would like to get the difference between the file list created by find -type f and the list in the MD5SUM file.

Any ideas how to do this in an easy and elegant manner? Thanks in advance!

Answer 1

If this is going to be an on-going process, then you'll need two files, the old and new (which would become the old for next time).

#!/bin/sh
# change directory to either first argument or to current directory
cd ${1:-"."} || exit 1 # if cannot cd, then exit
# get the md5 values for all the files in the directory tree
find . -type f -not -name .md5sum.last -exec md5sum {} \; | sort > .md5sum.tmp
# if called before, then get only the differences in the newer
if [ -f .md5sum.last ]; then
    comm -13 .md5sum.last .md5sum.tmp
else  # otherwise show all the output
    cat .md5sum.tmp
fi
# replace the older with the current for next time
mv .md5sum.tmp .md5sum.last

The sort and comm -13 are the key. Sort is obvious, but comm (short for "common") will show lines that are in the first file (column 1), second file (column 2) or both (column 3). The -13 option says to "take away column one and three" leaving only lines that are not in just the older and not common to both. Unfortunately, if you cannot trust the time stamps on the files, then this would be a very intensive process for large directory trees.

Answer 2

I think the easiest way would be to store the checksum of file _my_file_ in the file _my_file_.md5 avoid storing all checksum in a single file. This way, it is much simpler to know if a checksum has been computer before.

However, if you only add files to your flash drive (never modify, maybe delete, but never add a file that has once existed before), you could:

find _your_drive_path_ -type f |
  while read file; do
    grep -q $file _your_md5_file_ || md5sum $file >> _your_md5_file_
  done

This grep your checksum file many times and could be optimized by sorting the file list and keeping your checksum file sorted on the filename, but if you do not need this optimization, why bothering with its complexity...

Answer 3

If you can't trust the timestamps, then there's really no way to process only the files that have changed. Just repeat your original find command.

I would save the new MD5SUM file to a temporary location, and then diff the old and new files to see what changed before copying the updated file to flash. You might need to sort the file in order to get a useful diff.