1

I have my camera's SD card with pictures and films. On my home box, I have a folder structure with directory names like this: images/YYYY-MM-description, sometimes there are subfolders with varying names below. The file names of the files on the SD card are the same file names as in the folders.

I'd like to compare (checksum) the SD card to the folder structure to find out if my box has all the files on the SD card. I thought of MD5ing them, but any algorithm that fits the use case would do.

I thought of using rsync or diff, but I can't come up with a solution because of the multi-level-structure on my box.

System is Debian wheezy, in case it matters.

  • Can you say just a bit more about the structure? I assume YYYY-MM-description is a directory name? And then do the files have the same names in the directories, so you just want to see what file names exist, or do you want to checksum everything and compare file contents? – user3188445 Aug 17 '15 at 7:05
  • Added the info in my edit, thanks for making me aware that this was missing. – Jan Aug 17 '15 at 7:10
2

You can use md5deep.

sudo apt-get install md5deep

First save the results of one of your folders:

md5deep -r -s /dir1> dir1sums

Now compare those with the other folder:

md5deep -r -X dir1sums /dir2

If there is no output that means the directories are identical. Otherwise it will display the hashes of files that are different.

2

I'll do the following assumptions:

1) all file names are unique

2) You only want to check for missing files ( not if files have the same md5 sum on each device, i.e. if a pic is corrupted)

3) Files can only be missing on the machine, the SD card has all files by default

Apart form assumption 1, the script could be changed to find single files in either location or to even do an md5 crosscheck for each file pair.

That way, we can use find, select all file names and check the lists for unique file names

#!/bin/bash

#load all file names from local dir as array:
pics_local=( $( find /path/to/dir/ -type f -printf "%f\n" ) )
#load all file names from SD card as array:
pics_SD=( $( find /mnt/SD/ -type f -printf "%f\n" ) )
#see if files are only in one of them,
#i.e. if file names appear only once (save as array):
singulars=( $( printf "%s\n" ${pics_local[@]} ${pics_SD[@]} |\
            sort | uniq -u ) )
#print list of missing files with full paths using find:
for (( i=0 ; i<=${#singulars[@]}-1 ; i++ )) ; do
    find /mnt/SD/ -type f -name "${singulars[$i]}"
done

UPDATE: script with md5sum for each file: We have all files on the SD and search for missing ones on the local dir. All local files are identical to the SD ones (no corresponding files with md5 sum mismatch)

#!/bin/bash

#load file names and md5sums of files on SD card into array
files_SD=( $( find /mnt/SD/ -type f ) )
md5_SD=( $( find /mnt/SD/ -type f -exec md5sum {} + | cut -d' ' -f1 ) )
#load md5sums of files in local folder into array:
md5_loc=( $( find /local/dir/ -type f -exec md5sum {} + | cut -d' ' -f1 ) )

#check for the very unlikely possibility of md5sums
#matching for more than two files
#IMHO: can be safely skipped
if [[ $(sort <( printf '%s\n' ${md5_loc[@]} ${md5_SD[@]}) |\
      uniq -c | awk ' $1 >= 3 ' ) ]] ; then
  echo "md5 sums matching for more than 2 files!"
  echo "critical error, aborting"
  exit
fi

singular_md5s=( $( printf '%s\n' ${md5_loc[@]} ${md5_SD[@]} | sort | uniq -u ) )
for (( i=0 ; i<=${#singular_md5[@]}-1 ; i++ )) ; do
  #assume SD card has all files
  #print file that is missing in local folder:
  #1) find where it is in the array:
  n=$(( $(grep -n "${singular_md5s[$i]}" <( printf '%s\n' ${md5_SD[@]} ) | cut -d: -f1 )-1 ))
  #2) print file name
  echo "${files_SD[$n]} missing on local folder"
done
  • Thank you, but filenames may be duplicate, so I need to md5 the files. – Jan Aug 17 '15 at 21:40
  • see the latest update – Fiximan Aug 17 '15 at 22:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.