I basically have a directory a with lots of images. Now I want to check if all of these images are in directory b. The point is, that lots of images in b ain't directly in b but in subdirectories.

Also I don't want to depend on filenames, but file contents.

(because of the bash tag: I'd prefer a bash answer, but if it's some other language or if it's using another program, it's ok too)


Since you want to compare them by content, using hashes seems to be a way to do it.

You can use the find command to get a list of file paths of a directory. The -type f option will leave out all directories and only output paths to regular files. The -exec md5sum {} \; option will take the found paths and gives them to the md5sum command to turn into a list md5 hash + their filepaths ('md5_hash /path/to/file').

We pipe that list into the cut command. The first option -f 1 tells it to only take the first column (the hashes). The second one -d ' ' tells it to use a space character as the delimiter between columns. Default is a TAB.

We pipe that list of hashes into the sort command, to make it easier for diff.

The <( command ) operator is called Process Substitution . It takes the output of a command and turns it into a pseudo file for commands that demand them as input (for a less simple explanation follow the link). That way, it looks to diff as if we want to compare two files.

:~$ diff <(find folder1/ -type f -exec md5sum {} \; | cut -f 1 -d ' ' | sort) \
    <(find folder2/ -type f -exec md5sum {} \; | cut -f 1 -d ' ' | sort)

Note: Don't forget to substitute folder1/ and folder2/ with your actual folders.

This will give you a list of md5 hashes of the files that are only in one or the other.

If you want to know which files are actually missing you can do:

:~$ find folder1/ -type f -exec md5sum {} \; | sort | grep my_md5_hash

If you have a lot of files to check, it would be wise to save the results of the two <(find ...) commands and compare them like:

:~$ diff list1.txt list2.txt
:~$ cat list1.txt | grep my_md5_hash
  • Thanks for this answer. On Mac, the command is called md5 and it prints out some extra information by default. The cut used here doesn't cut in the right place (for md5 on Mac), and the sort isn't necessary if the files in the folders are actually the same. The -q option tells md5 to only print out the checksum. So for me the command became diff <(find folder1 -type f -exec md5 -q {} \;) <(find folder2 -type f -exec md5 -q {} \;). – AmadeusDrZaius May 21 '17 at 22:26

See the answer for a similar question from two weeks ago.

find . -type f -exec md5sum {} + | sort | sed 's/  */!/1' | awk -F\| 'BEGIN{first=1}{if($1==lastid){if(first){first=0;print lastid, lastfile}print$1, $2} else first=1; lastid=$1;lastfile=$2}'

Assuming those are all *.jpg files, do a:

To find the Files present:

grep -Ff <(for i in </path/to/directory/a>/*.jpg ; do md5sum $i | awk {'print $1'}; done) <(find </path/to/directoryb/ -iname "*.jpg" | xargs md5sum)

The for loop here creates a list of md5 checksums of all the *.jpg files in directory 'a' and the find here will create a list of md5 checksums of all the *.jpg files in directory 'b' (including sudirectories).

The grep -fF will compare these two lists and the full command will produce a 2 coloumn output with the 1st coloumn being the md5 checksum of the files that is present and the 2nd coloumns being the filename(with full path) of the files that match in directory 'b'. You can use an additional | awk {'print $2'} if you want to get only the filenames.

To find the files not present:

grep -vFf <(for i in </path/to/directory/a>/*.jpg ; do md5sum $i | awk {'print $1'}; done) <(find </path/to/directoryb/ -iname "*.jpg" | xargs md5sum)

Does the same thing as the first grep command, but uses the -v option to list only what does not match.

What you are looking for:

If all the files in directory 'a' is present, the second grep should not return any output.

Replace *.jpg with any extension that you may wish to search.

From the man page of grep:

   -f FILE, --file=FILE
          Obtain patterns from FILE, one per line.  The empty file contains zero patterns, and therefore matches nothing.
          (-f is specified by POSIX.)

    -F, --fixed-strings
          Interpret PATTERN as a list of fixed strings, separated by newlines, any of which is to  be  matched.   (-F  is
          specified by POSIX.)

-v, --invert-match
              Invert the sense of matching, to select non-matching lines. 
  • This answer as currently written this will not correctly handle filenames with spaces. – frederickjh Dec 14 '20 at 13:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.