Diffing two directories recursively based on checksums?

Question

I'm trying to find what files don't exist by a checksum of their content. I have two directories /foo and /bar, both of these directories represent arbitrary states on the system, I want to find all files in root /bar that do not exist in root /foo (recursively). A dictionary of /bar/** with

CheckSum -> filepath

less the dictionary of /foo/** with

CheckSum -> filepath  

I can write this using md5sum or something, but there must already be a utility that does this.

Just to be clear, if two files are identical except by location (name and path) I want them to be identified as the same.

Test case

Let's create a tree with some test data,

/tmp/foo
└── myFile (duplicate of /tmp/bar/quz/asdf/otherFileName)
/tmp/bar
├── qaz
│   └── findMe
└── quz
    └── asdf
        └── otherFileName (duplicate of /tmp/foo/myFile)

Script for creation,

mkdir -p /tmp/foo /tmp/bar/quz/asdf /tmp/bar/qaz

# One file that exists in both locations
dd if=/dev/urandom of=/tmp/foo/myFile bs=1k count=10
cp /tmp/foo/myFile /tmp/bar/quz/asdf/otherFileName

# One file (findMe) that exists in only /mpt/bar/
dd if=/dev/urandom of=/tmp/bar/qaz/findMe bs=1k count=10

So the program comparing /tmp/foo and /tmp/bar should find /tmp/bar/qaz/findMe

Answer 1

I'd generate an md5sum listing (a column for sum, and a column for filename) for both trees and pipe the result into a script (say awk or perl) which can use the sum as a hash for the filenames, and make the script report filenames which have the same hash value.

That's more than a one-liner, and is something frequently asked, frequently solved (a web search for duplicate files by content comes up with more than a million hits).

You might find this helpful:

• Finding duplicate files?