I'm trying to find what files don't exist by a checksum of their content. I have two directories /foo and /bar, both of these directories represent arbitrary states on the system, I want to find all files in root /bar that do not exist in root /foo (recursively). A dictionary of /bar/** with

CheckSum -> filepath

less the dictionary of /foo/** with

CheckSum -> filepath  

I can write this using md5sum or something, but there must already be a utility that does this.

Just to be clear, if two files are identical except by location (name and path) I want them to be identified as the same.

Test case

Let's create a tree with some test data,

└── myFile (duplicate of /tmp/bar/quz/asdf/otherFileName)
├── qaz
│   └── findMe
└── quz
    └── asdf
        └── otherFileName (duplicate of /tmp/foo/myFile)

Script for creation,

mkdir -p /tmp/foo /tmp/bar/quz/asdf /tmp/bar/qaz

# One file that exists in both locations
dd if=/dev/urandom of=/tmp/foo/myFile bs=1k count=10
cp /tmp/foo/myFile /tmp/bar/quz/asdf/otherFileName

# One file (findMe) that exists in only /mpt/bar/
dd if=/dev/urandom of=/tmp/bar/qaz/findMe bs=1k count=10

So the program comparing /tmp/foo and /tmp/bar should find /tmp/bar/qaz/findMe

  • Is the checksum that important? finding all files in bar that are not in foo or that are different in foo (that's what you want, right?) is a matter of roughly one find command.
    – xhienne
    Dec 16, 2016 at 19:39
  • What are you suggesting? Dec 16, 2016 at 19:43

1 Answer 1


I'd generate an md5sum listing (a column for sum, and a column for filename) for both trees and pipe the result into a script (say awk or perl) which can use the sum as a hash for the filenames, and make the script report filenames which have the same hash value.

That's more than a one-liner, and is something frequently asked, frequently solved (a web search for duplicate files by content comes up with more than a million hits).

You might find this helpful:

  • 1
    I've written it in multiple different languages myself, I'm just wondering if there is any reasonably ubiquitous unix command that can do this. Dec 16, 2016 at 21:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .