I'm trying to find what files don't exist by a checksum of their content. I have two directories /foo and /bar, both of these directories represent arbitrary states on the system, I want to find all files in root /bar that do not exist in root /foo (recursively). A dictionary of /bar/** with

CheckSum -> filepath

less the dictionary of /foo/** with

CheckSum -> filepath  

I can write this using md5sum or something, but there must already be a utility that does this.

Just to be clear, if two files are identical except by location (name and path) I want them to be identified as the same.

Test case

Let's create a tree with some test data,

/tmp/foo
└── myFile (duplicate of /tmp/bar/quz/asdf/otherFileName)
/tmp/bar
├── qaz
│   └── findMe
└── quz
    └── asdf
        └── otherFileName (duplicate of /tmp/foo/myFile)

Script for creation,

mkdir -p /tmp/foo /tmp/bar/quz/asdf /tmp/bar/qaz

# One file that exists in both locations
dd if=/dev/urandom of=/tmp/foo/myFile bs=1k count=10
cp /tmp/foo/myFile /tmp/bar/quz/asdf/otherFileName

# One file (findMe) that exists in only /mpt/bar/
dd if=/dev/urandom of=/tmp/bar/qaz/findMe bs=1k count=10

So the program comparing /tmp/foo and /tmp/bar should find /tmp/bar/qaz/findMe

  • Is the checksum that important? finding all files in bar that are not in foo or that are different in foo (that's what you want, right?) is a matter of roughly one find command. – xhienne Dec 16 '16 at 19:39
  • What are you suggesting? – Evan Carroll Dec 16 '16 at 19:43

I'd generate an md5sum listing (a column for sum, and a column for filename) for both trees and pipe the result into a script (say awk or perl) which can use the sum as a hash for the filenames, and make the script report filenames which have the same hash value.

That's more than a one-liner, and is something frequently asked, frequently solved (a web search for duplicate files by content comes up with more than a million hits).

You might find this helpful:

  • I've written it in multiple different languages myself, I'm just wondering if there is any reasonably ubiquitous unix command that can do this. – Evan Carroll Dec 16 '16 at 21:16

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.