Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a number of files in a directory, and I want to check that they are all unique. For simplicity, let's say I have three files: foo.txt, bar.txt and baz.txt. If I run this loop, I will check them all against each other:

$ for f in ./*; do for i in ./*; do diff -q "$f" "$i"; done; done
Files bar.txt and baz.txt differ
Files bar.txt and foo.txt differ
Files baz.txt and bar.txt differ
Files baz.txt and foo.txt differ
Files foo.txt and bar.txt differ
Files foo.txt and baz.txt differ

For the hundreds of files I want to deal with, this would become pretty unreadable; it would be better to list the files that do match, and then I can look over the list quickly and make sure that files are only matching themselves. From the manpage, I would have thought that the -s option would accomplish this:

$ for f in ./*; do for i in ./*; do diff -s "$f" "$i"; done; done
Files bar.txt and bar.txt are identical
Files baz.txt and baz.txt are identical
Files foo.txt and foo.txt are identical

...however, in fact it also prints out the whole contents of any files that differ. Is there any way to suppress this behaviour, so I only get the behaviour above?

Alternatively, is there some other tool that can accomplish this?

share|improve this question

4 Answers 4

up vote 3 down vote accepted

If you just want to check whether two files are identical or not, use cmp. To get an output only for identical files, you could use

for f in ./*; do for i in ./*; do cmp -s "$f" "$i" && echo "Files $f and $i are identical"; done; done

diff tries to produce a short, human-readable list of the differences, and this can take quite a lot of time, so avoid the overhead if you don't need it.

share|improve this answer

If you need to find identical files in a list, first sort them by size, for example with

ls -S

then for each group of identical sized files, run md5sum on them to see easily which are identical to which.

For big files, could be quicker to first checksum only a short piece of the whole file:

dd if=file bs=512 count=1 | md5sum

and then perform a full checksum only on suspect files.

share|improve this answer

This should do the trick:

diff -rs dir1 dir2 | egrep '^Files .+ and .+ are identical$'

where dir1 and dir2 are your two directories.

If you'd like to only print the matching directories from dir1:

diff -rs dir1 dir2 | egrep '^Files .+ and .+ are identical$' | awk -F '(Files | and | are identical)' '{print $2}'

And likewise, if you'd like to only print the matching directories from dir2:

diff -rs dir1 dir2 | egrep '^Files .+ and .+ are identical$' | awk -F '(Files | and | are identical)' '{print $3}'
share|improve this answer

The fastest tool written for that purpose is fdupes (it is available in the package repos of Fedora and Ubuntu and …)

Usage:

fdupes -r dir1 dir2
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.