18

I have a number of files in a directory, and I want to check that they are all unique. For simplicity, let's say I have three files: foo.txt, bar.txt and baz.txt. If I run this loop, I will check them all against each other:

$ for f in ./*; do for i in ./*; do diff -q "$f" "$i"; done; done
Files bar.txt and baz.txt differ
Files bar.txt and foo.txt differ
Files baz.txt and bar.txt differ
Files baz.txt and foo.txt differ
Files foo.txt and bar.txt differ
Files foo.txt and baz.txt differ

For the hundreds of files I want to deal with, this would become pretty unreadable; it would be better to list the files that do match, and then I can look over the list quickly and make sure that files are only matching themselves. From the manpage, I would have thought that the -s option would accomplish this:

$ for f in ./*; do for i in ./*; do diff -s "$f" "$i"; done; done
Files bar.txt and bar.txt are identical
Files baz.txt and baz.txt are identical
Files foo.txt and foo.txt are identical

...however, in fact it also prints out the whole contents of any files that differ. Is there any way to suppress this behaviour, so I only get the behaviour above?

Alternatively, is there some other tool that can accomplish this?

4 Answers 4

21

This should do the trick:

diff -rs dir1 dir2 | egrep '^Files .+ and .+ are identical$'

where dir1 and dir2 are your two directories.

If you'd like to only print the matching directories from dir1:

diff -rs dir1 dir2 | egrep '^Files .+ and .+ are identical$' | awk -F '(Files | and | are identical)' '{print $2}'

And likewise, if you'd like to only print the matching directories from dir2:

diff -rs dir1 dir2 | egrep '^Files .+ and .+ are identical$' | awk -F '(Files | and | are identical)' '{print $3}'
3
  • This is exactly what I was looking for, thanks! Apr 14, 2015 at 16:39
  • 2
    Use diff -qrs when comparing large files (quiet suppresses printing differences) Dec 18, 2015 at 11:33
  • Also don't forget to preppend LANG=C or LC_ALL=C to get English output if you set a different locale.
    – Amessihel
    Sep 24, 2020 at 17:08
7

If you just want to check whether two files are identical or not, use cmp. To get an output only for identical files, you could use

for f in ./*; do for i in ./*; do cmp -s "$f" "$i" && echo "Files $f and $i are identical"; done; done

diff tries to produce a short, human-readable list of the differences, and this can take quite a lot of time, so avoid the overhead if you don't need it.

4

The fastest tool written for that purpose is fdupes (it is available in the package repos of Fedora and Ubuntu and …)

Usage:

fdupes -r dir1 dir2
1
  • fdupes compares all files, not only ones with the same name. So if you have two copies of the same 100-file folders, diff will do 100 comparisons, while fdupes does 10k. Since fdupes probably uses hashes, it's not 100x as slow as diff, but it's definitely not faster.
    – Dakkaron
    Sep 2, 2022 at 7:34
0

If you need to find identical files in a list, first sort them by size, for example with

ls -S

then for each group of identical sized files, run md5sum on them to see easily which are identical to which.

For big files, could be quicker to first checksum only a short piece of the whole file:

dd if=file bs=512 count=1 | md5sum

and then perform a full checksum only on suspect files.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .