2

I have got a file list (<10) as output of some command. For example:

$ find foo* -name bar*Test.groovy

foofoo/foo-tasks/src/test/groovy/foo_main/store/barEarthTest.groovy
foo-co/foo-tasks/src/test/groovy/foo_main/store/barMercuryTest.groovy
foo_ba/foo-tasks/src/test/groovy/foo_main/store/barMarsTest.groovy
foo_tr/foo-tasks/src/test/groovy/foo_main/store/barMarsTest.groovy
foo_P1/foo-tasks/src/test/groovy/foo_main/store/barJupiterTest.groovy
foo_P2/foo-tasks/src/test/groovy/foo_main/store/barTatooineTest.groovy

I would like to know which of these files are identical and which are different. I do not care what exact difference is.

What is the way to get this information in more convenient form?

5

There are several codes that do much of this work for you, for example: fdupes jdupes rdfind duff

A few years ago I posted comparison runs of fdupes and rdfind at http://www.linuxforums.org/forum/programming-scripting/195467-find-same-size-file.html#post924892

Here are some details about these 4:

fdupes  finds duplicate files in a given set of directories (man)
Path    : /usr/bin/fdupes
Version : 1.51
Type    : ELF 64-bit LSB executable, x86-64, version 1 (SYS ...)
Help    : probably available with -h,--help
Repo    : Debian 8.9 (jessie) 
Home    : http://code.google.com/p/fdupes/ (pm)

jdupes  finds and performs actions upon duplicate files (man)
Path    : ~/executable/jdupes
Version : 1.5.1 (2016-11-01)
Type    : ELF 64-bit LSB executable, x86-64, version 1 (SYS ...)
Home    : https://github.com/jbruchon/jdupes (doc)

rdfind  finds duplicate files (man)
Path    : /usr/bin/rdfind
Version : 1.3.4
Type    : ELF 64-bit LSB executable, x86-64, version 1 (SYS ...)
Repo    : Debian 8.9 (jessie) 
Home    : http://rdfind.pauldreik.se/ (pm)

duff    duplicate file finder (man)
Path    : /usr/bin/duff
Version : 0.5.2
Type    : ELF 64-bit LSB executable, x86-64, version 1 (SYS ...)
Repo    : Debian 8.9 (jessie) 
Home    : http://duff.sourceforge.net/ (pm)

Bes wishes ... cheers, drl

3

You could do:

find foo* -name 'bar*Test.groovy' -type f -exec cksum {} + | sort

(assuming file paths don't contain newline characters) which would give you a checksum (and size) for each file, letting you easily see which are the same.

You can replace cksum with md5sum/shasum/sha512sum/sha3sum if available and you worry about potential checksum collisions.

3

Use return value of

diff file1 file2 >/dev/null

as it returns zero when files are the same and nonzero when files differ.

Compare the files in two nested for cycles. Something as:

for file1 in $(COMMAND)
do
    for file2 in $(COMMAND)
    do
        if ! diff "${file1}" "${file2}" > /dev/null 2>&1
        then
            echo "${file1} and ${file2} differ"
        fi
    done
done

Or, as Stéphane Chazelas suggested, you can simplify it:

for file1 in $(COMMAND)
do
    for file2 in $(COMMAND)
    do
        diff -q "${file1}" "${file2}"
    done
done

as diff -q prints the message from the first approach itself.

8
  • 1
    Don't use the test command ([]) when it's not necessary. Jan 26 '18 at 10:04
  • 1
    Use the double quotation on all variables when possible. Jan 26 '18 at 10:06
  • @Vlastimil: thank you, the test command syntax I didn't know and the quotation I forgot.
    – Adam Trhon
    Jan 26 '18 at 10:15
  • @Vlastimil BTW, the do and then on newline is a matter of style or is there any practical reason?
    – Adam Trhon
    Jan 26 '18 at 10:16
  • 1
    If you don't care about what the differences are, using cmp or diff -q would be a lot more efficient that redirecting the ouput of diff to /dev/null Jan 26 '18 at 13:13
1

Since you aren't interested in what the differences are, and you want all files compared against each other the most computationally efficient (see O-notation) method is going to be to perform a single md5sum action on each file, sort the output by the md5sum and mark duplicates, which will be consecutive.

find foo* -name bar*Test.groovy | md5sum | sort | \ awk 'key==$1{print file " is identical to "$2}{key=$1;file=$2}'

0

Install fdupes package using below yum install fdupes*

Then use the below command to find the identical files or duplicate files means the files which having same content

fdupes path

In current path it will display all files which are identical or duplicate

0

Use rmlint and pipe the results of find to it:

find foo* -name "bar*Test.groovy" | rmlint -T duplicates -

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.