how to compare all files inside ONE directory in bash?

Question

example given:

dir1/
 file1
 file2
 file3
 file4

• how do you compare file1-file2, file1-file3, file1-file4 then file2->file3, file2->file4 and so on. In this case the name is file_number, but could be any name.
• basically make a diff but NOT ony one file to others, but compare all to all.

i have been trying with:

for f in $(find ./ -type f | sort | uniq); do

    compare=$(diff -rq "$f" "$1" &>/dev/null)
    if [[ $compare -eq $? ]]; then
        echo "$(basename "$f") yes equal $1"
    else
        echo "$(basename "$f") no equal $1"
    fi
done

returns

./file1 yes equal ./file1
./file2 no equal ./file1
./file3 yes equal ./file1
./script no equal ./file1
./sd no equal ./file1

• is only comparing any file number with file1
• I'm thinking needs another loop, but now I'm stock[like bubble sort algo]
• how to make another IF STATEMENT for stop comparing file1 yes equal file1 [same file]

Answer 1

Yes, you do need two loops. But it looks like you don't need diff because you're discarding the diff output to /dev/null, you can use cmp instead.

For example:

#!/bin/bash

# Read the list of files into an array called $files - we're iterating over
# the same list of files twice, but there's no need to run find twice.
#
# This uses NUL as the separator between filenames, so will work with
# filenames containing ANY valid character. Requires GNU find and GNU
# sort or non-GNU versions that support the `-print0` and `-z` options.
#
# The `-maxdepth 1` predicate prevents recursion into subdirs, remove it
# if that's not what you want.
mapfile -d '' -t files < <(find ./ -maxdepth 1 -type f -print0 | sort -z -u)

for a in "${files[@]}" ; do
  for b in "${files[@]}" ; do
    if [ ! "$a" = "$b" ] ; then
      if cmp --quiet "$a" "$b" ; then
        echo "Yes. $a is equal to $b"
      else
        echo "No. $a is not equal to $b"
      fi
    fi
  done
done

---

BTW, this will generate a lot of output (n × (n-1) output lines, where n is the number of files). Personally, I'd delete or comment out the else and echo "No...." lines because files that are the same as other files are likely to be much rarer than files that are unique.

---

Also note that if files abc and xyz are the same then it will compare them twice and print Yes both times:

Yes. ./abc is equal to ./xyz
Yes. ./xyz is equal to ./abc

There are several ways to prevent that from happening, probably the easiest is to use an associative array to keep track of files we've compared against each other. e.g.

#!/bin/bash

# Read the list of files into an array called $files
mapfile -d '' -t files < <(find ./ -maxdepth 1 -type f -print0 | sort -z -u)

# declare that $seen is an associative array
declare -A seen

for a in "${files[@]}" ; do
  for b in "${files[@]}" ; do
    if [ ! "$a" = "$b" ] && [ -z "${seen[$a$b]}" ] && [ -z "${seen[$b$a]}" ] ; then
      seen[$a$b]=1
      seen[$b$a]=1
      if cmp --quiet "$a" "$b" ; then
        echo "Yes. $a is equal to $b"
      #else
      #  echo "No. $a is not equal to $b"
      fi
    fi
  done
done

Answer 2

Comparing many files in pairs grows cumbersome very quickly. Ten files requires 45 comparisons. 100 files takes almost 5000. My test set is 595 files (totalling 10 GB), which would require over 175,000 pair comparisons. (This set is 9 dated archive directories, containing metadata of full and partial backups from various partitions.)

The method is to calculate a checksum for every file (taking a total of just over two minutes on a Laptop), and then grouping the files by checksum using awk (taking under a second).

The checksum process is this Bash fragment:

#.. Checksum all the files, and show some statistics.
[ x ] && {
    time ( cd "${BackUpDir}" && cksum */* ) > "${CkSums}"
    du -s -h "${BackUpDir}"
    head -n 3 "${CkSums}"
    awk '{ Bytes += $2; }
        END { printf ("Files %d, %.2f MB\n", FNR, Bytes / (1024 * 1024)); }
        ' "${CkSums}"
}

which shows this log.

$ ./fileGroup
real    2m5.139s
user    1m3.141s
sys 0m24.685s
9.8G    /media/paul/WinData/tarMETADATA
2288228966 156844 20220107_002000/02_History.tld
1812380507 156992 20220107_002000/02_History.toc
3028427874 1000411 20220107_002000/06_TechHist.tld
Files 565, 10001.10 MB

real    0m0.024s  #.. (Runtime of the awk component)
user    0m0.018s
sys 0m0.008s

An extract of the results:

Group of 5 files for cksum 1459775330
    20220319_114500/lib64.tld
    20220401_182500/lib64.tld
    20220407_192000/lib64.tld
    20220503_190500/lib64.tld
    20220503_232500/lib64.tld

Group of 3 files for cksum 2937156162
    20220407_192000/sbin.tld
    20220503_190500/sbin.tld
    20220503_232500/sbin.tld

Group of 2 files for cksum 3291901599
    20220503_190500/30_Photos.tld
    20220503_232500/30_Photos.tld

Counted 304 non-grouped files.

The Bash script is around 60 lines, of which 30 are the embedded awk script (I am unaware of any GNU/specific syntax required).

#! /bin/bash --

#.. Determine groups of identical files.

BackUpDir="/media/paul/WinData/tarMETADATA"
CkSums="Cksum.txt"
Groups="Groups.txt"

#.. Group all the files by checksum, and report them.

fileGroup () {

    local Awk='
BEGIN { Db = 0; reCut2 = "^[ ]*[^ ]+[ ]+[^ ]+[ ]+"; }
{ if (Db) printf ("\n%s\n", $0); }

#.. Add a new cksum value.
! (($1,0) in Fname) {
    Cksum[++Cksum[0]] = $1;
    if (Db) printf ("Added Cksum %d value %s.\n", 
        Cksum[0], Cksum[Cksum[0]]);
    Fname[$1,0] = 0;
}
#.. Add a filename.
{
    Fname[$1,++Fname[$1,0]] = $0;
    sub (reCut2, "", Fname[$1,Fname[$1,0]]);
    if (Db) printf ("Fname [%s,%s] is \047%s\047\n",
        $1, Fname[$1,0], Fname[$1, Fname[$1,0]]);
}
#.. Report the identical files, grouped by checksum.
function Report (Local, k, ke, cs, j, je, Single) {
    ke = Cksum[0];
    for (k = 1; k <= ke; ++k) {
        cs = Cksum[k];
        je = Fname[cs,0];
        if (je < 2) { ++Single; continue; }
        printf ("\nGroup of %d files for cksum %s\n", je, cs);
        for (j = 1; j <= je; ++j) printf ("    %s\n", Fname[cs,j]);
    }
    printf ("\nCounted %d non-grouped files.\n", Single);
}
END { Report( ); }
'
    awk -f <( printf '%s' "${Awk}" )
}

#### Script Body Starts Here.

    #.. Checksum all the files, and show some statistics.
    [ x ] && {
        time ( cd "${BackUpDir}" && cksum */* ) > "${CkSums}"
        du -s -h "${BackUpDir}"
        head -n 3 "${CkSums}"
        awk '{ Bytes += $2; }
            END { printf ("Files %d, %.2f MB\n", FNR, Bytes / (1024 * 1024)); }
            ' "${CkSums}"
    }

    #.. Analyse the cksum data.
    time fileGroup < "${CkSums}" > "${Groups}"