Given a list of files, some duplicates, some not, show checksum of only the duplicates

Question

There must be an "easy" way to do this, but I can't figure out what it is.

Assume you have a plain text "file.txt" which has lines in this format (md5 sums followed by filenames):

5ee434a2ebcf4c3c98ee07e9c1efddc0 foo.txt
365a6d8b18cab348d92db610dfc46264 bar.txt
ae42d992bf622bdc425d37b04ec9c2d5 mini.txt
b8e9ff5502d5dbe38b3fd5e3363caacf tyrion.txt
5ee434a2ebcf4c3c98ee07e9c1efddc0 imac.txt
542ed609dfc4d0cae44c4b7be6d66382 mba.txt
310ee92ebc69ed79c1837fc53983b7f8 mini luoma.txt
542ed609dfc4d0cae44c4b7be6d66382 tyrion final.txt

I would like to sort file.txt and have the output:

1. Only show me lines if the md5 sum indicates the files are duplicates
2. Put a blank line between each "group" of duplicates.

so it would look like this:

542ed609dfc4d0cae44c4b7be6d66382 mba.txt
542ed609dfc4d0cae44c4b7be6d66382 tyrion final.txt

5ee434a2ebcf4c3c98ee07e9c1efddc0 foo.txt
5ee434a2ebcf4c3c98ee07e9c1efddc0 imac.txt

(In the real case, it could be 2 duplicates or 10, or more.)

I'm guessing there might be a ruby or python guru out there who can figure this one out, but I'm open to pretty much any practical solution out there.

Answer 1

$ sort file.txt \
| grep -f <(cut -d' ' -f1 file.txt | sort | uniq -d) \
| awk 'last && last != $1 { printf "\n" }; { last=$1 ; print}'

542ed609dfc4d0cae44c4b7be6d66382 mba.txt
542ed609dfc4d0cae44c4b7be6d66382 tyrion final.txt

5ee434a2ebcf4c3c98ee07e9c1efddc0 foo.txt
5ee434a2ebcf4c3c98ee07e9c1efddc0 imac.txt

(Thanks to "cas" for the awk suggestion.)

(And thanks to "steeldriver" for noticing a bug.)

Answer 2

With sort and uniq

sort file.txt | uniq -w 32 --all-repeated=separate

542ed609dfc4d0cae44c4b7be6d66382 mba.txt
542ed609dfc4d0cae44c4b7be6d66382 tyrion final.txt

5ee434a2ebcf4c3c98ee07e9c1efddc0 foo.txt
5ee434a2ebcf4c3c98ee07e9c1efddc0 imac.txt

• -w 32 limits uniq comparison to the MD5 hash (so it considers equals lines with same MD5)
• --all-repeated=separate tells it to show duplicates, while separating groups with a blank line

Answer 3

With a perl Hash of Arrays:

$ perl -alne '
    push @{ $h{$F[0]} }, $_ 
    }{ 
    for $k (sort keys %h) {
      @a = @{ $h{$k} }; 
      print join "\n", @a, "" if $#a  > 0
    }
' file.txt
542ed609dfc4d0cae44c4b7be6d66382 mba.txt
542ed609dfc4d0cae44c4b7be6d66382 tyrion final.txt

5ee434a2ebcf4c3c98ee07e9c1efddc0 foo.txt
5ee434a2ebcf4c3c98ee07e9c1efddc0 imac.txt

Note that this prints a trailing blank line after the last record. The sort is optional.

---

Similar thing in GNU awk:

gawk '
  {a[$1][NR] = $0} 
  END {
    for(k in a){
      if(length(a[k]) > 1) {
        for(v in a[k]) print a[k][v];
        print "";
      }
    }
  }   
' file.txt