only output when first columns are the same

Question

INPUT:

[USER@NOTEBOOK ~/proba] find . -type f | xargs -I "{}" md5sum "{}" | sort
18b17ef2e55f6fd2deb044943a8a769d  ./yes
2b00042f7481c7b056c4b410d28f33cf  ./tt.txt
2b00042f7481c7b056c4b410d28f33cf  ./tx.txt
698458eb994fafdf56c1f63295c942ad  ./laksjdasdff.txt
764efa883dda1e11db47671c4a3bbd9e  ./ize2.txt
764efa883dda1e11db47671c4a3bbd9e  ./ize3.txt
764efa883dda1e11db47671c4a3bbd9e  ./ize.txt
a787d6f5ce5deb6e2e4b004f95da5655  ./laksjdf.txt
[USER@NOTEBOOK ~/proba] 

OUTPUT:

[USER@NOTEBOOK ~/proba] find . -type f | xargs -I "{}" md5sum "{}" | sort | SOMEMAGIC
tt.txt tx.txt
ize2.txt ize3.txt ize.txt
[USER@NOTEBOOK ~/proba] 

So I need a "SOMEMAGIC" that only outputs filenames if the md5sum matches (one line per md5sum). (It will be a deduplication script.. )

Answer 1

The first script is an extension of Sergei Lomakov's answer, but it was a bit too long for a comment. It caters for spaces in filenames, and quotes each "name". This method does not requirea a sort step.

The second script is another way, using sort + awk, but without the array processing of the first method. Of course, it loses the input sequence, if that's an issue for you, (but its okay for the question as it uses a sort step anyhow).

Both methods also use sed to introduce \x00 as a field separator; to enable whitespace handling.

method 1, arrays in awk.

find . -type f | 
  xargs -I {} md5sum {} |
    sed 's/ [ *]/\x00/' | # "  "==text, " *"==binary
      awk -F"\x00" '{
             if( md5s[$1] == "" ) {sep=""} else {sep=FS} 
             md5s[$1]=md5s[$1] sep $2 }
        END{ for(md5 in md5s ) {
               if( (split(md5s[md5], names, FS)) > 1 ) {
                 sep="\""  
                 for( ix in names ) {
                   printf "%s%s", sep, names[ix]
                   sep="\" \"" }
                 print "\"" } } }'

method 2, sort + awk.

find . -type f | 
  xargs -I {} md5sum {} |
    sort |sed 's/ [ *]/\x00/' | # "  "==text, " *"==binary
      awk -F"\x00" '{
             if (pkey!=$1) { ct=-1; pkey=$1; pnam=$2 }
             else{if (++ct) { printf(" \"%s\"",$2) }
                  else { printf("%s\"%s\" \"%s\"",nl,pnam,$2)
                         nl="\n" } } }
        END{ print "" }' 

Output

"./tt.txt" "./tx.txt"
"./ize2.txt" "./ize3.txt" "./ize.txt"

Answer 2

Use this "magic" :)

find . -type f | xargs -I "{}" md5sum "{}" | awk '{count[$1]=count[$1]" "$2}END{for(j in count) if( (split(count[j], A)) > 1) print count[j]}' | sed -e 's/\.\///g'