Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top


[USER@NOTEBOOK ~/proba] find . -type f | xargs -I "{}" md5sum "{}" | sort
18b17ef2e55f6fd2deb044943a8a769d  ./yes
2b00042f7481c7b056c4b410d28f33cf  ./tt.txt
2b00042f7481c7b056c4b410d28f33cf  ./tx.txt
698458eb994fafdf56c1f63295c942ad  ./laksjdasdff.txt
764efa883dda1e11db47671c4a3bbd9e  ./ize2.txt
764efa883dda1e11db47671c4a3bbd9e  ./ize3.txt
764efa883dda1e11db47671c4a3bbd9e  ./ize.txt
a787d6f5ce5deb6e2e4b004f95da5655  ./laksjdf.txt
[USER@NOTEBOOK ~/proba] 


[USER@NOTEBOOK ~/proba] find . -type f | xargs -I "{}" md5sum "{}" | sort | SOMEMAGIC
tt.txt tx.txt
ize2.txt ize3.txt ize.txt
[USER@NOTEBOOK ~/proba] 

So I need a "SOMEMAGIC" that only outputs filenames if the md5sum matches (one line per md5sum). (It will be a deduplication script.. )

share|improve this question
If what you want is a deduplication script, they already exist. – Gilles Jun 25 '12 at 0:46
up vote 2 down vote accepted

The first script is an extension of Sergei Lomakov's answer, but it was a bit too long for a comment. It caters for spaces in filenames, and quotes each "name". This method does not requirea a sort step.

The second script is another way, using sort + awk, but without the array processing of the first method. Of course, it loses the input sequence, if that's an issue for you, (but its okay for the question as it uses a sort step anyhow).

Both methods also use sed to introduce \x00 as a field separator; to enable whitespace handling.

method 1, arrays in awk.

find . -type f | 
  xargs -I {} md5sum {} |
    sed 's/ [ *]/\x00/' | # "  "==text, " *"==binary
      awk -F"\x00" '{
             if( md5s[$1] == "" ) {sep=""} else {sep=FS} 
             md5s[$1]=md5s[$1] sep $2 }
        END{ for(md5 in md5s ) {
               if( (split(md5s[md5], names, FS)) > 1 ) {
                 for( ix in names ) {
                   printf "%s%s", sep, names[ix]
                   sep="\" \"" }
                 print "\"" } } }'

method 2, sort + awk.

find . -type f | 
  xargs -I {} md5sum {} |
    sort |sed 's/ [ *]/\x00/' | # "  "==text, " *"==binary
      awk -F"\x00" '{
             if (pkey!=$1) { ct=-1; pkey=$1; pnam=$2 }
             else{if (++ct) { printf(" \"%s\"",$2) }
                  else { printf("%s\"%s\" \"%s\"",nl,pnam,$2)
                         nl="\n" } } }
        END{ print "" }' 


"./tt.txt" "./tx.txt"
"./ize2.txt" "./ize3.txt" "./ize.txt"
share|improve this answer
handling spaces in filenames is better, I marked this as answer instead :) – gasko peter Jun 29 '12 at 11:06

Use this "magic" :)

find . -type f | xargs -I "{}" md5sum "{}" | awk '{count[$1]=count[$1]" "$2}END{for(j in count) if( (split(count[j], A)) > 1) print count[j]}' | sed -e 's/\.\///g'
share|improve this answer
Althought it does handle the sample filenames given in the question, it doesn't handle whitespace in filenames (via $2 in awk)... and just by the way, you don't need the quotes around {} (but neither do they upset anything). {} is already a bash word, and thereby is not subject to the shell's word splitting (but once the filename is within awk's scope, it is subject to awk's field seperator, which is whitespace. – Peter.O Jun 23 '12 at 18:47
Thank you very much for improvements! – Sergei Lomakov Jun 24 '12 at 9:46

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.