Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I need to write a bash script that finds duplicate files, but in addition to md5sum it needs to use cmp to do the final check. I tried the command below in my shell. How can I add a loop and put it into a bash script?

find . -type f | xargs -I%  md5sum % | sort | uniq -w32 -D| cut  -f3 -d' ' 
share|improve this question
md5sum gives you a practical (though not theoretic) certainty of file identity. So, why do you want to consume time performing a further control? –  enzotib Oct 5 '11 at 12:30
because this is my homework and I have to check also with cmp –  el ab Oct 5 '11 at 12:34
@el, you only want to to use cmp if the MD5 checksums are the same. Also, I would not try to do this in a one-liner. –  Kusalananda Oct 5 '11 at 22:15
add comment

1 Answer

up vote 6 down vote accepted

Since you state that this is homework, I'll give you a set of hints rather than a solution.

You can start with something like this:

find . -type f|xargs -I%  md5sum % |sort \
 |uniq -w32 -D|cut -f3 -d' '|while read filename ; do

   # code here


Inside this while loop, filename will contain the name of one of the files detected by your pipeline.

The idea is to compare each one with the previous files. To do that, at the end of the loop, add the current file to an array:

find . -type f|xargs -I%  md5sum % |sort \
 |uniq -w32 -D|cut -f3 -d' '|while read filename ; do

 # code here


All you need now is to fill in #code here :-) You have at your disposal:

  1. the file that needs comparing: $filename
  2. the files it needs to be compared to: the files array
  3. the number of files it should be compared against $count (watch out for off-by-one errors)

You can sort all this out with a while loop, comparing the files $filename and ${files[$some_counter]} at each step.

To make this better, you should only check files with the same md5sum against each other. For that, you could:

  1. remove the cut from your pipeline
  2. change the read to read in both the md5 sum and the filename
  3. reset the files array when you detect when you're moving on to another md5 sum

This will need one more variable to keep track of the "current" md5 sum.

You can make it even better by avoiding redundant comparisons. For that, you'll need to add a bit more logic to avoid adding a file to files if it is identical to one already in that array.

And you can do without the $count variable.

And something very important you should check (and fix if necessary): make sure your script works with filenames or directory names that contain spaces.

Read up on bash arrays.

share|improve this answer
add comment

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.