How to delete duplicates of files in directory and subdirs?

Question

Is there any fast method to delete duplicates of files based on any hash sum (i.e. SHA1 to be fast). Because I've got some mess in my music files.

Answer 1

There is package fdupes in linux (for example, it is present in debian repository). It uses md5sums and then a byte by byte comparison to find duplicate files within a set of directories. It also can delete dups with -d option, but I've never used that option. Also you can grep or sed from output files to delete and remove them from disk.

Answer 2

You will find this unix/linux text utils pipeline very useful. This command first compares size of files and if size is same then only compares hash.

OR

You can use fdupes utility ro identify duplicate files.

Both solutions use md5 hash

Answer 3

liten2 http://code.google.com/p/liten2/ works also on mac osx!
Is a python script with quite a few very useful features, like dry runs, interactive delete and more...
And is very fast.