Is there any fast method to delete duplicates of files based on any hash sum (i.e. SHA1 to be fast). Because I've got some mess in my music files.


There is package fdupes in linux (for example, it is present in debian repository). It uses md5sums and then a byte by byte comparison to find duplicate files within a set of directories. It also can delete dups with -d option, but I've never used that option. Also you can grep or sed from output files to delete and remove them from disk.

  • pacman told me that rmlint will be faster than fdupes, but I'm thankful for help :) – Hauleth Dec 9 '11 at 19:01

You will find this unix/linux text utils pipeline very useful. This command first compares size of files and if size is same then only compares hash.


You can use fdupes utility ro identify duplicate files.

Both solutions use md5 hash

  • 1
    Yeah, I know that pipelines are very useful byt I was looking for some faster way, cause it will be more than 1000 files so hashing can take some time. – Hauleth Dec 9 '11 at 19:00

liten2 http://code.google.com/p/liten2/ works also on mac osx!
Is a python script with quite a few very useful features, like dry runs, interactive delete and more...
And is very fast.

  • I don't like Python and I like native progs. Also I use Arch Linux so it isn't IMO great idea. But thank for interesting. – Hauleth Dec 9 '11 at 19:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.