Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Is there any fast method to delete duplicates of files based on any hash sum (i.e. SHA1 to be fast). Because I've got some mess in my music files.

share|improve this question
up vote 6 down vote accepted

There is package fdupes in linux (for example, it is present in debian repository). It uses md5sums and then a byte by byte comparison to find duplicate files within a set of directories. It also can delete dups with -d option, but I've never used that option. Also you can grep or sed from output files to delete and remove them from disk.

share|improve this answer
pacman told me that rmlint will be faster than fdupes, but I'm thankful for help :) – Łukasz Niemier Dec 9 '11 at 19:01

You will find this unix/linux text utils pipeline very useful. This command first compares size of files and if size is same then only compares hash.


You can use fdupes utility ro identify duplicate files.

Both solutions use md5 hash

share|improve this answer
Yeah, I know that pipelines are very useful byt I was looking for some faster way, cause it will be more than 1000 files so hashing can take some time. – Łukasz Niemier Dec 9 '11 at 19:00

liten2 http://code.google.com/p/liten2/ works also on mac osx!
Is a python script with quite a few very useful features, like dry runs, interactive delete and more...
And is very fast.

share|improve this answer
I don't like Python and I like native progs. Also I use Arch Linux so it isn't IMO great idea. But thank for interesting. – Łukasz Niemier Dec 9 '11 at 19:19

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.