Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Is there any fast method to delete duplicates of files based on any hash sum (i.e. SHA1 to be fast). Because I've got some mess in my music files.

share|improve this question
add comment

3 Answers 3

up vote 6 down vote accepted

There is package fdupes in linux (for example, it is present in debian repository). It uses md5sums and then a byte by byte comparison to find duplicate files within a set of directories. It also can delete dups with -d option, but I've never used that option. Also you can grep or sed from output files to delete and remove them from disk.

share|improve this answer
    
pacman told me that rmlint will be faster than fdupes, but I'm thankful for help :) –  Łukasz Niemier Dec 9 '11 at 19:01
add comment

You will find this unix/linux text utils pipeline very useful. This command first compares size of files and if size is same then only compares hash.

OR

You can use fdupes utility ro identify duplicate files.

Both solutions use md5 hash

share|improve this answer
1  
Yeah, I know that pipelines are very useful byt I was looking for some faster way, cause it will be more than 1000 files so hashing can take some time. –  Łukasz Niemier Dec 9 '11 at 19:00
add comment

liten2 http://code.google.com/p/liten2/ works also on mac osx!
Is a python script with quite a few very useful features, like dry runs, interactive delete and more...
And is very fast.

share|improve this answer
    
I don't like Python and I like native progs. Also I use Arch Linux so it isn't IMO great idea. But thank for interesting. –  Łukasz Niemier Dec 9 '11 at 19:19
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.