Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Is it possible to find duplicate files on my disk which are bit to bit identical but have different file-names?

share|improve this question
2  
Note that any possible method of doing this will invariably have to compare every single file on your system to every single other file. So this is going to take a long time, even when taking shortcuts. –  Shadur Apr 4 '13 at 14:02
    
@Shadur if one is ok with checksums, it boils down to comparing just the hashes - which on most systems is of the order of 10^(5+-1) usually <64-byte entries. Of course, you have to read the data at least once. :) –  peterph Apr 4 '13 at 14:57
3  
@Shadur That's not true. You can reduce the time by checking for matching st_sizes, eliminating those with only one of the same, and then only calculating md5sums for matching st_sizes. –  Chris Down Apr 4 '13 at 16:36
1  
@Shadur even an incredibly silly approach disallowing any hash operations could do this in Θ(n log n) compares—not Θ(n²)—using any of several sort algorithms (based on file content). –  derobert Apr 4 '13 at 17:09
    
@ChrisDown Yes, size matching would be one of the shortcuts I had in mind. –  Shadur Apr 4 '13 at 19:38

6 Answers 6

fdupes can do this. From man fdupes:

Searches the given path for duplicate files. Such files are found by comparing file sizes and MD5 signatures, followed by a byte-by-byte comparison.

In Debian or Ubuntu, you can install it with apt-get install fdupes. In Fedora/Red Hat/CentOS, you can install it with yum install fdupes. On Arch Linux you can use pacman -S fdupes, and on Gentoo, emerge fdupes.

To run a check descending from your filesystem root, which will likely take a significant amount of time and memory, use something like fdupes -r /.

As asked in the comments, you can get the largest duplicates by doing the following:

fdupes -r . | {
    while IFS= read -r file; do
        [[ $file ]] && du "$file"
    done
} | sort

This will break if your filenames contain newlines.

share|improve this answer
    
Thanks. How can I filter out the largest dupe? How can I make the sizes human readable? –  student Apr 5 '13 at 9:31
    
@student: use something along the line of (make sure fdupes just outputs the filenames with no extra informatinos, or cut or sed to just keep that) : fdupes ....... | xargs ls -alhd | egrep 'M |G ' to keep files in Human readable format and only those with size in Megabytes or Gigabytes. Change the command to suit the real outputs. –  Olivier Dulac Apr 5 '13 at 12:27
2  
@OlivierDulac You should never parse ls. Usually it's worse than your use case, but even in your use case, you risk false positives. –  Chris Down Apr 5 '13 at 13:13
    
@student - Once you have the filenames, du piped to sort will tell you. –  Chris Down Apr 5 '13 at 13:14
    
@ChrisDown: it's true it's a bad habit, and can give false positives. But in that case (interactive use, and for display only, no "rm" or anything of the sort directly relying on it) it's fine and quick ^^ . I love those pages you link to, btw (been reading them since a few months, and full of many usefull infos) –  Olivier Dulac Apr 5 '13 at 14:05

Short answer: yes.

Longer version: have a look at the wikipedia fdupes entry, it sports quite nice list of ready made solutions. Of course you can write your own, it's not that difficult - hashing programs like diff, sha*sum, find, sort and uniq should do the job. You can even put it on one line, and it will still be understandable.

share|improve this answer

Another good tool is fslint:

fslint is a toolset to find various problems with filesystems, including duplicate files and problematic filenames etc.

Individual command line tools are available in addition to the GUI and to access them, one can change to, or add to $PATH the /usr/share/fslint/fslint directory on a standard install. Each of these commands in that directory have a --help option which further details its parameters.

   findup - find DUPlicate files

On debian-based systems, youcan install it with:

sudo apt-get install fslint

You can also do this manually if you don't want to or cannot install third party tools. The way most such programs work is by calculating file checksums. Files with the same md5sum almost certainly contain exactly the same data. So, you could do something like this:

find / -type f -exec md5sum {} \; > md5sums
gawk '{print $1}' md5sums | sort | uniq -d > dupes
while read d; do echo "---"; grep $d md5sums | cut -d ' ' -f 2-; done < dupes 

Sample output (the file names in this example are the same, but it will also work when they are different):

$ while read d; do echo "---"; grep $d md5sums | cut -d ' ' -f 2-; done < dupes 
---
 /usr/src/linux-headers-3.2.0-3-common/include/linux/if_bonding.h
 /usr/src/linux-headers-3.2.0-4-common/include/linux/if_bonding.h
---
 /usr/src/linux-headers-3.2.0-3-common/include/linux/route.h
 /usr/src/linux-headers-3.2.0-4-common/include/linux/route.h
---
 /usr/src/linux-headers-3.2.0-3-common/include/drm/Kbuild
 /usr/src/linux-headers-3.2.0-4-common/include/drm/Kbuild
---

This will be much slower than the dedicated tools already mentioned, but it will work.

share|improve this answer
    
It would be much, much faster to find any files with the same size as another file using st_size, eliminating any that only have one file of this size, and then calculating md5sums only between files with the same st_size. –  Chris Down Apr 4 '13 at 16:34
    
@ChrisDown yeah, just wanted to keep it simple. What you suggest will greatly speed things up of course. That's why I have the disclaimer about it being slow at the end of my answer. –  terdon Apr 4 '13 at 16:37

If you believe a hash function (here MD5) is collision free on your domain:

find $target -type f -exec md5sum '{}' + | sort | uniq --all-repeated --check-chars=32
 | cut --characters=35-

Wanna identical file names grouped? Write a simple script not_uniq.sh to format output:

#!/bin/bash

last_checksum=0
while read line; do
    checksum=${line:0:32}
    filename=${line:34}
    if [ $checksum == $last_checksum ]; then
        if [ ${last_filename:-0} != '0' ]; then
            echo $last_filename
            unset last_filename
        fi
        echo $filename
    else
        if [ ${last_filename:-0} == '0' ]; then
            echo "======="
        fi
        last_filename=$filename
    fi

    last_checksum=$checksum
done

Then change find command to use your script:

chmod +x not_uniq.sh
find $target -type f -exec md5sum '{}' + | sort | not_uniq.sh

This is basic idea. Probably you should change find if your file names containing some characters. (e.g space)

share|improve this answer

Have a look on this wikipedia link displaying available open source software for this task.

http://en.wikipedia.org/wiki/List_of_duplicate_file_finders

I will add that the GUI version of fslint iq very insteresting, allowing to use mask to select which files to delete. Very usefull to clean duplicated photos.

share|improve this answer
3  
It is better to provide actual information here and not just a link, the link might change and then the answer has no value left –  Anthon Jan 29 at 11:22

You can use DuplicateFilesDeleter, it is a fast way to find and delete all duplicate files.

share|improve this answer
1  
Perhaps you could elaborate this a bit more? –  slm Oct 16 at 16:46

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.