Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Is it possible to find duplicate files on my disk which are bit to bit identical but have different file-names?

share|improve this question
3  
Note that any possible method of doing this will invariably have to compare every single file on your system to every single other file. So this is going to take a long time, even when taking shortcuts. – Shadur Apr 4 '13 at 14:02
2  
@Shadur if one is ok with checksums, it boils down to comparing just the hashes - which on most systems is of the order of 10^(5+-1) usually <64-byte entries. Of course, you have to read the data at least once. :) – peterph Apr 4 '13 at 14:57
8  
@Shadur That's not true. You can reduce the time by checking for matching st_sizes, eliminating those with only one of the same, and then only calculating md5sums for matching st_sizes. – Chris Down Apr 4 '13 at 16:36
4  
@Shadur even an incredibly silly approach disallowing any hash operations could do this in Θ(n log n) compares—not Θ(n²)—using any of several sort algorithms (based on file content). – derobert Apr 4 '13 at 17:09
1  
(continuing) Does .../camera_pictures/vacation contain all pictures from London and .../vacation/london was just a subset you showed to your neighbour? Or are all files in the london directory also present in the vacation directory? What I have really wanted for many years is a two pane file manager which could take file duplicate information as input to open the respective directories and show/mark which files are identical/different/unique. That would be a power tool. – hlovdal Apr 5 '13 at 7:58

fdupes can do this. From man fdupes:

Searches the given path for duplicate files. Such files are found by comparing file sizes and MD5 signatures, followed by a byte-by-byte comparison.

In Debian or Ubuntu, you can install it with apt-get install fdupes. In Fedora/Red Hat/CentOS, you can install it with yum install fdupes. On Arch Linux you can use pacman -S fdupes, and on Gentoo, emerge fdupes.

To run a check descending from your filesystem root, which will likely take a significant amount of time and memory, use something like fdupes -r /.

As asked in the comments, you can get the largest duplicates by doing the following:

fdupes -r . | {
    while IFS= read -r file; do
        [[ $file ]] && du "$file"
    done
} | sort

This will break if your filenames contain newlines.

share|improve this answer
    
Thanks. How can I filter out the largest dupe? How can I make the sizes human readable? – student Apr 5 '13 at 9:31
    
@student: use something along the line of (make sure fdupes just outputs the filenames with no extra informatinos, or cut or sed to just keep that) : fdupes ....... | xargs ls -alhd | egrep 'M |G ' to keep files in Human readable format and only those with size in Megabytes or Gigabytes. Change the command to suit the real outputs. – Olivier Dulac Apr 5 '13 at 12:27
2  
@OlivierDulac You should never parse ls. Usually it's worse than your use case, but even in your use case, you risk false positives. – Chris Down Apr 5 '13 at 13:13
    
@student - Once you have the filenames, du piped to sort will tell you. – Chris Down Apr 5 '13 at 13:14
    
@ChrisDown: it's true it's a bad habit, and can give false positives. But in that case (interactive use, and for display only, no "rm" or anything of the sort directly relying on it) it's fine and quick ^^ . I love those pages you link to, btw (been reading them since a few months, and full of many usefull infos) – Olivier Dulac Apr 5 '13 at 14:05

Another good tool is fslint:

fslint is a toolset to find various problems with filesystems, including duplicate files and problematic filenames etc.

Individual command line tools are available in addition to the GUI and to access them, one can change to, or add to $PATH the /usr/share/fslint/fslint directory on a standard install. Each of these commands in that directory have a --help option which further details its parameters.

   findup - find DUPlicate files

On debian-based systems, youcan install it with:

sudo apt-get install fslint

You can also do this manually if you don't want to or cannot install third party tools. The way most such programs work is by calculating file checksums. Files with the same md5sum almost certainly contain exactly the same data. So, you could do something like this:

find / -type f -exec md5sum {} \; > md5sums
gawk '{print $1}' md5sums | sort | uniq -d > dupes
while read d; do echo "---"; grep $d md5sums | cut -d ' ' -f 2-; done < dupes 

Sample output (the file names in this example are the same, but it will also work when they are different):

$ while read d; do echo "---"; grep $d md5sums | cut -d ' ' -f 2-; done < dupes 
---
 /usr/src/linux-headers-3.2.0-3-common/include/linux/if_bonding.h
 /usr/src/linux-headers-3.2.0-4-common/include/linux/if_bonding.h
---
 /usr/src/linux-headers-3.2.0-3-common/include/linux/route.h
 /usr/src/linux-headers-3.2.0-4-common/include/linux/route.h
---
 /usr/src/linux-headers-3.2.0-3-common/include/drm/Kbuild
 /usr/src/linux-headers-3.2.0-4-common/include/drm/Kbuild
---

This will be much slower than the dedicated tools already mentioned, but it will work.

share|improve this answer
1  
It would be much, much faster to find any files with the same size as another file using st_size, eliminating any that only have one file of this size, and then calculating md5sums only between files with the same st_size. – Chris Down Apr 4 '13 at 16:34
    
@ChrisDown yeah, just wanted to keep it simple. What you suggest will greatly speed things up of course. That's why I have the disclaimer about it being slow at the end of my answer. – terdon Apr 4 '13 at 16:37

Short answer: yes.

Longer version: have a look at the wikipedia fdupes entry, it sports quite nice list of ready made solutions. Of course you can write your own, it's not that difficult - hashing programs like diff, sha*sum, find, sort and uniq should do the job. You can even put it on one line, and it will still be understandable.

share|improve this answer

If you believe a hash function (here MD5) is collision free on your domain:

find $target -type f -exec md5sum '{}' + | sort | uniq --all-repeated --check-chars=32
 | cut --characters=35-

Wanna identical file names grouped? Write a simple script not_uniq.sh to format output:

#!/bin/bash

last_checksum=0
while read line; do
    checksum=${line:0:32}
    filename=${line:34}
    if [ $checksum == $last_checksum ]; then
        if [ ${last_filename:-0} != '0' ]; then
            echo $last_filename
            unset last_filename
        fi
        echo $filename
    else
        if [ ${last_filename:-0} == '0' ]; then
            echo "======="
        fi
        last_filename=$filename
    fi

    last_checksum=$checksum
done

Then change find command to use your script:

chmod +x not_uniq.sh
find $target -type f -exec md5sum '{}' + | sort | not_uniq.sh

This is basic idea. Probably you should change find if your file names containing some characters. (e.g space)

share|improve this answer

Have a look on this wikipedia link displaying available open source software for this task.

http://en.wikipedia.org/wiki/List_of_duplicate_file_finders

I will add that the GUI version of fslint is very interesting, allowing to use mask to select which files to delete. Very useful to clean duplicated photos.

On Linux you can use:

- FSLint: http://www.pixelbeat.org/fslint/

- FDupes: https://en.wikipedia.org/wiki/Fdupes

- DupeGuru: https://www.hardcoded.net/dupeguru/

The 2 last work on many systems (windows, mac and linux) I 've not checked for FSLint

share|improve this answer
3  
It is better to provide actual information here and not just a link, the link might change and then the answer has no value left – Anthon Jan 29 '14 at 11:22
    
Wikipedia page is empty. – IDvoretskyi Sep 10 '15 at 9:01
    
yes, it has been cleaned, what a pity shake... – MordicusEtCubitus Dec 21 '15 at 16:23
    
I've edited it with these 3 tools – MordicusEtCubitus Dec 21 '15 at 16:30

protected by Community Jan 14 at 12:14

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.