Recently I have the need to delete a lot of duplicates. I am merging three or four filesystems, and I want the space to be used economically. At first, fdupes seemed like it was the best tool for the job, but I am increasingly running into limitations.

Consider the command fdupes -rdN somedirectory/. This makes a hash of all of the files in the subdirectories of somedirectory.

And when it encounters duplicates, it deletes them, so that there is only one copy of everything.

But what if I want to keep somedirectory/subdirectory1/somefile and there are, in fact, four duplicates, and the program encounters one of the duplicates first? Then it deletes somedirectory/subdirectory1/somefile, which I don't want.

I want to be able to specify, somehow, which duplicates to keep. And so far, none of the standard programs for dealing with duplicates (duff, FSLint) seem to allow automation of that kind of behavior. I'd prefer not to roll my own, so that's why I'm asking this question.

I'd like to be able to write something like

killdupes -rdN --keep=filesin,somedirectories,separated,by,commas somedirectory/

What about hardlinking the duplicate files together? That way the space is only used once, but they still exist in all the paths. The catch with this is that hardlinked files should be modified in place (they should only be modified deleting the file and recreating it with the new content). The other approach is to symlink the files together, although you have the same issue of deciding which the "primary" file is. This could be done with the following script (although note that this doesn't handle filenames containing spaces).

fdupes --quiet --recurse --sameline somedirectory/ | while read SOURCE DESTS; do
    for DEST in $DESTS; do
        ln -f $SOURCE $DEST

I didn't see this one anywhere else: Say what you want is this. You have /mnt/folder-tree-1 /mnt/folder-tree-2. You don't want to remove every dupe, but if a file exists in tree-2, and an identical file exists in tree-1 with the exact same path and name, remove it from tree-2.

Warning: this quite terse and if you try to copy-paste this with limited shell skills, be careful.

fdupes -rn /mnt/folder-tree-1/ /mnt/folder-tree-2/ > dupes-all.txt

fgrep /mnt/folder-tree-1/ dupes-all.txt | while read line
if grep -q "`echo $line | sed -e 's|^/mnt/folder-tree-1/|/mnt/folder-tree-2/|'`" dupes-all.txt
    echo rm \"$(echo $line | sed -e 's|^/mnt/folder-tree-1/|/mnt/folder-tree-2//|')\"
done > rm-v2-dupes.sh

Or all on one line:

fdupes -rn /mnt/folder-tree-1/ /mnt/folder-tree-2/ > dupes-all.txt; fgrep /mnt/folder-tree-1/ dupes-all.txt | while read line; do if grep -q "`echo $line | sed -e 's|^/mnt/folder-tree-1/|/mnt/folder-tree-2/|'`" dupes-all.txt; then echo rm \"$(echo $line | sed -e 's|^/mnt/folder-tree-1/|/mnt/folder-tree-2/|')\"; fi; done > rm-v2-dupes.sh

Afterwards, inspect and execute rm-v2-dupes.sh


While the functionality you seek is not available in stock fdupes, I forked fdupes (my fork is called jdupes) and added some features that can solve this problem under certain circumstances. For example, in the stated case where you wish to keep somedirectory/subdirectory1/somefile when auto-deleting duplicates (the d and N switches together) and there are no separate files immediately beneath somedirectory, jdupes can be fed each immediate subdirectory path with subdirectory1 first and the -O switch (which sorts files by command-line parameter order first):

jdupes -nrdNO somedirectory/subdirectory1 somedirectory/subdirectory2 somedirectory/subdirectory3

This will auto-delete all but one file in a duplicate set and will guarantee that if the set contains a file in somedirectory/subdirectory1 it will be the first, thereby automatically becoming the preserved file in the set. There are still glaring limits to this approach such as the fact that another duplicate in somedirectory/subdirectory1 might be preserved instead of the one you wanted to keep, but in a good number of cases like yours, the jdupes parameter order option as a workaround is good enough.

In the near future, I plan to add a filtering system to jdupes that will enable a huge amount of control over inclusion/exclusion of files, preservation for -N actions, and application of such "filter stacks" on either a global or per-parameter basis. This feature is sorely needed; I envision something like this to "auto-delete non-zero duplicates recursively BUT always preserve somedirectory/subdirectory1/somefile as-is":

jdupes -nrdN --filter=preserve:somedirectory/subdirectory1/somefile somedirectory/


I had the same question. If you have many duplicates fdupes /my/directory/ -rdN keeps the file with the oldest modify date, or if several files have the same modify date, then the one found first.

If the modify date is not important to you, you can touch the files in the directory you want to keep. If you choose to touch them with the current date and time then fdupes -rdNi will keep the ones with the current date. Or you can touch the keep files with a date earlier than that of the ones you want deleted and use fdupes -rdN as normal.

If you need to keep the modify date, then you will need to use one of the other methods.


Just to add a twist to a previous answer. I've used the following code multiple times, slightly modifying a previous answer with a simple | grep to isolate the folder I want to delete from.

`fdupes -r -n -S /directory | grep /delete-from-directory | sed -r "s/^/rm \"/" | sed -r "s/$/\"/" >remove-duplicate-files.sh`

Again, this will create a sh file to delete all files listed, no commented lines. Of course you can still edit the file to comment out specific lines/files you want to keep.

Another hint for large directories is to run fdupes to a txt file, then experiment with | grep and | sed until I get the result I want.

`fdupes -r -n -S /directory > duplicate-files.txt`
`cat duplicate-files.txt | grep /delete-from-directory | sed -r "s/^/rm \"/" | sed -r "s/$/\"/" >remove-duplicate-files.sh`

Use sed to create a shell file that will contain commented out commands to delete each of your duplicate files:

fdupes -r -n -S /directory | sed -r "s/^/#rm \"/" | sed -r "s/$/\"/" >remove-duplicate-files.sh

The resulting remove-duplicate-files.sh file that we've just created will have each line commented out. Uncomment the files you want to delete. Then run sh remove-duplicate-files.sh. Voila!


Well, if you don't want to delete files only in certain directories, it's as simple as this:

fdupes -S /directory|sed '/^$/d' |sed -r "s/^[0-9]/#&/" > duple_list

python exclude_duplicates.py -f /path/to/dupe_list --delimiter='#' --keep=/full/path/to/protected/directory1,/full/path/to/protected/directory2\ with\ spaces\ in\ path >remove-duplicate-files-keep-protected.sh

Where exclude_duplicates.py is:

# -*- coding: utf-8 -*-
# exclude_duplicates.py
Provided a list of duplicates, such as fdupes or fslint output,
generate a bash script that will have all duplicates in protected
directories commented out. If none of the protected duplicates are
found in a set of the same files, select a random unprotected
duplicate for preserving.
Each path to a file will be transformed to an `rm "path"` string which
will be printed to standard output.     

from optparse import OptionParser
parser = OptionParser()
parser.add_option("-k", "--keep", dest="keep",
    help="""List of directories which you want to keep, separated by commas. \
        EXAMPLE: exclude_duplicates.py --keep /path/to/directory1,/path/to/directory\ with\ space\ in\ path2""",
parser.add_option("-d", "--delimiter", dest="delimiter",
    help="Delimiter of duplicate file groups", metavar="delimiter"
parser.add_option("-f", "--file", dest="file",
    help="List of duplicate file groups, separated by delimiter, for example, fdupes or fslint output.", metavar="file"

(options, args) = parser.parse_args()
directories_to_keep = options.keep.split(',')
file = options.file
delimiter = options.delimiter

pretty_line = '\n#' + '-' * 35
print '#/bin/bash'
print '#I will protect files in these directories:\n'
for d in directories_to_keep:
    print '# ' + d
print pretty_line

protected_set = set()
group_set = set()

def clean_set(group_set, protected_set, delimiter_line):
    not_protected_set = group_set - protected_set
    while not_protected_set:
        if len(not_protected_set) == 1 and len(protected_set) == 0:
            print '#randomly selected duplicate to keep:\n#rm "%s"' % not_protected_set.pop().strip('\n')
            print 'rm "%s"' % not_protected_set.pop().strip('\n')
    for i in protected_set: print '#excluded file in protected directory:\n#rm "%s"' % i.strip('\n')
    print '\n#%s' % delimiter_line
file = open(file, 'r')
for line in file.readlines():
    if line.startswith(delimiter):
        clean_set(group_set, protected_set, line)
        group_set, protected_set = set(), set()
        group_set = group_set|{line}
        for d in directories_to_keep:
            if line.startswith(d): protected_set = protected_set|{line}
    if line: clean_set(group_set, protected_set, line)

The resulting remove-duplicate-files-keep-protected.sh file that we've just created will have all files from protected directories commented out. Open this file in your favorite text editor, check that everything is OK. Then run it. Voila (sic)!

  • i thought of this, but it's not automated enough. stupidly, i caused data loss with this method when dealing with duplicates spaced across multiple filesystems... there's no way to assign a priority, given the output of fdupes. basically i would have had to trawl through 10000 files by hand in order to prevent that data loss... so, no thanks... in fact, that data loss is the very reason i asked this question. – ixtmixilix Mar 21 '12 at 21:12
  • @ixtmixilix, well, manual method is dependent on user attentiveness, here's nothing new. If you want something more automated, checkout an updated answer above. – Ivan Kharlamov Mar 22 '12 at 14:10

What about something like this?


PREFERRED_DIRS=("somedir/subdir1" "somedir/subdir2")
DUPE_FILE=/tmp/`basename $0`_found-duplicates

delete_dupes() {
    while read line ; do
        if [ -n "$line" ] ; then
            for pdir in "${PREFERRED_DIRS[@]}" ; do
                if [[ $line == $pdir/* ]] ; then
            if ! $matched ; then
                rm -v "$line"
    done < "$DUPE_FILE"

cleanup() {
    rm -f $DUPE_FILE

trap cleanup EXIT

# get rid of normal dupes, preserve first & preserve preferred
fdupes -rf "$DUPE_SEARCH_DIR" > $DUPE_FILE

# get rid of preserve dupes, preserve preferred
fdupes -r "$DUPE_SEARCH_DIR" > "$DUPE_FILE"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.