Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Recently I have the need to delete a lot of duplicates. I am merging three or four filesystems, and I want the space to be used economically. At first, fdupes seemed like it was the best tool for the job, but I am increasingly running into limitations.

Consider the command fdupes -rdN somedirectory/. This makes a hash of all of the files in the subdirectories of somedirectory.

And when it encounters duplicates, it deletes them, so that there is only one copy of everything.

But what if I want to keep somedirectory/subdirectory1/somefile and there are, in fact, four duplicates, and the program encounters one of the duplicates first? Then it deletes somedirectory/subdirectory1/somefile, which I don't want.

I want to be able to specify, somehow, which duplicates to keep. And so far, none of the standard programs for dealing with duplicates (duff, FSLint) seem to allow automation of that kind of behavior. I'd prefer not to roll my own, so that's why I'm asking this question.

I'd like to be able to write something like

killdupes -rdN --keep=filesin,somedirectories,separated,by,commas somedirectory/
share|improve this question

4 Answers 4

I didn't see this one anywhere else: Say what you want is this. You have /mnt/folder-tree-1 /mnt/folder-tree-2. You don't want to remove every dupe, but if a file exists in tree-2, and an identical file exists in tree-1 with the exact same path and name, remove it from tree-2.

Warning: this quite terse and if you try to copy-paste this with limited shell skills, be careful.

fdupes -rn /mnt/folder-tree-1/ /mnt/folder-tree-2/ > dupes-all.txt

fgrep /mnt/folder-tree-1/ dupes-all.txt | while read line
do
if grep -q "`echo $line | sed -e 's|^/mnt/folder-tree-1/|/mnt/folder-tree-2/|'`" dupes-all.txt
then
    echo rm \"$(echo $line | sed -e 's|^/mnt/folder-tree-1/|/mnt/folder-tree-2//|')\"
fi
done > rm-v2-dupes.sh

Or all on one line:

fdupes -rn /mnt/folder-tree-1/ /mnt/folder-tree-2/ > dupes-all.txt; fgrep /mnt/folder-tree-1/ dupes-all.txt | while read line; do if grep -q "`echo $line | sed -e 's|^/mnt/folder-tree-1/|/mnt/folder-tree-2/|'`" dupes-all.txt; then echo rm \"$(echo $line | sed -e 's|^/mnt/folder-tree-1/|/mnt/folder-tree-2/|')\"; fi; done > rm-v2-dupes.sh

Afterwards, inspect and execute rm-v2-dupes.sh

share|improve this answer

What about something like this?

#!/bin/bash

DUPE_SEARCH_DIR=somedir/
PREFERRED_DIRS=("somedir/subdir1" "somedir/subdir2")
DUPE_FILE=/tmp/`basename $0`_found-duplicates

delete_dupes() {
    while read line ; do
        if [ -n "$line" ] ; then
            matched=false
            for pdir in "${PREFERRED_DIRS[@]}" ; do
                if [[ $line == $pdir/* ]] ; then
                    matched=true
                    break
                fi
            done
            if ! $matched ; then
                rm -v "$line"
            fi
        fi
    done < "$DUPE_FILE"
}

cleanup() {
    rm -f $DUPE_FILE
}

trap cleanup EXIT

# get rid of normal dupes, preserve first & preserve preferred
fdupes -rf "$DUPE_SEARCH_DIR" > $DUPE_FILE
delete_dupes

# get rid of preserve dupes, preserve preferred
fdupes -r "$DUPE_SEARCH_DIR" > "$DUPE_FILE"
delete_dupes
share|improve this answer

Use sed to create a shell file that will contain commented out commands to delete each of your duplicate files:

fdupes -r -n -S /directory | sed -r "s/^/#rm \"/" | sed -r "s/$/\"/" >remove-duplicate-files.sh

The resulting remove-duplicate-files.sh file that we've just created will have each line commented out. Uncomment the files you want to delete. Then run sh remove-duplicate-files.sh. Voila!

UPDATE

Well, if you don't want to delete files only in certain directories, it's as simple as this:

fdupes -S /directory|sed '/^$/d' |sed -r "s/^[0-9]/#&/" > duple_list

python exclude_duplicates.py -f /path/to/dupe_list --delimiter='#' --keep=/full/path/to/protected/directory1,/full/path/to/protected/directory2\ with\ spaces\ in\ path >remove-duplicate-files-keep-protected.sh

Where exclude_duplicates.py is:

#/usr/bin/python
# -*- coding: utf-8 -*-
# exclude_duplicates.py
"""
THE SCRIPT DOESN'T DELETE ANYTHING, IT ONLY GENERATES TEXT OUTPUT.
Provided a list of duplicates, such as fdupes or fslint output,
generate a bash script that will have all duplicates in protected
directories commented out. If none of the protected duplicates are
found in a set of the same files, select a random unprotected
duplicate for preserving.
Each path to a file will be transformed to an `rm "path"` string which
will be printed to standard output.     
"""

from optparse import OptionParser
parser = OptionParser()
parser.add_option("-k", "--keep", dest="keep",
    help="""List of directories which you want to keep, separated by commas. \
        EXAMPLE: exclude_duplicates.py --keep /path/to/directory1,/path/to/directory\ with\ space\ in\ path2""",
    metavar="keep"
)
parser.add_option("-d", "--delimiter", dest="delimiter",
    help="Delimiter of duplicate file groups", metavar="delimiter"
)
parser.add_option("-f", "--file", dest="file",
    help="List of duplicate file groups, separated by delimiter, for example, fdupes or fslint output.", metavar="file"
)

(options, args) = parser.parse_args()
directories_to_keep = options.keep.split(',')
file = options.file
delimiter = options.delimiter

pretty_line = '\n#' + '-' * 35
print '#/bin/bash'
print '#I will protect files in these directories:\n'
for d in directories_to_keep:
    print '# ' + d
print pretty_line

protected_set = set()
group_set = set()

def clean_set(group_set, protected_set, delimiter_line):
    not_protected_set = group_set - protected_set
    while not_protected_set:
        if len(not_protected_set) == 1 and len(protected_set) == 0:
            print '#randomly selected duplicate to keep:\n#rm "%s"' % not_protected_set.pop().strip('\n')
        else:
            print 'rm "%s"' % not_protected_set.pop().strip('\n')
    for i in protected_set: print '#excluded file in protected directory:\n#rm "%s"' % i.strip('\n')
    print '\n#%s' % delimiter_line
file = open(file, 'r')
for line in file.readlines():
    if line.startswith(delimiter):
        clean_set(group_set, protected_set, line)
        group_set, protected_set = set(), set()
    else:
        group_set = group_set|{line}
        for d in directories_to_keep:
            if line.startswith(d): protected_set = protected_set|{line}
else:
    if line: clean_set(group_set, protected_set, line)

The resulting remove-duplicate-files-keep-protected.sh file that we've just created will have all files from protected directories commented out. Open this file in your favorite text editor, check that everything is OK. Then run it. Voila (sic)!

share|improve this answer
    
i thought of this, but it's not automated enough. stupidly, i caused data loss with this method when dealing with duplicates spaced across multiple filesystems... there's no way to assign a priority, given the output of fdupes. basically i would have had to trawl through 10000 files by hand in order to prevent that data loss... so, no thanks... in fact, that data loss is the very reason i asked this question. –  ixtmixilix Mar 21 '12 at 21:12
    
@ixtmixilix, well, manual method is dependent on user attentiveness, here's nothing new. If you want something more automated, checkout an updated answer above. –  Ivan Kharlamov Mar 22 '12 at 14:10

What about hardlinking the duplicate files together? That way the space is only used once, but they still exist in all the paths. The catch with this is that hardlinked files should be modified in place (they should only be modified deleting the file and recreating it with the new content). The other approach is to symlink the files together, although you have the same issue of deciding which the "primary" file is. This could be done with the following script (although note that this doesn't handle filenames containing spaces).

fdupes --quiet --recurse --sameline somedirectory/ | while read SOURCE DESTS; do
    for DEST in $DESTS; do
        ln -f $SOURCE $DEST
    done
done
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.