Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Recently I have the need to delete a lot of duplicates. I am merging three or four filesystems, and I want the space to be used economically. At first, fdupes seemed like it was the best tool for the job, but I am increasingly running into limitations.

Consider the command fdupes -rdN somedirectory/. This makes a hash of all of the files in the subdirectories of somedirectory.

And when it encounters duplicates, it deletes them, so that there is only one copy of everything.

But what if I want to keep somedirectory/subdirectory1/somefile and there are, in fact, four duplicates, and the program encounters one of the duplicates first? Then it deletes somedirectory/subdirectory1/somefile, which I don't want.

I want to be able to specify, somehow, which duplicates to keep. And so far, none of the standard programs for dealing with duplicates (duff, FSLint) seem to allow automation of that kind of behavior. I'd prefer not to roll my own, so that's why I'm asking this question.

I'd like to be able to write something like

killdupes -rdN --keep=filesin,somedirectories,separated,by,commas somedirectory/
share|improve this question
    
I was looking for the same thing and I found this superuser.com/a/561207/218922 – alexis Sep 1 '15 at 23:30

What about hardlinking the duplicate files together? That way the space is only used once, but they still exist in all the paths. The catch with this is that hardlinked files should be modified in place (they should only be modified deleting the file and recreating it with the new content). The other approach is to symlink the files together, although you have the same issue of deciding which the "primary" file is. This could be done with the following script (although note that this doesn't handle filenames containing spaces).

fdupes --quiet --recurse --sameline somedirectory/ | while read SOURCE DESTS; do
    for DEST in $DESTS; do
        ln -f $SOURCE $DEST
    done
done
share|improve this answer

I didn't see this one anywhere else: Say what you want is this. You have /mnt/folder-tree-1 /mnt/folder-tree-2. You don't want to remove every dupe, but if a file exists in tree-2, and an identical file exists in tree-1 with the exact same path and name, remove it from tree-2.

Warning: this quite terse and if you try to copy-paste this with limited shell skills, be careful.

fdupes -rn /mnt/folder-tree-1/ /mnt/folder-tree-2/ > dupes-all.txt

fgrep /mnt/folder-tree-1/ dupes-all.txt | while read line
do
if grep -q "`echo $line | sed -e 's|^/mnt/folder-tree-1/|/mnt/folder-tree-2/|'`" dupes-all.txt
then
    echo rm \"$(echo $line | sed -e 's|^/mnt/folder-tree-1/|/mnt/folder-tree-2//|')\"
fi
done > rm-v2-dupes.sh

Or all on one line:

fdupes -rn /mnt/folder-tree-1/ /mnt/folder-tree-2/ > dupes-all.txt; fgrep /mnt/folder-tree-1/ dupes-all.txt | while read line; do if grep -q "`echo $line | sed -e 's|^/mnt/folder-tree-1/|/mnt/folder-tree-2/|'`" dupes-all.txt; then echo rm \"$(echo $line | sed -e 's|^/mnt/folder-tree-1/|/mnt/folder-tree-2/|')\"; fi; done > rm-v2-dupes.sh

Afterwards, inspect and execute rm-v2-dupes.sh

share|improve this answer

Use sed to create a shell file that will contain commented out commands to delete each of your duplicate files:

fdupes -r -n -S /directory | sed -r "s/^/#rm \"/" | sed -r "s/$/\"/" >remove-duplicate-files.sh

The resulting remove-duplicate-files.sh file that we've just created will have each line commented out. Uncomment the files you want to delete. Then run sh remove-duplicate-files.sh. Voila!

UPDATE

Well, if you don't want to delete files only in certain directories, it's as simple as this:

fdupes -S /directory|sed '/^$/d' |sed -r "s/^[0-9]/#&/" > duple_list

python exclude_duplicates.py -f /path/to/dupe_list --delimiter='#' --keep=/full/path/to/protected/directory1,/full/path/to/protected/directory2\ with\ spaces\ in\ path >remove-duplicate-files-keep-protected.sh

Where exclude_duplicates.py is:

#/usr/bin/python
# -*- coding: utf-8 -*-
# exclude_duplicates.py
"""
THE SCRIPT DOESN'T DELETE ANYTHING, IT ONLY GENERATES TEXT OUTPUT.
Provided a list of duplicates, such as fdupes or fslint output,
generate a bash script that will have all duplicates in protected
directories commented out. If none of the protected duplicates are
found in a set of the same files, select a random unprotected
duplicate for preserving.
Each path to a file will be transformed to an `rm "path"` string which
will be printed to standard output.     
"""

from optparse import OptionParser
parser = OptionParser()
parser.add_option("-k", "--keep", dest="keep",
    help="""List of directories which you want to keep, separated by commas. \
        EXAMPLE: exclude_duplicates.py --keep /path/to/directory1,/path/to/directory\ with\ space\ in\ path2""",
    metavar="keep"
)
parser.add_option("-d", "--delimiter", dest="delimiter",
    help="Delimiter of duplicate file groups", metavar="delimiter"
)
parser.add_option("-f", "--file", dest="file",
    help="List of duplicate file groups, separated by delimiter, for example, fdupes or fslint output.", metavar="file"
)

(options, args) = parser.parse_args()
directories_to_keep = options.keep.split(',')
file = options.file
delimiter = options.delimiter

pretty_line = '\n#' + '-' * 35
print '#/bin/bash'
print '#I will protect files in these directories:\n'
for d in directories_to_keep:
    print '# ' + d
print pretty_line

protected_set = set()
group_set = set()

def clean_set(group_set, protected_set, delimiter_line):
    not_protected_set = group_set - protected_set
    while not_protected_set:
        if len(not_protected_set) == 1 and len(protected_set) == 0:
            print '#randomly selected duplicate to keep:\n#rm "%s"' % not_protected_set.pop().strip('\n')
        else:
            print 'rm "%s"' % not_protected_set.pop().strip('\n')
    for i in protected_set: print '#excluded file in protected directory:\n#rm "%s"' % i.strip('\n')
    print '\n#%s' % delimiter_line
file = open(file, 'r')
for line in file.readlines():
    if line.startswith(delimiter):
        clean_set(group_set, protected_set, line)
        group_set, protected_set = set(), set()
    else:
        group_set = group_set|{line}
        for d in directories_to_keep:
            if line.startswith(d): protected_set = protected_set|{line}
else:
    if line: clean_set(group_set, protected_set, line)

The resulting remove-duplicate-files-keep-protected.sh file that we've just created will have all files from protected directories commented out. Open this file in your favorite text editor, check that everything is OK. Then run it. Voila (sic)!

share|improve this answer
    
i thought of this, but it's not automated enough. stupidly, i caused data loss with this method when dealing with duplicates spaced across multiple filesystems... there's no way to assign a priority, given the output of fdupes. basically i would have had to trawl through 10000 files by hand in order to prevent that data loss... so, no thanks... in fact, that data loss is the very reason i asked this question. – ixtmixilix Mar 21 '12 at 21:12
    
@ixtmixilix, well, manual method is dependent on user attentiveness, here's nothing new. If you want something more automated, checkout an updated answer above. – Ivan Kharlamov Mar 22 '12 at 14:10

What about something like this?

#!/bin/bash

DUPE_SEARCH_DIR=somedir/
PREFERRED_DIRS=("somedir/subdir1" "somedir/subdir2")
DUPE_FILE=/tmp/`basename $0`_found-duplicates

delete_dupes() {
    while read line ; do
        if [ -n "$line" ] ; then
            matched=false
            for pdir in "${PREFERRED_DIRS[@]}" ; do
                if [[ $line == $pdir/* ]] ; then
                    matched=true
                    break
                fi
            done
            if ! $matched ; then
                rm -v "$line"
            fi
        fi
    done < "$DUPE_FILE"
}

cleanup() {
    rm -f $DUPE_FILE
}

trap cleanup EXIT

# get rid of normal dupes, preserve first & preserve preferred
fdupes -rf "$DUPE_SEARCH_DIR" > $DUPE_FILE
delete_dupes

# get rid of preserve dupes, preserve preferred
fdupes -r "$DUPE_SEARCH_DIR" > "$DUPE_FILE"
delete_dupes
share|improve this answer

While the functionality you seek is not available in stock fdupes, I forked fdupes (my fork is called jdupes) and added some features that can solve this problem under certain circumstances. For example, in the stated case where you wish to keep somedirectory/subdirectory1/somefile when auto-deleting duplicates (the d and N switches together) and there are no separate files immediately beneath somedirectory, jdupes can be fed each immediate subdirectory path with subdirectory1 first and the -O switch (which sorts files by command-line parameter order first):

jdupes -nrdNO somedirectory/subdirectory1 somedirectory/subdirectory2 somedirectory/subdirectory3

This will auto-delete all but one file in a duplicate set and will guarantee that if the set contains a file in somedirectory/subdirectory1 it will be the first, thereby automatically becoming the preserved file in the set. There are still glaring limits to this approach such as the fact that another duplicate in somedirectory/subdirectory1 might be preserved instead of the one you wanted to keep, but in a good number of cases like yours, the jdupes parameter order option as a workaround is good enough.

In the near future, I plan to add a filtering system to jdupes that will enable a huge amount of control over inclusion/exclusion of files, preservation for -N actions, and application of such "filter stacks" on either a global or per-parameter basis. This feature is sorely needed; I envision something like this to "auto-delete non-zero duplicates recursively BUT always preserve somedirectory/subdirectory1/somefile as-is":

jdupes -nrdN --filter=preserve:somedirectory/subdirectory1/somefile somedirectory/

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.