Conditional deleting of duplicate files

Question

I want to delete duplicates only if files share the same name.
Something like: rdfind -deleteduplicates true ~/folder will not accomplish my objective.

One idea could be to get the output (results.txt) of rdfind of a dry-run,

rdfind -n true "$PWD"

and determine which files have the same name of the 1st one of each group, identified by beginning with: DUPTYPE_FIRST_OCCURRENCE XXXX

DUPTYPE_FIRST_OCCURRENCE 5148 2 37934240 2054 16916792 1 /home/fig3.tif
DUPTYPE_WITHIN_SAME_TREE -5148 3 37934240 2054 17044654 1 /home/other/fig3.tif
DUPTYPE_WITHIN_SAME_TREE -5148 3 35435435 2054 16546546 1 /home/other2/fig3.tif

DUPTYPE_FIRST_OCCURRENCE 5160 2 116397930 2054 16916804 1 /home/file.psd
DUPTYPE_WITHIN_SAME_TREE -5160 2 116397930 2054 16916870 1 /home/folder/file.psd
DUPTYPE_WITHIN_SAME_TREE -5160 2 116397930 2054 17654654 1 /home/folder/file2.psd

In the example above, for the 1st group 2 duplicates should be removed, while in the 2nd group only one should be removed, which has the same name as the First occurrence (file.psd).

EDIT: An answer for rmlint would be desirable. Establishing the "original" file customarily is possible with rmlint

Answer 1

This script will post-process the results.txt file generated from rdfind as shown in your question:

#!/bin/bash
#
while read -r type x x x x x x path
do
    case "$type" in

        DUPTYPE_FIRST_OCCURRENCE)
            name="${path##*/}"
            echo "New path for $name" >&2
            ;;

        DUPTYPE_WITHIN_SAME_TREE|DUPTYPE_OUTSIDE_TREE)
            if [[ -n "$name" ]] && [[ ${path##*/} == $name ]]
            then
                echo "Remove duplicate $path" >&2
                # rm -f "$path"
            else
                echo "Skipping differently named $path" >&2
            fi
            ;;
    esac
done

Uncomment the rm when you are ready for it to do its work. Comment out the echo statements (or redirect stderr) if you don't want the verbosity.

The read -r type x x x x x x path statement at the top of the loop reads eight space-separated fields. The middle six are not relevant and so I read them into a junk variable that I then discard; only $type and $path are used.

Generate results.txt

rdfind -dryrun true -makeresultsfile true {directory...}

Post-process the generated file (rddedup is my script, shown above)

rddedup < results.txt

Answer 2

How to skin this cat another way? (OP edit requesting rmlint was made just as I was about to post this but never mind, may as well post this.... see edit at the bottom re rmlint)

Since the structure is known, the first occurence of a filename is the target and subsequent duplicate filenames before a blank line are to be deleted then we could also have fun with awk though likely somewhat slower than @roaima

awk 'NF==0{getline; target=""}
     target==""{l=split($8,t,"/"); target=t[l];getline}
     {l=split($8,q,"/");if (q[l]==target)print $8}' file1 | while read f; do echo rm $f ; done

Walkthrough

If we have a blank line skip it and reset our target filename to ""

awk 'NF==0{getline; target=""}

If the target is unset (i.e. after every blank line and at the start) then we grab the target filename from this line and skip to the next

     target==""{l=split($8,t,"/"); target=t[l];getline}

Now check the filename in the current line against the target and output the full path if there is a match

     {l=split($8,q,"/");if (q[l]==target)print $8}' file1

Then pipe awk through the loop to do the actual removal (just remove the echo) .

    | while read f; do echo rm $f ; done

Though you could also do this instead (again just remove the echo)

     {l=split($8,q,"/");if (q[l]==target)print $8}' file1 | xargs echo rm

Edit On the rmlint front it seems you cannot easily control the precedence of originality

By default, if you specify multiple paths in the rmlint command, the files in the first-named paths are treated as more “original” than the later named paths. If there are two files in the same path, then the older one will be treated as the original. If they have the same modification time then it’s just a matter of chance which one is selected as the original.

But for your explicit example, the -d option (original being the lowest in the file hierarchy) would work in combination with -b to match by file name and not content.

rmlint -d -b /home/

Admittedly this is a lot shorter than my script, but reading the rmlintpage took longer than writing the script. Take your pick.

Untested, by the way....

Answer 3

Here's roaima's script corrected for cases when the duplicates correspond to one another while they're not the first in the list for a given file. Eg. the same file has three names: a/x, b/y and c/y. Then when a/x is the base file (DUPTYPE_FIRST_OCCURRENCE), one of the others will be deleted.

#!/bin/bash

while read -r type x x x x x x path
do
    name="${path##*/}"
    [[ "$type" == '#' ]] || [[ -z "$name" ]] && 
        printf "%s\n" "skipping a non-entry looking line">&2
    case "$type" in
        DUPTYPE_FIRST_OCCURRENCE)
            printf "New path for %s: %s\n" "$name" "$path">&2
            unset ass
            declare -A ass
            ass["$name"]=1
            ;;

        DUPTYPE_WITHIN_SAME_TREE|DUPTYPE_OUTSIDE_TREE)
            if [ "${ass["$name"]}"x != "x" ]
            then
                printf "Remove duplicate: %s\n" "$path" >&2
                # rm -f "$path"
            else
                ass["$name"]=1
                printf "Skipping differently named file: %s\n" "$path" >&2
            fi
            ;;
    esac 
done < results.txt

A few other changes are included, which should be readable without comments. ass is an associative array.

I must admit this script has not been well tested, so there may still be some corner cases open.

Example usage. rdfind -n true . and then we have this:

$ ls *
results.txt  script.bash

a:
filea0  filea1  fileb0

b:
filea1  fileb0

c:
fileb0

results.txt:

$ cat results.txt 
# Automatically generated
# duptype id depth size device inode priority name
DUPTYPE_FIRST_OCCURRENCE 4 1 6 64773 4849814 3 ./a/filea0
DUPTYPE_WITHIN_SAME_TREE -4 1 6 64773 4849816 3 ./b/filea1
DUPTYPE_WITHIN_SAME_TREE -4 1 6 64773 4849817 3 ./a/filea1
DUPTYPE_FIRST_OCCURRENCE 6 1 6 64773 4849677 3 ./a/fileb0
DUPTYPE_WITHIN_SAME_TREE -6 1 6 64773 4849819 3 ./b/fileb0
DUPTYPE_WITHIN_SAME_TREE -6 1 6 64773 4986586 3 ./c/fileb0
# end of file

The script in action:

./script.bash
skipping a non-entry looking line
skipping a non-entry looking line
New path for filea0: ./a/filea0
Skipping differently named file: %s\n ./b/filea1
Remove duplicate: ./a/filea1
New path for fileb0: ./a/fileb0
Remove duplicate: ./b/fileb0
Remove duplicate: ./c/fileb0
skipping a non-entry looking line