Finding duplicate files with same filename AND exact same size

Question

I have a huge songs folder with a messy structure and files duplicated in multiple folders.

I need a recommendation for a tool or a script that can find and remove duplicates with simple two matches:

1. Exact same file name
2. Exact same file size

In this case, song.mp3 with 1234 bytes of file size is stored in /songs/album1 and /songs/albumz. The tool/script should keep only one of the copies.

I have tried czkawka on Fedora, but it can search by either filename or file size, but not both combined together.

Answer 1

rdfind may do what you want, but you'll need to depend on a cryptographic hash/checksum (md5, sha1 or sha256) in lieu of the filesize.

The hash is a stronger criteria than filesize, but this may or may not be what you want. For example, consider all of the metadata in a music file: If one file listed Schubert as the composer, and the other potential duplicate listed Bruckner as the composer and everything else in the file were exactly the same, the file size filter would classify it as a match, but the hash filter would not. The hash filter will use far more resources than file size filter, but that may not be a concern if you filter duplicates only occasionally.

Before you actually run rdfind, make sure to read man rdfind thoroughly and use the -dryrun option until you're confident the results are what you want.

FWIW, this tutorial lists rdfind, and 3 other utilities for finding duplicate files.

I don't know of a tool that filters on filesize, but if I were creating one from scratch, I think I'd cobble something together using find and awk. Let us know if that's what you want - I don't think it's terribly difficult, but there's not much point if rdfind meets your needs.

Answer 2

You really do not need external utilities for this. You can go quite some distance by chaining a set of standard commands, that you can always remember by storing them in a shell script or function.

To detect duplicates, you might store all file paths in an associative array, keyed on size plus filename. But because you indicate you have a large file collection, I suggest to go for sorting instead. Then sort a second time with the unique option, so that a compare of both results gives you the duplicates.

1. You want to use the output of the first sort twice. I can use a temporary file for that; named pipes and fifos and such do not require less work for this simple job. The command mktemp will print a safely unique filename in the /tmp/ directory for you. With the command substition construct $(...) you can assign that name to a variable named tmp or whatever. Echo it to know what you are doing.

2. Now you want to use 2 key fields: size and filename, and you need the full path as well. It is a convenience that the slash / is not a valid filename character: it is reserved for separating directorynames and filenames. As a consequence, you can use the / as a separator between these 3 fields. The -printf action of the find command can give you that: find all of -type f(ile), and print their size %s, filename %f and full path %p, all separated by /. The full paths will contain more slashes, but we know that only the first 2 slash separators, the ones we specify explicitely in the -printf format, are ours.

3. Pipe | the output of the find command, a list of sizes, filenames and full paths, to a sort command. Tell it that the fields are separated by our slashes: -t /, and that the key is in the first 2 fields: -k 1,2. You could store the sorted list immediately in your temporary file, but I chose to let the tee command do that, because that way you can keep one pipeline open, for better performance.

4. Pipe | the sorted list to tee, which stores a copy of it to the given filename "$tmp", and sort of echoes the pipe through to the next pipe.

5. Pipe | tee's echo of the sorted list to a second sort, on the same keys, but now make the output -u or --unique in the sort keys.

6. Pipe | the list of unique size+filename entries to the diff command as it's first input - (stdin), and use the temporary file "$tmp" as it's second. Normally diff adds some markup to it's output, so that you can identify the nature of changes. But we do not need that, and it would clutter the output for our needs. We know that every entry of the uniqued list is also on the complete list. We just want the plain extra lines that are only in the complete list. That is what the options to the diff command ask for.

7. The command up to here will output a hopefully short list of files that have a duplicate. One of each set of duplicates is omitted, because it still figured in the uniqued list. To handle each entry of this list, and act on it's last part, the full pathname, we pipe | it into a while read construct. That stores each line under the given parameter name dupl. With the bash parameter expansion syntax # we omit the prefix */*/, that is size plus filename with their slash / separators, and what remains is the full path to a file duplicate. I suggest you execute the complete command with an echo first, and replace that with the remove rm command after checking it thourougly, or better still, use one of the trashcan utilities to move the duplicates to your Trash, so that your directory is cleaned up, but nothing is lost.

   tmp=$(mktemp)
   echo temp file is "$tmp"
   find -type f -printf "%s/%f/%p\n" |
    sort -t / -k 1,2 |
    tee "$tmp" |
    sort -t / -k 1,2 -u |
    diff --new-line-format="%L" --unchanged-line-format="" - "$tmp" |
    while read dupl;do echo "${dupl#*/*/}"; done
   rm "$tmp"
   

8. Tidy up behind you, and clean up the temporary file "$tmp".

Answer 3

Here's a quick way to do it without installing any additional packages:

#!/bin/bash
find /path/to/compare/against -type f -printf "%s %f\n" | \
while read SIZE FILE; do
    find /where/to/look/for/duplicates -iname "$FILE" -size "${SIZE}c" | \
    while read DUPLICATE; do
        # whatever you want to do with the duplicate file
    done
done