6

I have a huge songs folder with a messy structure and files duplicated in multiple folders.

I need a recommendation for a tool or a script that can find and remove duplicates with simple two matches:

  1. Exact same file name
  2. Exact same file size

In this case, song.mp3 with 1234 bytes of file size is stored in /songs/album1 and /songs/albumz. The tool/script should keep only one of the copies.

I have tried czkawka on Fedora, but it can search by either filename or file size, but not both combined together.

3
  • Are you picky about which one of the duplicates you want to keep?
    – db-inf
    Nov 21 at 10:03
  • are you sure you just want to check filenames and sizes? this looks a lot like you'd actually want to use the file contents instead. Things like playlist files tend to have the same name and two randomly having the same size doesn't sound so unlikely. Nov 21 at 10:35
  • (also, if you're just after saving storage space, on XFS and BTRFS, you can use duperemove to save common extents in multiple files only once, with no other downside) Nov 21 at 12:56
6

rdfind may do what you want, but you'll need to depend on a cryptographic hash/checksum (md5, sha1 or sha256) in lieu of the filesize.

The hash is a stronger criteria than filesize, but this may or may not be what you want. For example, consider all of the metadata in a music file: If one file listed Schubert as the composer, and the other potential duplicate listed Bruckner as the composer and everything else in the file were exactly the same, the file size filter would classify it as a match, but the hash filter would not. The hash filter will use far more resources than file size filter, but that may not be a concern if you filter duplicates only occasionally.

Before you actually run rdfind, make sure to read man rdfind thoroughly and use the -dryrun option until you're confident the results are what you want.

FWIW, this tutorial lists rdfind, and 3 other utilities for finding duplicate files.

I don't know of a tool that filters on filesize, but if I were creating one from scratch, I think I'd cobble something together using find and awk. Let us know if that's what you want - I don't think it's terribly difficult, but there's not much point if rdfind meets your needs.

5
  • note that in MP3s, tags are written at the end of files, so files will grow with tags. Nov 21 at 10:36
  • I hadn't heard about rdfind before, but it sounds similar (at least for this ude case) to fdupes that I would use (there's also jdupes, that's a bit faster but have dropped '-1' that I find important). Nov 21 at 11:22
  • The hash is a stronger criteria than filesize True, but computing the hash requires reading the entire contents of all files. And that's unnecessary if two files being compared have different sizes in the first place - they can't have the same content if they have different sizes. Nov 21 at 13:10
  • @AndrewHenle: Right. And if you don't trust your hash, and do a bit-wise comparison if two files have the same size+hash, you shouldn't waste your time calculating hashes if there are only a few files of a given size, as comparing them might exit quite fast if they are not identical. (If there are just two you know it will be at least as fast to read both files to the first bit that is different than reading both fles all the way through and calculate their hashes. You should trust that the tool you're using has the optimisations that make sense. And having a tool is often better a fast tool. Nov 21 at 17:35
  • 1
    @AndrewHenle: Your point is well-taken. There is indeed a tradeoff: The convenience of an existing tool/utility vs the fact that rdfind will use CPU cycles calculating hashes that (at least in this use-case) may be superfluous. As it is with most choices.
    – Seamus
    Nov 21 at 21:29
2

You really do not need external utilities for this. You can go quite some distance by chaining a set of standard commands, that you can always remember by storing them in a shell script or function.

To detect duplicates, you might store all file paths in an associative array, keyed on size plus filename. But because you indicate you have a large file collection, I suggest to go for sorting instead. Then sort a second time with the unique option, so that a compare of both results gives you the duplicates.

  1. You want to use the output of the first sort twice. I can use a temporary file for that; named pipes and fifos and such do not require less work for this simple job. The command mktemp will print a safely unique filename in the /tmp/ directory for you. With the command substition construct $(...) you can assign that name to a variable named tmp or whatever. Echo it to know what you are doing.

  2. Now you want to use 2 key fields: size and filename, and you need the full path as well. It is a convenience that the slash / is not a valid filename character: it is reserved for separating directorynames and filenames. As a consequence, you can use the / as a separator between these 3 fields. The -printf action of the find command can give you that: find all of -type f(ile), and print their size %s, filename %f and full path %p, all separated by /. The full paths will contain more slashes, but we know that only the first 2 slash separators, the ones we specify explicitely in the -printf format, are ours.

  3. Pipe | the output of the find command, a list of sizes, filenames and full paths, to a sort command. Tell it that the fields are separated by our slashes: -t /, and that the key is in the first 2 fields: -k 1,2. You could store the sorted list immediately in your temporary file, but I chose to let the tee command do that, because that way you can keep one pipeline open, for better performance.

  4. Pipe | the sorted list to tee, which stores a copy of it to the given filename "$tmp", and sort of echoes the pipe through to the next pipe.

  5. Pipe | tee's echo of the sorted list to a second sort, on the same keys, but now make the output -u or --unique in the sort keys.

  6. Pipe | the list of unique size+filename entries to the diff command as it's first input - (stdin), and use the temporary file "$tmp" as it's second. Normally diff adds some markup to it's output, so that you can identify the nature of changes. But we do not need that, and it would clutter the output for our needs. We know that every entry of the uniqued list is also on the complete list. We just want the plain extra lines that are only in the complete list. That is what the options to the diff command ask for.

  7. The command up to here will output a hopefully short list of files that have a duplicate. One of each set of duplicates is omitted, because it still figured in the uniqued list. To handle each entry of this list, and act on it's last part, the full pathname, we pipe | it into a while read construct. That stores each line under the given parameter name dupl. With the bash parameter expansion syntax # we omit the prefix */*/, that is size plus filename with their slash / separators, and what remains is the full path to a file duplicate. I suggest you execute the complete command with an echo first, and replace that with the remove rm command after checking it thourougly, or better still, use one of the trashcan utilities to move the duplicates to yuor Trash, so that your directory is cleaned up, but nothing is lost.

    tmp=$(mktemp)
    echo temp file is "$tmp"
    find -type f -printf "%s/%f/%p\n" |
     sort -t / -k 1,2 |
     tee "$tmp" |
     sort -t / -k 1,2 -u |
     diff --new-line-format="%L" --unchanged-line-format="" - "$tmp" |
     while read dupl;do echo "${dupl#*/*/}"; done
    rm "$tmp"
    
  8. Tidy up behind you, and clean up the temporary file "$tmp".

1

Here's a quick way to do it without installing any additional packages:

#!/bin/bash
find /path/to/compare/against -type f -printf "%s %f\n" | \
while read SIZE FILE; do
    find /where/to/look/for/duplicates -iname "$FILE" -size "${SIZE}c" | \
    while read DUPLICATE; do
        # whatever you want to do with the duplicate file
    done
done
2
  • 1
    if the two paths overlap (and it seems from the question that both may be /songs) this will report every file as being identical to itself (it is, but that's probably irrelevant). Nov 21 at 8:38
  • If there is an overlap, you could pipe the second find through count -l to count the number of matches. It is very likely that you would want to then dump a list of these files for manual review.
    – EpicVoyage
    Nov 21 at 19:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.