This question already has an answer here:

I found this command used to find duplicated files but it was quite long and made me confused. For example, If i remove -printf "%s\n", nothing came out. Why was that? Besides, why they used xargs -I{} -n1? Is there any easier way to find duplicated files?

[4a-o07-d1:root/798]#find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate
0bee89b07a248e27c83fc3d5951213c1  ./test1.txt
0bee89b07a248e27c83fc3d5951213c1  ./test2.txt

marked as duplicate by Anthon, Raphael Ahrens, roaima, dr01, MelBurslan Apr 20 '16 at 12:08

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    By "quick", do you mean quickest to type, or quickest to finish? If you want the latter, it will pay to partition by file sizes prior to computing and partitioning by MD5 hashes. – reinierpost Apr 20 '16 at 10:22
  • Sorry i think i didn't make it clear. I want to use command line with least complicated to find duplicated files. – The One Apr 22 '16 at 1:07
up vote 20 down vote accepted

You can make it shorter:

find . ! -empty -type f -exec md5sum {} + | sort | uniq -w32 -dD

Do md5sum of found files on the -exec action of find and then sort and do uniq to get the files having same the md5sum separated by newline.

  • 1
    It worked! Thank you. – The One Apr 20 '16 at 5:33
  • 1
    Why -exec sh -c 'md5sum "$1"' _ {} \; instead of simply -exec md5sum {} +? – MvG Apr 20 '16 at 10:45
  • @MvG You are absolutely right..edited..while writing the answer i thought from my head that md5sum does not take multiple arguments but duh.... – heemayl Apr 20 '16 at 11:38
  • 2
    This is not the quickest. For several GB large files, there's no need to hash it whole. You can hash first N kB and then do a full one if same hash is found. – Ondra Žižka Jun 2 '17 at 0:20
  • What does -dD mean to uniq? – Tim Jul 23 at 2:52

You can use fdupes. From man fdupes:

Searches the given path for duplicate files. Such files are found by comparing file sizes and MD5 signatures, followed by a byte-by-byte comparison.

You can call it like fdupes -r /path/to/dup/directory and it will print out a list of dupes.

Update

You can give it try to fslint also. After setting up fslint, go to cd /usr/share/fslint/fslint && ./fslint /path/to/directory

In case you want to understand the original command, let's go though that step by step.

find -not -empty -type f

Find all non-empty files in the current directory or any of its subdirectories.

   -printf "%s\n"

Print its size. If you drop these arguments, it will print paths instead, breaking subsequent steps.

 | sort -rn

Sort numerically (-n), in reverse order (-r). Sorting in ascending order and comparing as strings not numbers should work just as well, though, so you may drop the -rn flags.

 | uniq -d

Look for duplicate consecutive rows and keep only those.

 | xargs -I{} -n1

For each line of input (i.e. each size that occurs more than once), execute the following command, but replace {} by the size. Execute the command once for each line of input, as opposed to passing multiple inputs to a single invocation.

   find -type f -size {}c -print0

This is the command to run for each size: Find files in the current directory which match that size, given in characters (c) or more precisely bytes. Print all the matching file names, separated by null bytes instead of newlines so filenames which contain newlines are treated correctly.

 | xargs -0 md5sum

For each of these null-separated names, compute the MD5 checksum of said file. This time we allow passing multiple files to a single invocation of md5sum.

 | sort

Sort by checksums, since uniq only considers consecutive lines.

 | uniq -w32 --all-repeated=separate

Find lines which agree in their first 32 bytes (the checksum; after that comes the file name). Print all members of such runs of duplicates, with distinct runs separated by newlines.

Compared to the simpler command suggested by heemayl, this has the benefit that it will only checksum files which have another file of the same size. It pays for that with repeated find invocations, thus traversing the directory tree multiple times. For those reasons, this command is particularly well-suited for directories with few but big files, since in those cases avoiding a checksum call may be more important than avoiding repeated tree traversal.

  • Very detailed!!! Thanks so much. – The One Apr 21 '16 at 2:25

Not the answer you're looking for? Browse other questions tagged or ask your own question.