1

I have a folder with around 300 text files, is there any command that reads every every file indivdually and deletes the duplicate files? I mean the content and not the file names.

  • I would start with md5sum * and then count how many files have the same checksum. – glenn jackman Oct 30 '14 at 16:52
  • 1
    Needless to say: back up your folder before trying any of the solutions suggested here. – Joseph R. Oct 30 '14 at 17:19
  • Yeah, i always do that – DisplayName Oct 30 '14 at 17:29
1

If you have fdupes, it can list all the duplicated files in your folder.

You could refer to this online tutorial on how to use the fdupes command.

Testing

I created 3 files named file1, file2 and file3 with file1 and file2 having exactly similar contents.

Now, I executed my command as,

fdupes -rdN .

Where (quoting from the above refered link),

  1. The r option makes fdupes search for files recursively.
  2. The d option makes fdupes delete duplicates.
  3. The N option, when used together with d, preserve the first file in each set of duplicates and delete the others without prompting the user.

After executing the above command, I have file1 and file3 in my folder and file2 got deleted.

  • Hmm, i will try the command i got it via homebrew. – DisplayName Oct 30 '14 at 16:29
  • @DisplayName, do you want to keep the filenames? – Ramesh Oct 30 '14 at 16:30
  • What does the . do? – DisplayName Oct 30 '14 at 18:14
  • @DisplayName, . is used for the current working directory. – Ramesh Oct 30 '14 at 18:55
0

If fdupes isn't available, you could also use:

for first in *.txt
do
    for second in *.txt
    do
        if  diff $first $second >/dev/null 2>&1 && [ "$first" != "$second" ]
        then
            #echo $first and $second match. Deleting ${second}. # Optional, uncomment to use.
            rm $second
        fi
    done
done 

Note: This is very inefficient. It will execute diff 90,000 times for 300 files. It'll still be very quick if they're fairly small files, but could take a long time if they're big files.

0

As a secondary answer, when fdupes is not available, a more efficient way would use md5 to get the hash, and sort and uniq to find duplicates without a double shell loop

Something like : (put all on one line, without comments)

find . -type f -name '*.txt'   // get recursively all .txt files
| xargs md5sum         // compute the md5 sum
| awk '{print $2,$1}'  // reverse the md5sum output
| sort -k 2            // sorts on the md5 hash
| uniq --all-repeated=prepend -f 1     // get groups of duplicate files
| awk '/^$/ { I=1 }; /^./ { if (I==0) {print $1} I = 0; }'  // see below
| xargs rm             // delete

would delete every .txt file already encountered

(Rigorously, I'm neglecting the case of MD5 collisions, as they shouldn't occur in normal situations.)

Explaining the uniq and awk lines :

Let's assume :
file1:This is a 1st content
file2:This is a 1st content
file3:This is a 2nd content
file4:This is a 3rd content
file5:This is a 1st content
file6:This is a 3rd content

Result of sort is :

file4 801620325e6bc5efa4333a9413811e23
file6 801620325e6bc5efa4333a9413811e23
file3 8f9722a09b4c6f0ddf867e268193ea1b
file1 a066d80d23803dffa9fbc1cdcd95e163
file2 a066d80d23803dffa9fbc1cdcd95e163
file5 a066d80d23803dffa9fbc1cdcd95e163

uniq --all-repeated=prepend -f 1 keeps only duplicates, prepending each block with a blank line :

(blank line)
file4 801620325e6bc5efa4333a9413811e23
file6 801620325e6bc5efa4333a9413811e23
(blank line)
file1 a066d80d23803dffa9fbc1cdcd95e163
file2 a066d80d23803dffa9fbc1cdcd95e163
file5 a066d80d23803dffa9fbc1cdcd95e163

Then, a mini awk script ignores the blank lines, and print only the first field of lines not following a blank line

(--> not printed: blank line)
(--> not printed: file4)
file6
(--> not printed: blank line)
(--> not printed: file1)
file2
file5

Then, a xargs rm can rm the remaining (that is, duplicates files)

  • Adding -type f to the find command, to ignore symbolic links and directories. – Pierre-Olivier Vares Oct 30 '14 at 18:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.