0

I have a directory structure that I know contains a LOT of duplicate files. (the dir tree has 67K files with around 8K duplicates). Naturally, the duplicates are all same size in bytes and the same content but filenames are all completely different. When using fdupes /dir/path -rm I get only 125 duplicates. However if I dump the SHA256 of the entire dir tree content into a text file:

    input_dir=$1
    IFS=$'\n'
    for i in $(find ${input_dir} -type f); do
       sha256sum "${i}" >> dupfilenames.txt
    done

and then grep for duplicate SHA256 signatures:

  cat dupfilenames.txt | cut -d " " -f1 | sort | uniq | while read sha; do
  count=`cat dupfilenames.txt | grep ${sha} | wc -l`
  if [ $count -gt 1 ]; then
      echo "${sha}:${count}"
      IFS=$'\n'
      files=( $(cat dupfilenames.txt | grep ${sha} | cut -d " " -f3) );
      orig_size=`stat -c%s "${files[0]}"`
      for i in "${files[@]:1}"; do
       if [ $orig_size -eq `stat -c%s "${i}"` ]; then
          echo "Origsize:${orig_size} vs. `stat -c%s "${i}"` '${i}'"
       else
          echo "SHA matches but filesize doesn't for '${i}'!!!"
       fi
      done
    fi
  done

After code execution I get around 8000 duplicate signatures.

Is this due to deficiency of MD5 signature which fdupes uses? (I know for a fact that I have much more than 125 files with exactly the same content, not to mention byte-for-byte filesize.

  • thanks for replying. no, it finds a, b and c with the same SHA256 signature and prints out only b and c. that part works fine though :) the problem is that I do not understand why would fdupes which is designed to weed out duplicates be insufficient for the task in my case. The files I am looking at by the way are PDF and JPEG images, so it's quite easy to establish that the content is duplicate. – sheikhness May 22 '18 at 10:49
  • yes, of course, it loops over every hash, not every file – ilkkachu May 22 '18 at 10:52
  • Ok, I figured it out - it turns out fdupes will disregard hardlinked files if no explicit switch ( -H ) is given. This is why the difference in number of duplicate files. When running with -H switch, the number of duplicates is the same as my code quoted above produces. – sheikhness May 22 '18 at 12:53
  • 1
    Well, that sounds like an answer, post it as one! – ilkkachu May 22 '18 at 13:07
1

Ok, I figured it out - it turns out fdupes will disregard hardlinked files if no explicit switch ( -H ) is given. This is why the difference in number of duplicate files. When running with -H switch, the number of duplicates is the same as my code quoted above produces.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.