md5 sum (fdupes) vs sha256sum - why different results - ubuntu server

Question

I have a directory structure that I know contains a LOT of duplicate files. (the dir tree has 67K files with around 8K duplicates). Naturally, the duplicates are all same size in bytes and the same content but filenames are all completely different. When using fdupes /dir/path -rm I get only 125 duplicates. However if I dump the SHA256 of the entire dir tree content into a text file:

    input_dir=$1
    IFS=$'\n'
    for i in $(find ${input_dir} -type f); do
       sha256sum "${i}" >> dupfilenames.txt
    done

and then grep for duplicate SHA256 signatures:

  cat dupfilenames.txt | cut -d " " -f1 | sort | uniq | while read sha; do
  count=`cat dupfilenames.txt | grep ${sha} | wc -l`
  if [ $count -gt 1 ]; then
      echo "${sha}:${count}"
      IFS=$'\n'
      files=( $(cat dupfilenames.txt | grep ${sha} | cut -d " " -f3) );
      orig_size=`stat -c%s "${files[0]}"`
      for i in "${files[@]:1}"; do
       if [ $orig_size -eq `stat -c%s "${i}"` ]; then
          echo "Origsize:${orig_size} vs. `stat -c%s "${i}"` '${i}'"
       else
          echo "SHA matches but filesize doesn't for '${i}'!!!"
       fi
      done
    fi
  done

After code execution I get around 8000 duplicate signatures.

Is this due to deficiency of MD5 signature which fdupes uses? (I know for a fact that I have much more than 125 files with exactly the same content, not to mention byte-for-byte filesize.

Answer 1

Ok, I figured it out - it turns out fdupes will disregard hardlinked files if no explicit switch ( -H ) is given. This is why the difference in number of duplicate files. When running with -H switch, the number of duplicates is the same as my code quoted above produces.