1

I'm currently running rdfind on a directory containing more than 4TB of files. Since the checksum part takes an inordinate amount of time I'm looking for alternatives. I know fairly certain that there aren't any size collisions based on find /mnt/local/ -type f -printf "%s\t%p\n" | sort --numeric. Is there a tool or script that can hard link any duplicate files based on just size or maybe the hash of the first couple thousand of bytes?

1
  • Just checked and fdupes can't disable the checksum check either (and you are correct, checksumming could be done in blocks of whatever to enable fail-fast, that would actually be a good feature request). Commented Sep 9, 2020 at 15:21

1 Answer 1

2

jdupes can be configured to only checksum the first block of data in each file, using the -T option; thus in your case,

jdupes -T -T -r -L /mnt/local

would hard-link all duplicates found under /mnt/local, based only on their size and the hashes of the first block.

1
  • Works like a charm, thanks!
    – Martijn
    Commented Sep 9, 2020 at 15:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .