1

I am storing some multi-GB files on two hard drives. After several years in offline storage (unfortunately in far from ideal conditions), I often get some files with bit-rot (the two copies differ), and want to recover the file. The problem is, the files are so big, that within the same file, on some storage devices one bit gets rotten, whereas on another one a different bit gets bit-rotten, and so neither of the disks contains an uncorrupted file.

Therefore, instead of calculating the MD5 checksums of the entire files, I would like to calculate these checksums of each 1KB-chunk. With such a small chunk, there is a lot less chance that the same 1KB-chunk will get corrupted on both hard drives.

How can this be done? I am sure it shouldn't be hard, but I spent over an hour trying different ways, and keep failing.

3

I am not offering a complete solution here, but rather I'm hoping to be able to point you along the way to building your own solution. Personally I think there are better tools, such as rsync, but that doesn't seem to fit the criteria in your question.

I really wouldn't use split because that requires you to be able to store the split data as well as the original. Instead I'd go for extracting blocks with dd. Something like this approach may be helpful for you.

file=/path/to/file
blocksize=1024    # Bytes per block
numbytes=$(stat -c '%s' "$file")
numblocks=$((numbytes / blocksize))
[[ $((numblocks * blocksize)) -lt $numbytes ]] && : $((numblocks++))

blockno=0
while [[ $blockno -lt $numblocks ]]
do
    md5sum=$(dd bs=$blocksize count=1 skip=$blockno if="$file" 2>/dev/null | md5sum)
    # Do something with the $md5sum for block $blockno
    # Here we write to stdout
    echo "$blockno $md5sum"

    : $((blockno++))
done
  • Thank you so much. I heard there was a way to do it with dd, but this is my first time actually seeing how. – Alex Jun 30 '17 at 19:38
2

I have a similar problem with bitrot on optical media (currently BD-R, but I've used the same approach on CD-R and DVD-R).

There is a program called par2 which generates recovery data (using Reed-Solomon codes) such that a certain number of errors can be not only detected but corrected. You configure a block size, and a percent redundancy (which also is the amount of extra disk space required). E.g., if you use 1,000 blocks and 10% redundancy, you'll consume an extra 10% disk space for 100 blocks of redundancy, for 1100 total. But in exchange, you can fully recover the file as long as you have any 1000 uncorrupted blocks. So as long as 100 or fewer blocks contain bitrot, you can recover the file.

The downside of par2 is that computing that recovery data takes a while, and the more of it you generate, the longer it takes (generating 20% takes longer than 10%).

Another similar tool is zfec, though I haven't personally used it.

  • Nice. That is definitely easier than manually copying over several 1-KB chunks in a multi-GB file. – Alex Jul 3 '17 at 21:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.