4

I have realized that I need to protect all of my photographs against bit rot (file corruption occurring at random due to errors in hard drives or network transfer).

I recently discovered par2 which seems like a great program to create redundancy files and give the ability to detect and repair file corruptions.

I don't think journaling file systems are the right solution here, since I want the protection to follow along with the files into my backup and when migrating onto new laptops.

So, what I think I need is a script that can be run as a cronjob, maybe once an hour. It would look through all of the files that needs protection and update the redundancy files if files are added or changed (file has edit timestamp newer than redundancy arhive), and it would repair files if any file has been corrupted (file has changed but edit timestamp hasn't been updated).

Is there any script or program that would do this? Or are there programs that solve the problem in another way? Or should I just write such a script myself (a would prefer not to, I want something robust and tested by a lot of users)?

  • 2
    Are you sure that random file corruption (bitrot) is your issue? That should be a super rare case (I've yet to see it happen on any of my disks). Ironically it's usually the image viewers / library management softwares that alter your images, and that is where filesystem checksumming and other redundancy fails. Keeping your own checksum lists and multiple backups on read-only filesystems/media is not the worst choice when it comes to preserving digital data. – frostschutz Jun 13 '14 at 23:23
  • @frostschutz - I don't think its that uncommon. Depends on the quality of the spindles you are running and the I/O hardware and firmware. I recently hit a whole bunch of compressed VMs on a dmraid linux server that were fine a year ago and now will no longer uncompress. I tried different decompression utilities with the same result. IMO only explanation is bitrot. I don't have checksums to validate 100% unfortunately but its highly suspect. – Timothy C. Quinn Dec 20 '17 at 5:23
  • @TimothyC.Quinn HDDs checksum every sector so you see read errors before bad data. There are tons of ways to get bitrot caused by software. For example extended/logical dos partitions and then playing with parted - that already does it. Partition information is stored all over the place and fixing a broken partition table does not undo damage. With only very few bytes changed, it might be fixable. You certainly can fix JPEGs with a single flipped bit, as long as it's the original file, not converted nor truncated. – frostschutz Dec 20 '17 at 16:37
3

The canonical solution is to use a filesystem that supports checksumming and to do regular backups.

In addition to that you can also use a redundant storage scheme (RAID) - at the filesystem layer - to avoid time consuming restores from the backup, if possible.

Examples of such filesystems are ZFS or Btrfs.

The checksumming feature of such filesystems is implemented using an cryptographic hash function. Thus, bit-errors downstream the storage stack are detected with a very high probability - because of that their capability to detect corruptions is on the same level or higher as with tools like par2.

Those filesystems also include redundant storage features - similar to RAID levels. The difference to a classical layering RAID approach is that in the case of a detected corruption the filesystem is able to chose the 'correct' leg, i.e. the side of the RAID mirror that returns the block with the correct checksum.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.