I have realized that I need to protect all of my photographs against bit rot (file corruption occurring at random due to errors in hard drives or network transfer).

I recently discovered par2 which seems like a great program to create redundancy files and give the ability to detect and repair file corruptions.

I don't think journaling file systems are the right solution here, since I want the protection to follow along with the files into my backup and when migrating onto new laptops.

So, what I think I need is a script that can be run as a cronjob, maybe once an hour. It would look through all of the files that needs protection and update the redundancy files if files are added or changed (file has edit timestamp newer than redundancy arhive), and it would repair files if any file has been corrupted (file has changed but edit timestamp hasn't been updated).

Is there any script or program that would do this? Or are there programs that solve the problem in another way? Or should I just write such a script myself (a would prefer not to, I want something robust and tested by a lot of users)?

  • 3
    Are you sure that random file corruption (bitrot) is your issue? That should be a super rare case (I've yet to see it happen on any of my disks). Ironically it's usually the image viewers / library management softwares that alter your images, and that is where filesystem checksumming and other redundancy fails. Keeping your own checksum lists and multiple backups on read-only filesystems/media is not the worst choice when it comes to preserving digital data. Jun 13 '14 at 23:23
  • 1
    @frostschutz - I don't think its that uncommon. Depends on the quality of the spindles you are running and the I/O hardware and firmware. I recently hit a whole bunch of compressed VMs on a dmraid linux server that were fine a year ago and now will no longer uncompress. I tried different decompression utilities with the same result. IMO only explanation is bitrot. I don't have checksums to validate 100% unfortunately but its highly suspect. Dec 20 '17 at 5:23
  • @TimothyC.Quinn HDDs checksum every sector so you see read errors before bad data. There are tons of ways to get bitrot caused by software. For example extended/logical dos partitions and then playing with parted - that already does it. Partition information is stored all over the place and fixing a broken partition table does not undo damage. With only very few bytes changed, it might be fixable. You certainly can fix JPEGs with a single flipped bit, as long as it's the original file, not converted nor truncated. Dec 20 '17 at 16:37

The canonical solution is to use a filesystem that supports checksumming and to do regular backups.

In addition to that you can also use a redundant storage scheme (RAID) - at the filesystem layer - to avoid time consuming restores from the backup, if possible.

Examples of such filesystems are ZFS or Btrfs.

The checksumming feature of such filesystems is implemented using an cryptographic hash function. Thus, bit-errors downstream the storage stack are detected with a very high probability - because of that their capability to detect corruptions is on the same level or higher as with tools like par2.

Those filesystems also include redundant storage features - similar to RAID levels. The difference to a classical layering RAID approach is that in the case of a detected corruption the filesystem is able to chose the 'correct' leg, i.e. the side of the RAID mirror that returns the block with the correct checksum.

  • Yes ZFS, Btrfs, integritysetup, or a text file listing md5sums. Jul 18 '20 at 22:48

This is an older question, but still relevant in 2019.

Yes, parity files are a viable solution to bit-rot

While there are detriments to non-filesystem-level parity checking, there is also one huge benefit:


The fanciest filesystem-based error-checking in the world could theoretically keep your data bitrot-free forever, but protection is locked to that filesystem. As soon as files leave they are unprotected.

"too heavy"by brapps is licensed under CC BY-NC 2.0 (He should really switch to ZFS and a roof before putting the data in)

Images with parity files next to them can be moved to USB drives, transferred over packlossy connections, compressed, backed up individually to other media, and a hundred other things that I can think up off the top of my head.

They can also prevent changes based on bad programs (metadata analyzers that accidentally write incorrect metadata, for example)

PAR2 is a viable choice of parity file

PAR2 files particularly can be used cross-platform, and their biggest detriments are not actually a problem in the use-case of protecting large photos (those being that PAR2 doesn't support subfolders, and doesn't handle <1KB files efficiently).

PAR files are not a complete backup solution on thier own, however

They still need to be part of a 3-copy backup strategy to be "bulletproof", especially as with PAR files there's no change history: If a change is detected as valid, and the parity files recreated, there's no going back without referencing a separate backup (an off-site backup solution with built-in change history would be the winner here).

Next steps

The answer to the original question of “Is there any script or program that would [automatically create and maintain parity files]?” is still No in 2019, but making a script yourself or having one made is fairly trivial. If you go this route the best advice I can give is: create a test environment with every edge case you can think of, and run any script through all tests before trusting it.

And, if anyone reading this does go this route, consider open-sourcing it for the benefit of everyone.

Edited addendum: The technology used for PAR files is https://en.wikipedia.org/wiki/Reed–Solomon_error_correction , and while PAR development has stalled, there are dozens if not hundreds of projects using Reed-Solomon for file-level data integrity/repair.

  • 1
    I am not sure what to think about your answer: you make some not too bad points (but in a weird way somehow), told about the subject at some point (filesystem, PAR2 files) and ended with "the answer is no", because you rejected the check-summing filesystems? Not to mention the transitions between the different parts are weirdly made... Not sure where you are going to with this.
    – Paradox
    Aug 3 '19 at 21:23
  • @Paradox Valid points. After rereading the day after I’ve reworked the wording to clarify my points and conclusions. Appreciate the input.
    – joshfindit
    Aug 4 '19 at 14:28
  • 1
    Glad I could help. Indeed, it is much more readable and clearer now and looks more directed toward answering the question. If I could upvote an edit, I definitely would!
    – Paradox
    Aug 4 '19 at 16:13

I wrote a lightweight tool to detect bitrot called chkbit.

It creates a hash that follows your data independent of the filesystem. So chkbit allows you to verify the integrity of your data on your main system, on the backup media and after a restore.

chkbit does not allow you to repair the data, you will still need one or more backups.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.