6

I'm looking for an easy solution to protect against random bit flips (so-called bit rot) of data stored on various drives. They are not disk arrays, just single disks, that I backup once a week. So I'm not looking for redundancy, but for file integrity -- i.e. I want to know if my files that I haven't accessed in a long time got randomly damaged or not, and hopefully repair them if possible.

Please note that I want a generic solution, I'm specifically not looking for a filesystem like ZFS or btrfs (which I'm already aware of), partly because they have way too much overhead just for checksumming, and they're too complex / unstable (btrfs case).

It doesn't have to be an automatic thing. That is, if I have to run a command to generate checksums (and maybe recovery) for new written files, that's fine, but it should be simple to use, not something like manually storing checksums and verifying and then copying the bad files back etc (which I'm doing already, that's why I ask for something simpler, less manual).

At a glance, SnapRAID seems to do what I want, except it's made for disk arrays, which is my problem. I think that it could work with just 1 data disk and 1 parity disk, in which case the parity disk will probably be a mirror (backup) of the data disk, but I'm not sure.

Other than that it does what I need: checksumming files, ability to verify this, and even repair them from the backup (parity). I'll still run a weekly backup on external media, but this local backup needs to be less manual because it's starting to be a pain to manage.

Are there other tools like SnapRAID which are made for just 1 data disk or filesystem that they protect with automatic checksumming/backup, or should I just use SnapRAID? Does it work fine with just 1 disk?

Because it uses a parity disk for the backup, I'll have to completely wipe my local backup disk before using it with SnapRAID, so I'm hesitant to just "test" it for myself without confirmation. One downside to this is that the parity disk will not be accessible as a normal disk, even though in this case it's not really a parity disk but just a mirror.

So if there's another similar easy-to-use tool for dealing with just backing up and integrity of files of 1 disk instead of a disk array, I'd like to know. Thanks.

Improve this question

4 Answers 4

Reset to default
2

You should have a look at bup

Very efficient backup system based on the git packfile format, providing fast incremental saves and global deduplication (among and within files, including virtual machine images).

bup supports bup-fsck (with par2)

verify or repair a bup repository

Improve this answer
1
  • This... is pretty amazing. I'm shocked of how little-known it is (this is literally the first time for me) and I've searched a lot, despite even working with git quite often. I imagine par2 by itself could be used semi-automatically with a simple script, if one doesn't want incremental backups and only recovery for their latest data. But even so, bup can use par2 too, which makes it even better!
    – kktsuri
    May 22, 2018 at 15:18
1

Here is another more lightweight option (by me):

https://github.com/laktak/chkbit-py

It creates hashes so you can verify the data integrity on your primary and backup media.

Improve this answer
0

I want to know if my files that I haven't accessed in a long time got randomly damaged or not, …

This is exactly one of the motivations behind creating Datimprint, Jordial's free software for data statistics, fingerprint, and verification. Datimprint is written in Java so it runs on Linux, Mac, and Windows. It uses the latest file APIs and encoding standards, and is multithreaded for quick file system traversal and checksum processing.

Datimprint creates a "data imprint" of a directory tree and stores it in datim file.

datimprint generate /var/data --output /var/imprints/data-2022-11-12.datim

You can later check the data tree against the imprint file, to verify a backup or to check for data degradation, for example.

datimprint check /var/backup/data --imprint /var/imprints/data-2022-11-12.datim

The source code is publicly available via its GitHub project.

… and hopefully repair them if possible.

Repairing files based on some backup is a possibility for future enhancement. As the author of Datimprint I welcome you to request new features or report bugs by filing issues. You can provide other feedback in discussions about the program.

Improve this answer
0

integrity of files of 1 disk instead of a disk array

I know you said not ZFS, but here is a hack to consider. I did this for my local backup drives.

  1. Divide the drive into 11 equal partitions
  2. Use each partition as a separate "device". Format it as Raid5 using ZFS (I actually use BTRFS, but I know the risks)

It mounts as a single device. 10 of the partitions space is used for data, and 1 partition space for parity. Whenever you make any file changes, this will give you instant 10% parity of all your data. It's Near Zero management effort- just be sure to run a monthly scrub

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .