Filesystem with checksums?

Question

I have a single hard drive. I want to use a filesystem that will give me less storage space, but as a tradeoff, give me checksums or any other method to help preserve data integrity.

It is my understanding that something like ext4 or xfs will not do this, and thus you can suffer from silent data corruption, aka bitrot.

zfs looks like an excellent choice, but everything I have read says you need more than one disk to use it. Why is this? I realize having only one disk will not tolerate a single disk failure, but that is what multiple backup schemes are for. What backups won't help with is something like bitrot.

So can I use zfs on a single hard drive for the single purpose of preventing bitrot? If not, what do you recommend?

Answer 1

You could use either ZFS or btrfs.

Both of them are copy-on-write filesystems with error detection (and correction too, if there's sufficient redundancy to repair the original data - e.g. mirror drives or RAID-Z), transparent compression, snapshots, etc.

ZFS allows you to set the copies attribute on a dataset to keep more than one copy of a file - e.g. on ZFS you can run zfs set copies=2 pool/dataset to tell ZFS to keep two copies of everything on that particular dataset - see man zfsprops and search for copies=. I think btrfs has a similar feature, but it's been a long time since I used btrfs and can't find it in the docs.

These extra copies do provide redundancy for error correction (in case of bitrot) but won't protect you from disk failure. You'll need at least a mirror vdev (i.e. RAID-1) for that, or make regular backups (but you should be doing that anyway - RAID or RAID-like tech like ZFS or btrfs is NOT a substitute for backups).

Backing up could be as simple as using zfs snapshot and zfs send/zfs receive to send the initial and then incremental backup to a single-drive zfs pool plugged in via USB. Or to a pool on another machine over the network. Even using zfs send to store the backup in files on a non-ZFS filesystem is better than nothing.

If your machine has the physical space and hardware to support a second drive, you should add one. You can do this when you first create a pool, or you can add a mirror drive to any single-drive or mirror vdev at any time with zpool attach pool device new-device.

NOTE: it's important to use zpool attach, not zpool add for this. attach adds a mirror to an existing drive in a vdev, while add adds another vdev to an existing pool. Adding a single-drive vdev to an existing pool will effectively make a RAID-0 with the other vdevs in the pool, putting ALL of the data at risk. This is a fairly common mistake, and (if the pool contains any RAID-Z vdevs), the only fix is to backup the entire pool, destroy it, re-create it from scratch, and restore. If the pool only has mirror or single-drive vdevs (i.e. no RAID-Z vdevs), it is possible to use zpool remove to remove an accidentally added single drive.

Answer 2

I would add to @cas answer that checksums can be done on the block level too with dm-integrity and then you can use any filesystem on top of that. Similarly to checksums in btrfs dm-integrity provides only error detection so you need RAID to be able to actually fix the errors. You can do this manually by placing the integrity device under each leg of a RAID 1 using integritysetup and mdadm but that might be a little bit complicated, newer versions of LVM support creating LVM RAID with integerity simply by using

lvcreate --type raidN --raidintegrity y

As I said you can then place any filesystem on top of the RAID logical volume and LVM will take care of detecting the errors and correct them on read.