3

I want to detect silent corruption of block devices similar to how BTRFS does that for files. I'd even like to do that below BTRFS (and disable BTRFS's native checksumming) so that I can tweak more parameters than BTRFS allows. DM-Integrity seems like the best choice and in principle it must be doing the same thing as BTRFS.

The problem is that it's incredibly, unusably slow. While sequential writes on BTRFS are 170+ MiB/s (with compression disabled), on DM-Integrity they're 8-12 MiB/s. I tried to match DM-Integrity parameters with BTRFS (sector size, hashing algorithm, etc) and I tried lots of combinations of other parameters (data interleaving, bitmapping, native vs generic hashing drivers, etc).

The writes were asynchronous, but the speed was calculated based on the time it took for writes to be committed (so I don't think the difference was due to memory caching). Everything was on top of a writethrough Bcache, which should be reordering writes (so I don't think it could be BTRFS reordering writes). I can't think of any other reason that could explain this drastic performance difference.

I'm using Debian 11 with a self-compiled 6.0.12 Linux kernel and sha256 as my hashing algorithm. My block layers are (dm-integrity or btrfs)/lvm/dm-crypt/bcache/dm-raid.

Is there a flaw in my testing? Or some other explanation for this huge performance difference? Is there some parameter I can change with DM-Integrity to achieve comparable performance to BTRFS?

5
  • 2
    Why? What actual benefit do you think you'll get from DM-Integrity that you wouldn't get from btrfs? BTW, if you want pool (or mirror/raidz vdev, actually) level error detection & correction, you might want to look into zfs instead of btrfs. It'll certainly be a lot simpler and easier to work with than layering "(dm-integrity or btrfs)/lvm/dm-crypt/bcache/dm-raid", while providing all of the features of those and a whole lot more (incl. transparent compression, cheap snapshots, send/recv). And it'll perform a lot better.
    – cas
    Dec 30, 2022 at 13:32
  • @cas I want to eventually use the integrity functionality of cryptsetup (which is literally just a dm-integrity layer nested within dm-crypt), and even more eventually aead (which requires the prior layout). I already use integrity checking at the raid-level, but since it's below bcache and dm-crypt, it can't detect bcache/ssd errors or intentional tampering.
    – ATLief
    Dec 30, 2022 at 14:38
  • @cas Also, btrfs can't protect other block-devices/lvs or other filesystems. Personally I prefer having separate layers so I can tweak more things and switch technologies as I see fit (for example, bcache to dm-cache or mdadm to lvm-raid). I've considered zfs in the past (although admittedly don't know a ton about it), but not being built into Linux and having less configurability of the underlying technologies are strikes against it for me. Btrfs has the other features you mentioned.
    – ATLief
    Dec 30, 2022 at 14:47
  • Not being built-in to the kernel isn't a big deal. ZFS dkms packages are available for most distros, and dkms makes compiling and installing the kernel module completely trivial. If you've never actually tried it, I suggest setting up a VM with a boot/root drive and a bunch of 100M (which is way more than enough for experimenting with) or so extra virtual drives, and try them in different configurations - mirror, raidz-1, raid-z2, with various options enabled - encryption, compression, etc. add enough virtual drives and you can set up two separate pools for experimenting with send/recv.
    – cas
    Dec 30, 2022 at 15:40
  • BTW, the license incompatibility is only a problem if you intend to distribute binaries that combine the linux kernel (GPL2) and ZFS (CDDL). Ubuntu thinks it's not a problem, but legally they're wrong about that (practically, neither oracle nor the linux devs are likely to sue). Compiling and using ZFS with linux is perfectly legal as neither restrict what you do in-house, on your own system(s)...the license issue only rears up when you want to distributed the combined binaries.
    – cas
    Dec 30, 2022 at 15:44

1 Answer 1

0

I've found at least 2 flaws with my prior testing, but I don't yet know whether they entirely explain the discrepancy. Neither the journal/bitmap mode nor the integrity algorithm are stored in the DM-Integrity superblock. While I provided those options when formatting, I didn't provide those options while activating, and thus they would have reverted to their default values (of journaling mode and sha256).

The journaling mode has a very large effect on performance, which probably explains most of the discrepancy. I still need to recreate all of the tests to know the true impact.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .