12

LVM and mdadm / dmraid are both offering software RAID functionality on Linux. This is pretty much a follow-up post to this question from 2014. Back then, @derobert recommends to prefer mdadm over LVM raid for it's matureness - but that was over 4 years ago. I can imagine, things have changed since then. But I have never used LVM raid before, and I couldn't find any recent experiences on that either.

So what's the state of LVM raid? Has it grown more mature now? Are the flaws mentioned in @derobert's post resolved now, or still present? What about

  • stability,
  • features (grow, shrink, convert),
  • repair and recovery,
  • community support,
  • performance

of LVM raid in comparison to mdadm?

I want to find out if people actually use it, or if everyone still sticks with mdadm. Is it more recommended to use LVM on top of mdadm for logical volume management, or is it fine to let LVM manage the raid as well? Is it even considerable to use LVM raid instead of mdadm, even if you don't expect to need the advantages of logical volume management?

I considered just adding a comment under that original answer, asking @derobert to update his post, but decided for asking a new question. I want to reach out to other members, get new and fresh experiences and not just bring the old post to a present tense.

9
  • 1
    I prefer ZFS actually. Apr 29, 2019 at 11:04
  • LVM on top of LUKS on top of MDADM. Or ZFS on top of LUKS on top of MDADM, which gives me encryption that ZFS can't (quite yet) provide. Apr 29, 2019 at 11:44
  • 1
    This article says that extending an LVM mirror is possible, and gives some caveats.
    – Tom Hale
    Nov 22, 2019 at 8:04
  • 2
    I had the same question and would be keen to know the latest status. LVM is a great tool and using it without mdadm seems advantageous. Jan 15, 2020 at 23:12
  • @roaima Just out of interest, could you go into detail what the problem with ZFS encryption is? Feb 20, 2022 at 11:26

1 Answer 1

7

I am senior in a team where we have several environments (around 5 now if I am not mistaken, but will be getting some more by the end of the year).

The environments weight from 8 to 25 physical hosts (often fully loaded to the max on CPUs and memory), and somewhere from 50 to 400 virtual servers running on each.

Storage is always on fibrechannel, but fabric switches and disk arrays differ quite a lot depending on a client (what deal they got, relationship with storage companies they have etc).

Each environment spans 2 datacenters and those are interconnected by DWDM (making the two DC networks (both ip and fc) appear as one). Networks are of course split by vlans and fc zoning into smaller pieces.

We have hypervisors ranging from vmware (it's cool when it works), raw qemu+kvm under virsh, qemu+kvm under virsh runing under pacemaker clusters and qemu+kvm under virsh coordinated by ovirt.

We use both hypervisor clustering and in-vm clusters.

The oldest environment is 10+ years, but renovated cyclically (incredible chore, if you can imagine).

Why am describing all this? As you can see, such zoos are rather vibrant. I am grateful to have seen all this technology at work daily, for last almost 4 years (my god time flies). I should not need to add, that in environments like these, there are often thousands of LVM volumes, and during you work you'll eventually touch all of them.

Oldest environment is fully LVM based and what can I say: it works, until it doesn't.

The main problem I have with LVM is that if it does some of it's stupidities, you are on your own. It often happens when you least expect it (or rather need it) and in production environment (not devevelopment, not testing or preproduction).

Also commands are quite baroque, and kinda reversible but only to the point, when you start pumping data on the volume. Once that happens and you discovered a mistake only after, you should simply burn the volume and start a new. It will be faster and probably more robust and you'll do less errors.

I've seen several weird-ass LVM bugs that basically meant a loss of whole LVM setup.

Most jarring one was neophyte admin's extension of LVM stack by few hundredth Gigs of storage, which resulted in extended LV suddenly reporting -4trilions in size. The weird negative size of the volume made it impossible to run umount, fsck or any other fixing tools and introduced other problems. Fortunately descending into directories still worked, so we rebuilt the whole VM again and used rsync to transfer (mostly read only) data. Data team then did analysis and they have not found any data loss - so it was probably just free space getting somehow mucked up. But the final result was that LVM caused such complication and locked volume in a way that not even basic data recovery tools were able to run.

Also the original system was lost and had to be replaced and then dismantled. I and our architect, we did the analysis of commands issued, and it was done completely by the book, so I am not sure what happened there.

We also have miniscule amounts of LVM mirrors using cling extension (to make LV sub-devices stick to proper datacenters on physical layer). This ensures that should cross DC link break that mirror would assemble at least on one side. All I will say is, that you don't want to be dealing with these setups in the middle of the night.

We never had the balls to use LVM snapshots, despite them being supposedly fixed. There are many horror stories about them on the net and I am not willing to try them, especially since now we have tools to avoid these issues entirely.

Regarding a normal use, my major issue with LVM and overall state of linux filesystems, is their inability to check their own shit.

I have not had time to dive deep into LVM mirroring, but I still have not found a person or explicit written confirmation, whether LVM mirror actually calculates checksums of the blocks (and any checksum would do even crc32). So even if I run LVM mirror recalc, what is it actually doing? And if a progress counter reaches 100% and mismatches counter is 0, does it mean just that data between mirrors match, or was full checksumming done and there are no errors (those two are quite different things, right)?

Second issue with LVM I have is more indirect: most usual filesystems: ext4, xfs, jfs one puts on top of it don't have checksums for userdata. This might have been cool in 90s of the last century, but is a big nono nowadays. Now we know better: you don't actually care about userdata metadata that much, as you care about actual userdata content. What is the point of storage anyway? To know when foto last changed, or to actually see what is on that photo?

I noticed there were some plans to add at least rudimentary userdata cheksums to xfs, but that is not there yet.

Why does it matter? In clustering environments, fencing often happens, and sometimes such fencing ends up in stonith loops. So after an incident, when cluster finally stabilizes, now how can you say if data are allright?

With LVM + FS you simply cannot, as there is no way. Yeah and comparing to backup... let's leave it at that, right?

Finally, LVM is brittle. Especially in passive/active cluster volumes or even clustered lvm setups, it requires you to mark up, which lvm parts construct root, into the lvm.conf. Otherwise LVM doesn't know which parts are clustered and which are root it wants to boot, so it assembles them all - this is a big nono in clusters. To solve this you need then to ensure copy of this lvm.conf gets copied also into the initrd (looking at you dracut). If you don't ensure all this, next time when both (or more) nodes boot at the same time, they'll all try to activate same lvm volumes - and you can imagine the fun then.

I've lost count how many times I had to fix this after neophite admin configured and assembled a cluster (and they were instructed about it sepecifically by me). Even with the written notes they often forgot, that means the step is hard.

This is very nice ticking bomb you can leave for your colleagues to solve, because usually it shows only after the first fencing :).

So during the years I became persuaded LVM should go - it served it's purpose, but ZFS and BTRFS can do everything it does much, much better, and even do much more.

Both ZFS and BTRFS store all the pool metadata directly in the pool. No dracut bound btrfs/zfs.confs, the pool is completely disconnected from init ramdisk as it should have been from the start. You can specify root on pool to use in kernel's command line.

Most importantly, after any malfunction, you can run scrub on both of them, BTRFS and ZFS, and actually rescan your storage for real userdata(!) errors. If anything, scrub is the single most important killer feature, why you should want to run any of the nextgen FSes. With scrub you can be actually confident you didn't have any silent data corruption.

Second thing of greatest importance is the fact, that the snapshots do work. Always. Snapshot is basic unit of work of COW system, a linchpin of it all, so if it didn't work, you would have bigger problems.

Finally, if you are on the "poorer" side, then BTRFS is a way to go simply because of crazy amount of transformation dances, it is able to do with the data. It can split them, shrink them, rebalance them and do many other strange things to them. You can dance with disks in BTRFS system until you find your sweet optimum. It's ultimate wet dream of cheap linux admin (that means 90% of linux admins), who cannot afford the storage. Or who likes to rebuild storage 3 times, while the same data are still being accessed, until they find optimal solution.

ZFS is slowly growing capabilities in this area, but it is still quite far from malleability of BTRFS. But one thing about ZFS is that unlike BTRFS, which is kinda a linux buggy, ZFS is robust data truck (or even tanker).

ZFS has had insane amount of testing done, tools have incredible polish to them, just comparing working with it and with BTRFS, you see immediately how much money was spent and where. Hint: you cannot even run query commands against BTRFS pool without root access, while with ZFS you have full access control list for every ZFS operation, and you can delegate that to specific users.

So to conclude, in my hunch ZFS will slowly match BTRFS in feature parity in a few years, while BTRFS, which I guesstimate is only 20% finished, will remain unfinished forever, as is too much common in linux world.

Any of the two will save you incredible amount of LVM hassle though.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .