XFS on LVM on soft raid10 stripe alignment

Question

I am working on setting up a backup server using a bunch of 3TB disks. I've put a lot of thought into this, and I'd like to use the following setup:

I'll partition the disks with a single partition, these partitions will be setup in raid10 offset -- just two disks to start.

On these, I'll use LVM as there could be quite a few partitions.

On each logical partition, there will be a LUKS device.

Inside each LUKS device, there will be an xfs filesystem.

I've searched quite a bit, but cannot find a good explanation of how to best address the striping inherent to raid10 and xfs. I know that when put directly onto an mdraid device, mkfs.xfs detects the striping and aligns itself. As I understand it, when you add an LVM, this goes out the window. One article I read suggested setting a sizeable (~1M) stripe on the raid device, then setting a consistent chunk on LVM. However, doing so would seem to neglect to consider the possibility for something like the LUKS header to create additional offset.

Is there any effective way to guarantee the alignment of xfs with the raid10 device?

Is the complexity I'm creating by stacking all this mitigate any performance gains of raid10, and should I therefore just use lvm in raid1 and let it worry about the striping?

I also apologize in advance, I think I may have intermittently mixed up the ideas of chunks and stripes in this post.

Answer 1

I've been traveling for work, and only just now have the time to respond. I've elected to answer my own question in case this is of use to others later.

First, the comments claiming that Linux md-raid's raid10 on two devices is impossible were wrong. To be clear, Linux's md-raid's raid10 is not some standard raid1+0. Md-raid's raid10 essentially amounts to a raid1, striped. If you spend a few minutes thinking about the benefits of raid1 and the benefits of raid0, it becomes clear that they are not mutually exclusive, and could readily be combined. Indeed, Wikipedia specifically addresses this situation: https://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10

Is it truly raid10? NO. Does it still mirror and stripe? Yes. Do copies of each block appear on more than one device, offering a real mirror with some speed benefits? Maybe. The comments referring to the seek times of using raid10 on two drives may indeed prevail, killing the performance benefits, you can see more about that here: https://blog.a2o.si/2014/09/07/linux-software-raid-why-you-should-always-use-raid-10-instead-of-raid-1/ Can the array be scaled and expanded by adding drives? Yes. Is there anything wrong with using two disks for a md-raid 10 array? By all accounts, absolutely not, but you may not see the anticipated speed increases.

Second, ZFS. Today, the ZFS on Linux accepted a pull that does introduce encryption. That's AWESOME! I think however that ZFS is often treated as the fix-all to fs-level questions and concerns. For my needs, it yields a level of complexity for the install that I'd rather avoid. It also tends to use a lot of memory and can, under certain workloads be very slow. Sources (some of these tests have issues in their methodology, but they give a broader, general impression): [I was going to post some links here, but cannot as I do not yet have 10 posts -- a bit of googling should yield a number of relevant articles]

Moving on to my actual question: can you align xfs stripes to the md-raid striping? The answer seems to be no. After considerable googling, a lot of poking around and the like, it seems that xfs inteligently aligns itself when you build the fs directly on the partition. Indeed, ext4 does this, too. However once you layer lvm and luks in, xfs (and ext4) can no longer detect the partition stripe. By defining the stripe size when creating the xfs partition, you can specify a stripe width, you cannot however force the stripe to align. Some have suggested moving lvm, raid, and other headers to the end of partitions and then offsetting by 1MB (or some amount divisible by the stripe size), but there is no real guarantee that as lvm assigns more space to a volume it will do so in alignment with the raid stripe.

To clarify, this affects md-raid striped topologies with striping file systems. Just because I specified two drives in my question and comments revolved around that, this would crop up with 4 drives, 8 drives, or 800 drives. It would also appear in any raid setup that used striping, be it raid0, raid5, raid6, or raid10. This ultimately is the appeal of using systems like zfs or btrfs that integrate partitioning, device management/pooling, etc.

So, what am I going to do? The stability and proven reliability of md-raid, lvm, and xfs are very appealing. The seek times associated with md-raid raid10 is not appealing. I will likely use the raid1 setting in lvm, which is well supported by my chosen os, centos. I will then create my lvm volumes and in those, create my luks volumes. On these, I will build by xfs file systems. These last two are a bit up in the air. The new encryption feature in ext4 is appealing, and I may replace the luks+xfs bits with encrypted ext4.