I am working on setting up a backup server using a bunch of 3TB disks. I've put a lot of thought into this, and I'd like to use the following setup:

I'll partition the disks with a single partition, these partitions will be setup in raid10 offset -- just two disks to start.

On these, I'll use LVM as there could be quite a few partitions.

On each logical partition, there will be a LUKS device.

Inside each LUKS device, there will be an xfs filesystem.

I've searched quite a bit, but cannot find a good explanation of how to best address the striping inherent to raid10 and xfs. I know that when put directly onto an mdraid device, mkfs.xfs detects the striping and aligns itself. As I understand it, when you add an LVM, this goes out the window. One article I read suggested setting a sizeable (~1M) stripe on the raid device, then setting a consistent chunk on LVM. However, doing so would seem to neglect to consider the possibility for something like the LUKS header to create additional offset.

Is there any effective way to guarantee the alignment of xfs with the raid10 device?

Is the complexity I'm creating by stacking all this mitigate any performance gains of raid10, and should I therefore just use lvm in raid1 and let it worry about the striping?

I also apologize in advance, I think I may have intermittently mixed up the ideas of chunks and stripes in this post.

  • with only two drives, you don't have RAID-10. You have RAID-1 (or, if you don't care about your data, RAID-0), so optimising and testing for something you don't have seems a bit premature. btw, btrfs on luks is another option worth considering. or if you're in no particular hurry, the next release of zfsonlinux (0.7) will have built in support for encrypted zfs filesystems. – cas Jul 23 '17 at 10:19
  • Dm has raid10, which offsets striping on two devices, it's more a conceptual raid10, accomplished by striping redundantly, not a 'true' raid1-0. I've had pretty good success with this soft raid10 elsewhere. The zfs encryption is intriguing, but the system has non-ECC memory, which is apparently a deal breaker on zfs. Source re-raid10: ilsistemista.net/index.php/linux-a-unix/… – osteichthyes Jul 23 '17 at 23:54
  • yes, but you only have two drives. my point was that you can't have raid-10 or anything similar with only two drives, you need at least four. BTW non-ECC isn't a deal breaker for ZFS, most of the systems i run zfs on don't have ECC RAM. also btw, the reason i mentioned btrfs was that it can be layered on top of luks, it's easy to add drives (e.g. to go from raid-1 to raid-10), and it supports error detection and correction of data, snapshots, sub-volumes, and more. – cas Jul 24 '17 at 0:03
  • See the link I posted, mdadm allows for raid10 on two drives. Again, it stripes and offsets the stripes. Btrfs isn't going to work because it would mean luks under the raid, and that would be more complicated for the user mounting on login. I also will probably need to add drives, which would make zfs a lot more challenging. I attempted to edit my comment re ecc, but was locked out. I'm also leary of newly implemented encryption (ala zfs). Raid1 seems like it may be my best bet, though it would likely be no faster than a poorly aligned raid10, which is sort of the point. – osteichthyes Jul 24 '17 at 0:08
  • It would appear I had some misconceptions about zfs. It could readily handle everything I want to do. I'll investigate if centos supports the encryption yet and start experimenting. – osteichthyes Jul 24 '17 at 0:40

I've been traveling for work, and only just now have the time to respond. I've elected to answer my own question in case this is of use to others later.

First, the comments claiming that Linux md-raid's raid10 on two devices is impossible were wrong. To be clear, Linux's md-raid's raid10 is not some standard raid1+0. Md-raid's raid10 essentially amounts to a raid1, striped. If you spend a few minutes thinking about the benefits of raid1 and the benefits of raid0, it becomes clear that they are not mutually exclusive, and could readily be combined. Indeed, Wikipedia specifically addresses this situation: https://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10

Is it truly raid10? NO. Does it still mirror and stripe? Yes. Do copies of each block appear on more than one device, offering a real mirror with some speed benefits? Maybe. The comments referring to the seek times of using raid10 on two drives may indeed prevail, killing the performance benefits, you can see more about that here: https://blog.a2o.si/2014/09/07/linux-software-raid-why-you-should-always-use-raid-10-instead-of-raid-1/ Can the array be scaled and expanded by adding drives? Yes. Is there anything wrong with using two disks for a md-raid 10 array? By all accounts, absolutely not, but you may not see the anticipated speed increases.

Second, ZFS. Today, the ZFS on Linux accepted a pull that does introduce encryption. That's AWESOME! I think however that ZFS is often treated as the fix-all to fs-level questions and concerns. For my needs, it yields a level of complexity for the install that I'd rather avoid. It also tends to use a lot of memory and can, under certain workloads be very slow. Sources (some of these tests have issues in their methodology, but they give a broader, general impression): [I was going to post some links here, but cannot as I do not yet have 10 posts -- a bit of googling should yield a number of relevant articles]

Moving on to my actual question: can you align xfs stripes to the md-raid striping? The answer seems to be no. After considerable googling, a lot of poking around and the like, it seems that xfs inteligently aligns itself when you build the fs directly on the partition. Indeed, ext4 does this, too. However once you layer lvm and luks in, xfs (and ext4) can no longer detect the partition stripe. By defining the stripe size when creating the xfs partition, you can specify a stripe width, you cannot however force the stripe to align. Some have suggested moving lvm, raid, and other headers to the end of partitions and then offsetting by 1MB (or some amount divisible by the stripe size), but there is no real guarantee that as lvm assigns more space to a volume it will do so in alignment with the raid stripe.

To clarify, this affects md-raid striped topologies with striping file systems. Just because I specified two drives in my question and comments revolved around that, this would crop up with 4 drives, 8 drives, or 800 drives. It would also appear in any raid setup that used striping, be it raid0, raid5, raid6, or raid10. This ultimately is the appeal of using systems like zfs or btrfs that integrate partitioning, device management/pooling, etc.

So, what am I going to do? The stability and proven reliability of md-raid, lvm, and xfs are very appealing. The seek times associated with md-raid raid10 is not appealing. I will likely use the raid1 setting in lvm, which is well supported by my chosen os, centos. I will then create my lvm volumes and in those, create my luks volumes. On these, I will build by xfs file systems. These last two are a bit up in the air. The new encryption feature in ext4 is appealing, and I may replace the luks+xfs bits with encrypted ext4.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.