I'm trying to determine if I should re-setup my RAID array due to poor IO performance. First off, the system:

  • i7 920
  • 4 4TB WD 5400 Green drives
  • CentOS 6.3 host

Secondly, the disk setup:

  • /dev/sda2,b2,c2,d2 are individually LUKS encrypted
  • /dev/mapper/a2,b2,c2,d2 are all part of a software RAID5 /dev/md1
  • /dev/md1 has LVM on top of that
  • LVM is used to separate /, /storage, and swap

I choose this structure to allow for multiple instances of kcryptd, thinking that by doing this, I would get multithread support on encryption since one instance is running per drive. However, I'm beginning to wonder if that was a good idea.

For instance, if I run a heavy decompression routine on a RAR file of random data, my IO Wait goes up to around 25% and it slows the overall system down. I'm wondering if all the instruction sets are getting backed up somehow due to all the kcryptd processes.

Therefore, I'm considering changing to:

  • /dev/sda2,b2,c2,d2 are put into /dev/md1
  • /dev/md1 is encrypted and mapped to /dev/mapper/1
  • LVM on top of /dev/mapper/1

This would drop down to a single kcrpytd process, which could be a bottleneck in it's own right, too. Does anyone think this will help with my IO issue?


Your layering is suboptimal because putting the raid 5 on top of the encryption means that you increase the number of encrypt/decrypt operations by 25 % - since 4 * 4 TB are encrypted.

When putting the encryption on top of the raid 5 only 3 * 4 TB are encrypted.

The reasoning behind that is: you don't have to encrypt parity data (which takes up 4 TB in your example) of encrypted data because it does not increase your security.

Your presumption about multiple kcrypt processes is just that. When basing decisions on it, it is a premature optimization that may have quite the opposite effect. Your i7 is quite beefy, probably even including some special instructions that help to speed up AES - and the Linux kernel includes several optimized variants of cryptographic primitives that are automatically selected during boot.

You can verify if the optimized routines for your CPU are used via looking at /proc/cpuinfo (e.g. flag aes there), /proc/crypto, lsmod (unless the aes modules are compiled into the kernel) and the kernel log.

You should benchmark the throughput of kryptd without involving any slow disks to see what the upper bound really is (i.e. on a RAM disk using iozone).

To be able to diagnose potential performance issues later it is also useful to benchmark your RAID-setup of choice without any encryption to get an upper bound on that end.

In addition to the crypto topic, RAID 5 involves more IO-Operations than RAID 1 or 10. Since storage is kind of cheap perhaps it is an option to buy more harddisks and use another RAID level.

| improve this answer | |
  • Marking this the best answer as you were the first to point out the parity oversight on my part, while pointing out RAID 10 differentiation. I'll do some benchmarking and move forward in this route. – Fmstrat Mar 14 '14 at 18:44

I would Raid 1+0 [a2,b2]+[c2,d2], then LVM over LUKS.


$ sudo mdadm --create /dev/md0 -v --raid-devices=4 \
      --level=raid10 /dev/sdb1 /dev/sdc1 /dev/sde1 /dev/sde1

NOTE: Structuring it this way will create a stripe of mirrors allowing a maxium of 2 disks to fail (one in each mirror max) and it will give you a total/2 space as opposed to raid5 which is total*~0.75.

Also I believe this schema is significantly faster because RAID5 is known to hurt performance, but you'll have less space available.

You can also check the cipher, although I think aes-cbc-essiv is the default and reasonably fast, but you could use aes-xts-plain which should be faster.

| improve this answer | |
  • Can you elaborate on the RAID config. you're suggesting he use? Try and dumb it down a bit more so that others that may pass by this Q&A will be able to learn from it as well! – slm Feb 9 '14 at 17:38
  • When creating your raid do something like: mdadm --create /dev/md0 -v --raid-devices=4 --level=raid10 /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2 and it will create a stripe of mirrors allowing a maxium of 2 disks to fail (one in each mirror max) and it will give you a total/2 space as opposed to raid5 which is total*~0.75 – Roberto Rodriguez Alcala Feb 9 '14 at 17:41
  • 1
    How about the LVM and LUKS? I'm only asking to elaborate b/c this area is poorly understood by most and your A will likely help many! – slm Feb 9 '14 at 17:45
  • This seems like a good methodology, but it doesn't specifically answer my questions about LVM and LUKS. It also is a solution that changes available size and structure (I actually used to do exactly this, but switch to Raid 5 with a hot-spare and external backups). – Fmstrat Mar 14 '14 at 18:43

Your setup means that more data has to be encrypted in total when writing (the parity data). If your encryption is already slow, the multi core property may not be sufficient to offset that. On reads it should not make a difference (parity data is normally not read). That is not yet considering any side effects with mdadm timing or whatever.

I took a different approach; instead of making one big RAID, I partitioned my disks and created several smaller ones (e.g. 8x 250G partitions on a 2TB disk). That means 8 RAIDs instead of 1, 8 LUKS containers, and LVM ties it all back together in one big VG.

Then as long as you have processes working in different areas of the disk, the various LUKS containers and RAIDs would work independently of one another. It's not true parallel encryption (does the kernel still not support that on its own?), but it worked really well for me.

I kept that setup around even on my new Haswell box where encryption is not an issue whatsoever thanks to AES-NI. I did this because there are other positive side effects. For example a single defective sector would cause only a 250G portion of a disk to drop out of the RAID, while the other 1750G stay redundant; or if there is a bug like the RAID5 kernel panic in 3.13.0, only one of the RAIDs has to resync instead of all of them.

At the same time I didn't notice any performance issues, unlike other solutions such as write intent bitmap etc.

| improve this answer | |
  • Interesting methodology, but considering maxschlepzig's comment, the i7 may be optimized enough for it not to matter. I also worry that since it's a 4 core with hyperthreading, splitting up into 8's may cause thread waits, which could be what I'm running into now as well. – Fmstrat Mar 14 '14 at 18:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.