Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm experiencing very poor read performance over raid1/crypt/lvm. In the same time, write speeds are about 2x+ faster on the same setup. On another raid1 setup on the same machine I get normal read speeds (maybe because I'm not using cryptsetup).

OS related disks: sda + sdb. I have raid1 configuration with two disks, both are in place. I'm using LVM over the RAID. No encryption. Both disks are WD Green, 5400 rpm.

IO test results on this raid1:

dd if=/dev/zero of=/tmp/output.img3 bs=8k count=256k conv=fsync
 - 2147483648 bytes (2.1 GB) copied, 22.3392 s, 96.1 MB/s

sync
echo 3 > /proc/sys/vm/drop_caches
dd if=/tmp/output.img3 of=/dev/null bs=8k
 - 2147483648 bytes (2.1 GB) copied, 15.9 s, 135 MB/s

And here is the problematic setup (on the same machine). Currently I have only one sdc (WD Green, 5400rpm) configured in software raid1 + crypt (luks, serpent-xts-plain) + lvm. Tomorrow I will attach another disk (sdd) to complete this two-disk raid1 setup.

IO tests results on this raid1:

dd if=/dev/zero of=output.img3 bs=8k count=256k conv=fsync
2147483648 bytes (2.1 GB) copied, 17.7235 s, 121 MB/s

sync
echo 3 > /proc/sys/vm/drop_caches
dd if=output.img3 of=/dev/null bs=8k
2147483648 bytes (2.1 GB) copied, 36.2454 s, 59.2 MB/s

We can see that the read performance is very very bad (59MB/s compared to 135MB/s when using no encryption). Nothing is using the disks during benchmark. I can confirm this because I checked with iostat and dstat.

Details on the hardware:

  • disks: all are WD green, 5400rpm, 64mb cache.
  • cpu: FX-8350 at stock speed
  • ram: 4x4GB at 1066Mhz.

Details on the software:

  • OS: Debian Wheezy 7, amd64
  • mdadm: v3.2.5 - 18th May 2012
  • LVM version: 2.02.95(2) (2012-03-06)
  • LVM Library version: 1.02.74 (2012-03-06)
  • LVM Driver version: 4.22.0
  • cryptsetup: 1.4.3

Here is how I configured the slow raid1+crypt+lvm setup:

parted /dev/sdc

  1. mklabel gpt
  2. type: ext4
  3. start: 2048s
  4. end: -1

Now the raid, crypt and the lvm configuration:

  1. mdadm --create /dev/md1 --level=1 --raid-disks=2 missing /dev/sdc
  2. cryptsetup --cipher serpent-xts-plain luksFormat /dev/md1
  3. cryptsetup luksOpen /dev/md1 md1_crypt
  4. vgcreate vg_sql /dev/mapper/md1_crypt
  5. lvcreate -l 100%VG vg_sql -n lv_sql
  6. mkfs.ext4 /dev/mapper/vg_sql-lv-sql
  7. mount /dev/mapper/vg_sql-lv_sql /sql

So guys, can you help me identify the reason and fix it? It has to be something with the cryptsetup as there is no such read slowdown on the other setup (sda+sdb) where no encryption is present. But I have no idea what to do.

Thanks!

share|improve this question

migrated from serverfault.com May 28 at 12:28

This question came from our site for system and network administrators.

    
"Both disks are WD Green, 5400 rpm." That'll be your problem. –  Tom O'Connor May 27 at 15:04
3  
I find it odd that WD Green drives are used but a power-hungry AMD is also used... –  Nathan C May 27 at 15:08
    
That's just like having a bowl of french fries, and loads of ketchup, because ketchup comes from tomatoes, and they're healthy, right? –  Tom O'Connor May 27 at 15:25
1  
What I expect is the crypted raid to be as fast on reading as the non-crypted raid. Why? Why would you expect that? –  HopelessN00b May 27 at 15:38
2  
@Android5360 Except that encryption (WRT symetric block ciphers) isn't slower than decryption. They're mathematically equivalent operations, the same speed in each direction. You're seeing the impact of caches, buffers and seek times - disk operations - not anything inherent in encryption of decryption (of symmetric block ciphers). –  HopelessN00b May 27 at 15:54

2 Answers 2

It should be obvious that encryption adds considerable overhead, but disregarding that:

You're also using non-AES encryption which means you don't get the acceleration from your processor (AES hardware acceleration). You're also using a desktop-quality processor and extremely slow consumer drives.

This setup is exactly why you're having such poor performance. Put decent drives in along with a server-quality processor (is this even a server?) and use AES for LUKS.

share|improve this answer
    
Regarding that: decryption usually is faster than encryption, isn't it?.. –  poige May 27 at 15:00
    
@poige Not always as it depends on caching. Unlike reads, writes are cached by the drive's controller and reported written when the drive doesn't have the data written to it yet. Reads are cached by the OS, but the OP dropped those caches before reading. Encryption/decryption are typically similar in speeds. –  Nathan C May 27 at 15:05
    
@poige Encryption is generally symetrical - equally fast in both directions. If you're experiencing faster decryption than encryption, it's probably actually an issue of your disk read being faster than your disk write. –  HopelessN00b May 27 at 15:06
    
you guys awesome. serverfault.com/a/598838/67675 –  poige May 27 at 15:11
1  
"Raid 1 is just a mirror, so performance isn't improved in this case" again you're terribly wrong. OMG, study the subject before trying to teach others. kernelnewbies.org/KernelProjects/Raid1ReadBalancing –  poige May 27 at 16:16

It's usual not correctly done testing. Loading Linux Software RAID with 8k read operation is the way to ask it beahave poorly (you did). Try using larger bs.


I'm also adding this output of cryptsetup benchmark for ppl in comments below

#  Algorithm | Key |  Encryption |  Decryption
     aes-cbc   128b   476.6 MiB/s  1891.3 MiB/s
 serpent-cbc   128b    79.1 MiB/s   222.6 MiB/s
 twofish-cbc   128b   135.0 MiB/s   158.7 MiB/s
     aes-cbc   256b   194.7 MiB/s   892.5 MiB/s
 serpent-cbc   256b    46.2 MiB/s   211.4 MiB/s
 twofish-cbc   256b   144.6 MiB/s    93.3 MiB/s
     aes-xts   256b  1542.0 MiB/s  1727.0 MiB/s
 serpent-xts   256b   114.0 MiB/s   200.2 MiB/s
 twofish-xts   256b   111.4 MiB/s   131.7 MiB/s
     aes-xts   512b   573.6 MiB/s   829.3 MiB/s
 serpent-xts   512b   199.8 MiB/s   191.1 MiB/s
 twofish-xts   512b    81.8 MiB/s   107.3 MiB/s

-- clearly seen (not always, but often) decryption is faster.

share|improve this answer
    
Wonderful that your system does this, that, or the other, but you're seriously misinformed about encryption at a fundamental level. Here's a good rundown of the reason for the apparent speed difference between decryption and encryption on symmetric block ciphers, like those being discussed in this question. –  HopelessN00b May 27 at 15:50
    
No chances for me to look at this since you've already tried explaining things using "caches". Thanks, I'm not buying. ) –  poige May 27 at 16:05
    
@HopelessN00b You are aware that the linked answer claims reading to be faster whereas this question is about reading being slower? Furthermore this seems a strange argument to me. Who would implement parallel execution within a drive sector instead of working on different sectors (or groups of sectors) in parallel? –  Hauke Laging May 28 at 12:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.