1

I currently setup a NAS with Ubuntu 18.04 on a Netgear ReadyNAS 102device. The machine has a Armada Marvell 370/XP SoC with a hardware encryption accelerator called "CESA". The kernel drivers for CESA are running. As I could verify with cat /proc/crypto:

[...]
name         : cbc(aes)
driver       : mv-cbc-aes
module       : marvell_cesa
priority     : 300
refcnt       : 1
selftest     : passed
internal     : no
type         : skcipher
async        : yes
blocksize    : 16
min keysize  : 16
max keysize  : 32
ivsize       : 16
chunksize    : 16
walksize     : 16
[... other marvel_cesa entries are available too...]

I've setup two disks (normal hdds with rotating discs) as a RAID1 via mdadm. The resulting RAID device I use is called /dev/md127. It's write speed is about 65 MB/s:

# dd if=/dev/zero of=/dev/md127 bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 15.945 s, 65.8 MB/s

The benchmark of cryptsetup tells me, that encryption using "aes-xts" should achieve about 30 MB/s (which cryptsetup uses per default; since "aes-cbc" is not secure anymore).

# cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1        72979 iterations per second for 256-bit key
PBKDF2-sha256     125547 iterations per second for 256-bit key
PBKDF2-sha512      49498 iterations per second for 256-bit key
PBKDF2-ripemd160   69497 iterations per second for 256-bit key
PBKDF2-whirlpool   18875 iterations per second for 256-bit key
argon2i       4 iterations, 25926 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
argon2id      4 iterations, 26234 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
#     Algorithm |       Key |      Encryption |      Decryption
        aes-cbc        128b        55.8 MiB/s        56.6 MiB/s
    serpent-cbc        128b        17.9 MiB/s        17.7 MiB/s
    twofish-cbc        128b        22.3 MiB/s        25.2 MiB/s
        aes-cbc        256b        53.0 MiB/s        53.4 MiB/s
    serpent-cbc        256b        17.9 MiB/s        17.7 MiB/s
    twofish-cbc        256b        23.0 MiB/s        25.3 MiB/s
        aes-xts        256b        30.7 MiB/s        31.0 MiB/s
    serpent-xts        256b        19.9 MiB/s        18.8 MiB/s
    twofish-xts        256b        28.0 MiB/s        28.3 MiB/s
        aes-xts        512b        29.8 MiB/s        30.1 MiB/s
    serpent-xts        512b        20.6 MiB/s        18.8 MiB/s
    twofish-xts        512b        28.0 MiB/s        28.2 MiB/s

Now, I used cryptsetup luksFormat /dev/md127 to create an encrypted block device from it.

   root@ReadyNAS102:~# cryptsetup luksFormat /dev/md127

   WARNING!
   ========
   This will overwrite data on /dev/md127 irrevocably.

   Are you sure? (Type uppercase yes): YES
   Enter passphrase for /dev/md127:
   Verify passphrase:

Then I verified using cryptsetup luksDump which cyphers were really chosen. As far as I can see it's 'aes-xts-plain64` with 512 bit key size for encryption of data and sha256 to hash the passphrase, which ich used to encrypt the symmetric 512-bit-key for AES.

   root@ReadyNAS102:~# cryptsetup luksDump /dev/md127
   LUKS header information
   Version:        2
   Epoch:          3
   Metadata area:  16384 [bytes]
   Keyslots area:  16744448 [bytes]
   UUID:           1f4917c4-ce8c-48d8-99f6-2c19ccb4ecc9
   Label:          (no label)
   Subsystem:      (no subsystem)
   Flags:          (no flags)

   Data segments:
     0: crypt
           offset: 16777216 [bytes]
           length: (whole device)
           cipher: aes-xts-plain64
           sector: 512 [bytes]

   Keyslots:
     0: luks2
           Key:        512 bits
           Priority:   normal
           Cipher:     aes-xts-plain64
           Cipher key: 512 bits
           PBKDF:      argon2i
           Time cost:  4
           Memory:     26404
           Threads:    1
           Salt:       0a ef ca 34 29 9a 23 a1 78 30 8e d8 d7 15 29 21
                       67 84 02 31 f4 b4 66 11 20 c7 82 ab 98 11 cd 93
           AF stripes: 4000
           AF hash:    sha256
           Area offset:32768 [bytes]
           Area length:258048 [bytes]
           Digest ID:  0
   Tokens:
   Digests:
     0: pbkdf2
           Hash:       sha256
           Iterations: 7953
           Salt:       5d 2d de 4d c3 72 e1 ab 70 d2 ba 8a b3 7e 18 89
                       2e 3a 28 e3 51 c8 80 fe ba 73 ce 5e 53 db 72 99
           Digest:     36 6e 0e b0 b7 8c df 28 8f ec 49 bf 97 15 25 68
                       25 b2 92 35 c6 5e 0d ce f4 44 05 8d 3f 5b 78 93

Finally, I opened the luks devices and checked the write speed again:

   root@ReadyNAS102:~# cryptsetup luksOpen /dev/md127 md127-decrypt
   Enter passphrase for /dev/md127:
   root@ReadyNAS102:~# dd if=/dev/zero of=/dev/mapper/md127-decrypt bs=1M count=1000
   1000+0 records in
   1000+0 records out
   1048576000 bytes (1.0 GB, 1000 MiB) copied, 120.121 s, 8.7 MB/s

As you can see, only ~8.7 MB/s were achieved. Shouldn't it be about 30 MB/s or maybe at least over 20 MB/s? (speed tests have been repeated multiple times, to make sure the disk/cpu isn't just busy)

Why is it so slow? And how can I possibly speed it up?

EDIT #1: Requested statistics from top and iostat

As requested here is what top shows while writing to /dev/mapper/md127-decrypt:

   top - 15:30:52 up 1 day, 11:07,  1 user,  load average: 3.33, 1.09, 0.39
   Tasks:  80 total,   2 running,  78 sleeping,   0 stopped,   0 zombie
   %Cpu(s):  1.6 us, 98.4 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
   MiB Mem :    492.5 total,     14.9 free,     58.0 used,    419.6 buff/cache
   MiB Swap:    511.4 total,    510.2 free,      1.3 used.    422.2 avail Mem

     PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
     245 root     -51   0       0      0      0 S  40.5   0.0   3:20.16 irq/38-d0090000
   10791 root       0 -20       0      0      0 I  34.1   0.0   2:24.71 kworker/u3:1-kcryptd
   30143 root      20   0       0      0      0 S   7.9   0.0   0:15.33 dmcrypt_write
   29939 root      20   0       0      0      0 R   7.1   0.0   0:15.21 kworker/u2:1+flush-253:0
   28396 root      20   0    7064   2272   1236 D   6.3   0.5   0:05.60 dd
      28 root      20   0       0      0      0 S   1.6   0.0   0:08.80 kswapd0
     648 sdwarfs   20   0   10468   3560   2756 S   1.6   0.7   0:08.58 sshd
   28398 root      20   0    9360   2632   2124 R   1.6   0.5   0:01.20 top
   28397 root      20   0       0      0      0 I   0.8   0.0   0:00.12 kworker/u2:2-events_unbound
       1 root      20   0   29916   4416   3172 S   0.0   0.9   0:07.86 systemd
       2 root      20   0       0      0      0 S   0.0   0.0   0:00.06 kthreadd
       3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp

And here is what iostat -kx 5 shows:

   avg-cpu:  %user   %nice %system %iowait  %steal   %idle
              0.20    0.00   99.80    0.00    0.00    0.00

   Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
   sda              0.00  410.40      0.00   8494.10     0.00  1713.60   0.00  80.68    0.00    0.56   0.18     0.00    20.70   0.42  17.04
   sdb              0.00  401.80      0.00   8494.10     0.00  1722.20   0.00  81.08    0.00    0.49   0.14     0.00    21.14   0.30  11.92
   md127            0.00 2123.60      0.00   8494.40     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     4.00   0.00   0.00
   md0              0.00    0.60      0.00      2.40     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     4.00   0.00   0.00
   md1              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
   dm-0             0.00 2123.60      0.00   8494.40     0.00     0.00   0.00   0.00    0.00    2.42   5.13     0.00     4.00   0.47 100.00

   avg-cpu:  %user   %nice %system %iowait  %steal   %idle
              0.00    0.00  100.00    0.00    0.00    0.00

   Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
   sda              0.00  415.80      0.00   8436.30     0.00  1693.40   0.00  80.29    0.00    0.49   0.18     0.00    20.29   0.43  18.08
   sdb              0.00  415.80      0.00   8436.30     0.00  1693.40   0.00  80.29    0.00    0.30   0.10     0.00    20.29   0.24  10.00
   md127            0.00 2108.20      0.00   8432.80     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     4.00   0.00   0.00
   md0              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
   md1              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
   dm-0             0.00 2107.80      0.00   8431.20     0.00     0.00   0.00   0.00    0.00    2.16   4.55     0.00     4.00   0.47 100.00

EDIT #2: Trying larger sector size (4096 instead of 512) for encryption...

Since top shows that an IRQ-handler uses most of the CPU resources, I hope that a larger block size for encryption could reduce it. For testing I reformat the disk using cryptsetup --sector-size=4096 luksFormat /dev/md127. The luksDump command of cryptsetup confirms that it uses a encryption sector size of 4096 now:

   root@ReadyNAS102:~# cryptsetup luksDump /dev/md127

   [...]

   Data segments:
     0: crypt
           offset: 16777216 [bytes]
           length: (whole device)
           cipher: aes-xts-plain64
           sector: 4096 [bytes]

   [...]

I then applied luksOpen and took another measurement:

root@ReadyNAS102:~# cryptsetup luksOpen /dev/md127 md127-decrypt
Enter passphrase for /dev/md127:

root@ReadyNAS102:~# dd if=/dev/zero of=/dev/mapper/md127-decrypt bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 73.9352 s, 14.2 MB/s

It seems that the CPU load for the IRQ dropped a bit:

     PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   10791 root       0 -20       0      0      0 I  31.7   0.0   3:32.92 kworker/u3:1-kcryptd
     245 root     -51   0       0      0      0 S  22.9   0.0   4:38.98 irq/38-d0090000
   29296 root      20   0       0      0      0 R  15.9   0.0   0:04.14 kworker/u2:0+flush-253:0
   29356 root      20   0       0      0      0 R  15.9   0.0   0:04.09 dmcrypt_write
   29538 root      20   0    7064   2280   1252 D   9.2   0.5   0:02.83 dd

Here is what iostat -kx 5 shows:

   Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
   sda              0.00 1030.00      0.00  14018.70     0.00  2474.80   0.00  70.61    0.00    0.56   0.54     0.00    13.61   0.44  45.44
   sdb              0.00 1029.80      0.00  14018.70     0.00  2475.00   0.00  70.62    0.00    0.34   0.32     0.00    13.61   0.28  28.72
   md127            0.00 3504.40      0.00  14017.60     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     4.00   0.00   0.00
   md0              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
   md1              0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.00     0.00   0.00   0.00
   dm-0             0.00 3504.20      0.00  14016.80     0.00     0.00   0.00   0.00    0.00    2.91  10.18     0.00     4.00   0.28  99.44

Seems a bigger sector size helps to increase the data rate! I'll try this for different sector sizes now. However, I think that I can not just increase the sector size too much without getting bad performance for small files. Since I want to use btrfs on top of md127-decrypt, a sector size of 4096 should be no problem. Btrfs uses a 16kB node size per default, which is probably a reasonable upper limit.

EDIT #3: Datarate vs encryption sector size

I wrote an automated test (bash script) , that checks different sector sizes. However, using a sector size of 8192 was not supported by cryptsetup (Message: "cryptsetup: Unsupported encryption sector size."). Hence, I'll use 4096 bytes.

   Sector Size   Speed
     512 bytes    8.7 MB/s
    1024 bytes   11.8 MB/s
    2048 bytes   13.8 MB/s
    4096 bytes   14.4 MB/s
4
  • 1
    Were CPU(s) pegged at 100% while writing? Also, have you checked iostat -kx 5 or similar while its running (to see how much its waiting for I/O)? BTW: I would guess it's not really using the AES hardware... – derobert Jan 4 '20 at 18:38
  • @derobert I added CPU/disk statistics above. Seems CPU is used ~100% and the physical block devices sda/sdb have a load of about 10-20% while dm-0 seems to be the bottleneck. Since most CPU-load is used for irq/38-d0090000, I would guess the hardware accelerator is used. But, it probably is slowed down by a lot of context switches and IRQ handling that it causes such a huge amount of CPU load. What do you think? – SDwarfs Jan 5 '20 at 14:48
  • I think AES CBC is still fine. XTS is better, and faster for encryption, but I don't think there's been a successful attack on AES CBC. – Omnifarious Feb 19 '20 at 21:33
  • Did you check your stripe cache size? cat /sys/block/mdmd127/md/stripe_cache_size – ioMatrix Jul 7 '20 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.