Slow Write Speed(<2 MB/s) on Unencrypted and LUKS Encrypted ext4 filesystems using mdadm Software RAID1 Debian 8 Dell PowerEdge T330 Server

Question

We have installed Debian 8 on a new Dell PowerEdge T330, there are two partitions, / and /var, in a RAID1 array using mdadm. During testing the primary applications: mysql and tomcat were stopped. We are getting abysmal write performance from both partitions although the read performance is adequate. This is the observations from one of two identical servers setup the same way. Any help would be appreciated.

Performance

root@bcmdit-519:/home/bcmdit# FILE=/tmp/test_data && dd bs=16k \ count=102400 oflag=direct if=/dev/zero of=$FILE && \ rm $FILE && FILE=/var/tmp/test_data && dd bs=16k \ count=102400 oflag=direct if=/dev/zero of=$FILE && rm $FILE

102400+0 records in
102400+0 records out
1677721600 bytes (1.7 GB) copied, 886.418 s, 1.9 MB/s

102400+0 records in
102400+0 records out
1677721600 bytes (1.7 GB) copied, 894.832 s, 1.9 MB/s

root@bcmdit-519:/home/bcmdit# hdparm -t /dev/sda ; hdparm -t /dev/sdb ; hdparm -t /dev/md0 ; hdparm -t /dev/md1

/dev/sda:

    Timing buffered disk reads: 394 MB in  3.00 seconds = 131.15 MB/sec

/dev/sdb:

    Timing buffered disk reads: 394 MB in  3.01 seconds = 131.05 MB/sec

/dev/md0:

    Timing buffered disk reads: 398 MB in  3.00 seconds = 132.45 MB/sec

/dev/md1:

    Timing buffered disk reads: 318 MB in  3.00 seconds = 106.00 MB/sec

References

https://severfault.com/questions/832117/how-increase-write-speed-of-raid1-mdadm https://wiki.mikejung.biz/Software_RAID Write access time slow on RAID1 https://bbs.archlinux.org/viewtopic.php?id=173791 et al...

Configuration

Encryption was setup using:

root@bcmdit-519:/home/bcmdit# cryptsetup luksDump UUID=1e7b64ac-f187-4fac-9712-8e0dacadfca7|grep -E 'Cipher|Hash'

Cipher name:    aes
Cipher mode:    xts-plain64
Hash spec:      sha1

Config snippets

root@bcmdit-519:/home/bcmdit# facter virtual productname lsbdistid \
                     lsbdistrelease processor0 blockdevice_sda_model \  
                     blockdevice_sdb_model bios_version && uname -a && uptime
----------

    bios_version => 2.4.3
    blockdevice_sda_model => ST1000NX0423
    blockdevice_sdb_model => ST1000NX0423
    lsbdistid => Debian
    lsbdistrelease => 8.10
    processor0 => Intel(R) Xeon(R) CPU E3-1230 v6 @ 3.50GHz
    productname => PowerEdge T330
    virtual => physical
    Linux bcmdit-519 3.16.0-4-amd64 #1 SMP Debian 3.16.51-3 (2017-12-13) x86_64 GNU/Linux
     14:45:58 up  2:49,  2 users,  load average: 0.06, 0.17, 0.44

 root@bcmdit-519:/home/bcmdit# grep GRUB_CMDLINE_LINUX_DEFAULT /etc/default/grub

    GRUB_CMDLINE_LINUX_DEFAULT="quiet erst_disable elevator=deadline"

root@bcmdit-519:/home/bcmdit# free -m         

             total       used       free     shared    buffers     cached
Mem:         32202       1532      30670          9         17        369
-/+ buffers/cache:       1145      31056
Swap:            0          0          0

root@bcmdit-519:/home/bcmdit# parted /dev/sda print

    Model: ATA ST1000NX0423 (scsi)
    Disk /dev/sda: 1000GB
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos
    Disk Flags: 

    Number  Start   End     Size   Type      File system  Flags
     1      1049kB  500GB   500GB  primary                boot, raid
     2      500GB   1000GB  500GB  extended
     5      500GB   1000GB  500GB  logical                raid

root@bcmdit-519:/home/bcmdit# parted /dev/sdb print

    Model: ATA ST1000NX0423 (scsi)
    Disk /dev/sdb: 1000GB
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos
    Disk Flags: 

    Number  Start   End     Size   Type      File system  Flags
     1      1049kB  500GB   500GB  primary                raid
     2      500GB   1000GB  500GB  extended
     5      500GB   1000GB  500GB  logical                raid

----------

root@bcmdit-519:/home/bcmdit# cat /proc/mdstat

    Personalities : [raid1] 
    md1 : active raid1 sda5[0] sdb5[1]
          488249344 blocks super 1.2 [2/2] [UU]
          bitmap: 3/4 pages [12KB], 65536KB chunk

    md0 : active raid1 sda1[0] sdb1[1]
          488248320 blocks super 1.2 [2/2] [UU]
          bitmap: 2/4 pages [8KB], 65536KB chunk

    unused devices: <none>

root@bcmdit-519:/home/bcmdit# mdadm --query --detail /dev/md0

    /dev/md0:
            Version : 1.2
      Creation Time : Mon Apr 16 13:46:51 2018
         Raid Level : raid1
         Array Size : 488248320 (465.63 GiB 499.97 GB)
      Used Dev Size : 488248320 (465.63 GiB 499.97 GB)
       Raid Devices : 2
      Total Devices : 2
        Persistence : Superblock is persistent

      Intent Bitmap : Internal

        Update Time : Tue May 15 14:26:47 2018
              State : clean 
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0

               Name : bcmdit-519:0  (local to host bcmdit-519)
               UUID : afd3968c:2e8b191d:4504f21e:255b6470
             Events : 1703

        Number   Major   Minor   RaidDevice State
           0       8        1        0      active sync   /dev/sda1
           1       8       17        1      active sync   /dev/sdb1

 root@bcmdit-519:/home/bcmdit# mdadm --query --detail /dev/md1

    /dev/md1: 

            Version : 1.2
      Creation Time : Mon Apr 16 13:47:06 2018
         Raid Level : raid1
         Array Size : 488249344 (465.63 GiB 499.97 GB)
      Used Dev Size : 488249344 (465.63 GiB 499.97 GB)
       Raid Devices : 2
      Total Devices : 2
        Persistence : Superblock is persistent

      Intent Bitmap : Internal

        Update Time : Tue May 15 14:15:11 2018
              State : active 
     Active Devices : 2
    Working Devices : 2
     Failed Devices : 0
      Spare Devices : 0

               Name : bcmdit-519:1  (local to host bcmdit-519)
               UUID : e46f968a:e8fff775:ecee9cfb:4ad88574
             Events : 2659

        Number   Major   Minor   RaidDevice State
           0       8        5        0      active sync   /dev/sda5
           1       8       21        1      active sync   /dev/sdb5

root@bcmdit-519:/home/bcmdit# cat /etc/crypttab

    crypt1 UUID=1e7b64ac-f187-4fac-9712-8e0dacadfca7 /root/.crypt1 luks

root@bcmdit-519:/home/bcmdit# grep -v '^#' /etc/fstab

    UUID=c6baa173-8ea6-4598-a965-eee728a93d69 /               ext4    defaults,errors=remount-ro 0       1
    /dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0
    /dev/mapper/crypt1 /var ext4 defaults,errors=remount-ro 0       2
    /var/swapfile1 none swap sw,nofail 0       0

root@bcmdit-519:/home/bcmdit# smartctl -a /dev/sda|head -n 20

    smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.16.0-4-amd64] (local build)
    Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

    === START OF INFORMATION SECTION ===
    Device Model:     ST1000NX0423
    Serial Number:    W4713QXE
    LU WWN Device Id: 5 000c50 0abb06247
    Add. Product Id:  DELL(tm)
    Firmware Version: NA07
    User Capacity:    1,000,204,886,016 bytes [1.00 TB]
    Sector Size:      512 bytes logical/physical
    Rotation Rate:    7200 rpm
    Form Factor:      2.5 inches
    Device is:        Not in smartctl database [for details use: -P showall]
    ATA Version is:   ACS-3 (minor revision not indicated)
    SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
    Local Time is:    Tue May 15 14:29:03 2018 PDT
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled

root@bcmdit-519:/home/bcmdit# smartctl -a /dev/sdb|head -n 20

    smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.16.0-4-amd64] (local build)
    Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

    === START OF INFORMATION SECTION ===
    Device Model:     ST1000NX0423
    Serial Number:    W4714VDQ
    LU WWN Device Id: 5 000c50 0abf99927
    Add. Product Id:  DELL(tm)
    Firmware Version: NA07
    User Capacity:    1,000,204,886,016 bytes [1.00 TB]
    Sector Size:      512 bytes logical/physical
    Rotation Rate:    7200 rpm
    Form Factor:      2.5 inches
    Device is:        Not in smartctl database [for details use: -P showall]
    ATA Version is:   ACS-3 (minor revision not indicated)
    SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
    Local Time is:    Tue May 15 14:29:11 2018 PDT
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled

Update 1

With 16M

root@bcmdit-519:/tmp# FILE=/tmp/test_data \
&& dd bs=16M count=102 oflag=direct if=/dev/zero of=$FILE \
&& rm $FILE \
&& FILE=/var/tmp/test_data \
&& dd bs=16M count=102 oflag=direct if=/dev/zero of=$FILE \
&& rm $FILE
102+0 records in
102+0 records out
1711276032 bytes (1.7 GB) copied, 16.6394 s, 103 MB/s
102+0 records in
102+0 records out
1711276032 bytes (1.7 GB) copied, 17.8649 s, 95.8 MB/s

Update 2 Seagate drive serial number found SMART indicates an Enterprise grade drive: https://www.cnet.com/products/seagate-enterprise-capacity-2-5-hdd-v-3-1tb-sata-512n/specs/

Update 3 I found that drive write cacheing was off, but by setting it to on:

hdparm -W1 /dev/sd*

I get much better results with bs=16k now

root@bcmdit-519:/home/bcmdit# FILE=/tmp/test_data && dd bs=16k count=102400 oflag=direct if=/dev/zero of=$FILE && rm $FILE
102400+0 records in         
102400+0 records out
1677721600 bytes (1.7 GB) copied, 14.0708 s, 119 MB/s

Update 4

root@ecm-oscar-519:/home/bcmdit# cryptsetup benchmark

# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1      1394382 iterations per second
PBKDF2-sha256     923042 iterations per second
PBKDF2-sha512     728177 iterations per second
PBKDF2-ripemd160  804122 iterations per second
PBKDF2-whirlpool  313569 iterations per second
#  Algorithm | Key |  Encryption |  Decryption
     aes-cbc   128b  1149.9 MiB/s  3655.8 MiB/s
 serpent-cbc   128b    99.6 MiB/s   743.4 MiB/s
 twofish-cbc   128b   219.0 MiB/s   400.0 MiB/s
     aes-cbc   256b   867.5 MiB/s  2904.5 MiB/s
 serpent-cbc   256b    99.6 MiB/s   742.6 MiB/s
 twofish-cbc   256b   218.9 MiB/s   399.8 MiB/s
     aes-xts   256b  3615.1 MiB/s  3617.3 MiB/s
 serpent-xts   256b   710.8 MiB/s   705.0 MiB/s
 twofish-xts   256b   388.1 MiB/s   394.5 MiB/s
     aes-xts   512b  2884.9 MiB/s  2888.1 MiB/s
 serpent-xts   512b   710.7 MiB/s   704.7 MiB/s
 twofish-xts   512b   388.0 MiB/s   394.3 MiB/s

Please could you format the blocks of output properly. Select the blocks and use the {} to turn it into fixed-width output (like you'd see on a terminal). It's so much easier to read and digest. — roaima, May 15, 2018 at 22:37
is mdadm balancing? Show it's setup, status, logs, dmesg, SMART. Also try bs=16m instead of bs=16k... I'd use luks/btrfs or lvm/luks/ext4 .. — user1133275, May 15, 2018 at 23:20
@user1133275 what is needed from dmesg? mdadm --query --detail output has been provided, what else is needed? — Thom L., May 16, 2018 at 2:47
2MB/sec using 16K write operations is 125 IO operations/sec. That's actually a pretty good number for a consumer-grade SATA disk, which is what I assume you're running. 7,200 RPM SATA disks usually can reach a max of about 70 IO operations/sec, but the fact that you're doing sequential 16K writes probably means the disk is able to coalesce a lot of the individual write operations you send to it. Try doing single-block-sized writes to random locations so the disk can't coalesce data and the IO rate could drop to 30kb/sec or less. — Andrew Henle, May 16, 2018 at 10:41

user1133275 · Accepted Answer · 2018-05-17 16:53:06Z

0

When you ask dd for bs=16K and oflag=direct you are asking for many small writes, this is what HDDs are bad at and what SSDs are good at.

You can use LVMCache to get the benefit of both (up to the SSD size)

If you use bs=16M or no oflag then the writes are split/combined/cached in RAM and written at an optimal size.

Why is dd using direct slower writing to disk than to a file

for example;

> dd if=/dev/zero of=test.bin bs=16k count=1000 oflag=direct
1000+0 records in
1000+0 records out
16384000 bytes (16 MB, 16 MiB) copied, 3.19453 s, 5.1 MB/s

> dd if=/dev/zero of=test.bin bs=16M count=1 oflag=direct
1+0 records in
1+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.291366 s, 57.6 MB/s

> dd if=/dev/zero of=test.bin bs=16k count=1000
1000+0 records in
1000+0 records out
16384000 bytes (16 MB, 16 MiB) copied, 0.0815558 s, 201 MB/s

> uname -r
4.14.41-130

edited May 17, 2018 at 16:53

answered May 16, 2018 at 8:26

user1133275

5,4941 gold badge19 silver badges37 bronze badges

You should test your theories before posting, as everything you wrote has no basis in reality. Block size makes a difference, but nowhere near what the OP is seeing. Here are two examples from a very ancient system, using a consumer grade 500GB 7200 RPM SATA drive:
– ajeh
May 17, 2018 at 0:49
$ dd if=/dev/zero of=test.bin bs=16k count=100000 oflag=direct 100000+0 records in 100000+0 records out 1638400000 bytes (1.6 GB, 1.5 GiB) copied, 50.4272 s, 32.5 MB/s
– ajeh
May 17, 2018 at 0:49
$ dd if=/dev/zero of=test.bin bs=16M count=100 oflag=direct 100+0 records in 100+0 records out 1677721600 bytes (1.7 GB, 1.6 GiB) copied, 17.9688 s, 93.4 MB/s
– ajeh
May 17, 2018 at 0:49
1

@ajeh disk write cacheing was off, so the performance was abysmal. Perhaps because it was in software raid1 using mdadm?
– Thom L.
May 18, 2018 at 19:17
1

@Michael the 5/60/200 MB/s example is on ~2018 10TB Seagate (SMR likely)
– user1133275
May 12, 2020 at 22:26

| Show 6 more comments

Vlastimil Burián · Accepted Answer · 2018-05-16 10:34:57Z

0

Using pv (man page) utility like so:

pv --average-rate < /dev/urandom > /mdX-MountPoint/SomeFileName

may prove to be more effective method of measuring the speed than your dd (man page).

I edited the input file to be random data, which is better than static zeroes.

The problem with dd (man page) is that you must adjust the block size.

While with pv (man page) that is not the case as it establishes maximum speed itself for the best throughput.

edited May 16, 2018 at 10:34

answered May 16, 2018 at 9:40

Vlastimil Burián

27.8k56 gold badges181 silver badges309 bronze badges

What does /dev/urandom have to do with HDD write speed?
– ajeh
May 17, 2018 at 0:42
@ajeh if the file system has compression enabled it will make a huge difference
– user1133275
May 12, 2020 at 22:21

Add a comment |

Stack Exchange Network

Slow Write Speed(<2 MB/s) on Unencrypted and LUKS Encrypted ext4 filesystems using mdadm Software RAID1 Debian 8 Dell PowerEdge T330 Server

2 Answers 2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
debian
ext4
io
luks
mdadm
.

Linked

Hot Network Questions

Slow Write Speed(<2 MB/s) on Unencrypted and LUKS Encrypted ext4 filesystems using mdadm Software RAID1 Debian 8 Dell PowerEdge T330 Server

2 Answers 2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged debianext4ioluksmdadm.

Linked

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
debian
ext4
io
luks
mdadm
.