Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I've run out of ideas for tuning disk write speeds on a handful of CentOS VMs running on a lab XenServer 6.2 host. Running under the free license (so vendor support not a concern), the lab host is a glorified desktop with 3 x WD3200AAKS disks I'm using for a bit of fun.

Firstly, I installed XenServer on a USB stick and configured a software RAID5 array of the three disks to act as a storage repository. XenServer puts an LVM on this, which it then carves up into the virtual disks for the VMs. CentOS 6.4 is installed from the minimal CD, with either an ext3 or ext4 filesystem. i.e.:

/dev/sd[bcd] -> /dev/sd[bcd]1 -> /dev/md0 -> lvm -> vol1 -> xvda -> xvda1 -> ext3

The write speed in a freshly-installed VM was roughly 9.5MB/s (averaged across various dd if=/dev/zero of=~/spool.tmp bs=<X> and dd if=/dev/zero bs=<X> | pv > ~/spool.tmp runs).

So I started from the top, looking at direct writes to the RAID device (dd if=/dev/zero of=/dev/md0), tweaking the stripe_cache_size of md0 (along with blockdev --setra for the md0 and underlying sd[bcd], and max_sectors_kb for good measure), until I got from approx 12MB/s to a more-reasonable 59MB/s.

I recreated the storage repo and carved out a new VM, did the miminal install of CentOS 6.4 and tried my test again. I'm still only getting 9-12MB/s on average (peaking at about 19MB/s for a few seconds).

With higher readahead settings, I can achieve an easy 200MB/s read (dd if=~/spool.tmp bs=4K | pv > /dev/null), and my baseline /dev/zero read speed is approx 240MB/s, but I'm stuck with this lousy write speed and I don't know what else to try.

What else can I look at?


Some detail below, of anything I thought might be pertinent, for reference:

[root@host ~]# parted
Model: ATA WDC WD3200AAKS-7 (scsi)
Disk /dev/sdb: 320072933375B
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start          End            Size           File system  Name  Flags
 1      1048576B       309237645823B  309236597248B               sdb1       
 2      309237645824B  310311388159B  1073742336B                 sdb2       

Model: ATA WDC WD3200AAKS-7 (scsi)
Disk /dev/sdb: 320072933375B
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start          End            Size           File system  Name  Flags
 1      1048576B       309237645823B  309236597248B               sdb1       
 2      309237645824B  310311388159B  1073742336B                 sdb2       

Model: ATA WDC WD3200AAKS-7 (scsi)
Disk /dev/sdd: 320072933375B
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start          End            Size           File system  Name  Flags
 1      1048576B       309237645823B  309236597248B               sdd1       
 2      309237645824B  310311388159B  1073742336B    ext3         sdd2       

[root@host ~]# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sdb1[0] sdd1[2] sdc1[1]
      603977600 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      bitmap: 3/144 pages [12KB], 1024KB chunk

unused devices: <none>

[root@host ~]# cat /sys/block/md0/md/stripe_cache_size
4096
[root@host ~]# cat /sys/block/sd[bcd]/queue/max_sectors_kb
16384
16384
16384
[root@host ~]# blockdev --getra /dev/sd[bcd] /dev/md0
4096
4096
4096
512    

[root@host~]# vgdisplay VG_XenStorage-c9e965db-ed0f-3fdb-7d40-abbf2bda20e4
  --- Volume group ---
  VG Name               VG_XenStorage-c9e965db-ed0f-3fdb-7d40-abbf2bda20e4
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  24
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                6
  Open LV               4
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               575.98 GB
  PE Size               4.00 MB
  Total PE              147452
  Alloc PE / Size       25663 / 100.25 GB
  Free  PE / Size       121789 / 475.74 GB
  VG UUID               jjdM19-L505-a0UW-tT4Z-ThbL-LK0A-Ml0bKr
[root@host~]# lvdisplay /dev/VG_XenStorage-c9e965db-ed0f-3fdb-7d40-abbf2bda20e4/VHD-ee6ae3d2-18ab-4b14-902c-4019b832ea59 
  --- Logical volume ---
  LV Name                /dev/VG_XenStorage-c9e965db-ed0f-3fdb-7d40-abbf2bda20e4/VHD-ee6ae3d2-18ab-4b14-902c-4019b832ea59
  VG Name                VG_XenStorage-c9e965db-ed0f-3fdb-7d40-abbf2bda20e4
  LV UUID                euBFtP-e8Hf-EzoP-AVe1-4jZO-mb3x-SaiW2i
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                16.04 GB
  Current LE             4106
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     1024
  Block device           252:2

[root@vm ~]# parted
Model: Xen Virtual Block Device (xvd)
Disk /dev/xvdc: 17179869184B
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start     End           Size          Type     File system  Flags
 1      1048576B  17179869183B  17178820608B  primary  ext3

[root@vm ~]# blockdev --getra /dev/xvdc
4096
[root@centos02 xvdc1]# tune2fs -l /dev/xvdc1 | grep Block
Block count:              4194048
Block size:               4096
Blocks per group:         32768
[root@centos02 xvdc1]# tune2fs -l /dev/xvdc1 | grep RAID
RAID stride:              16
RAID stripe width:        32
share|improve this question

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.