Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 2 down vote favorite
1

I don't know why, but I have big trouble with speed of my raid.

I have 4x WD RE4 7200RPM 500GB. Size of RAID 5 is 1000GB. 1 of HDDs is Spare.

Problem is here:

Every 1,0s: cat /proc/mdstat                                                                                                                    Mon Dec 31 10:01:11 2012

Personalities : [raid1] [raid6] [raid5] [raid4]
md1 : active raid5 sdc2[0] sdb2[3](S) sdd2[2] sde2[1]
      974812160 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
      [=>...................]  check =  9.0% (44241152/487406080) finish=6696.5min speed=1102K/sec

md0 : active raid1 sdc1[0] sdb1[3](S) sdd1[2] sde1[1]
      975860 blocks super 1.2 [3/3] [UUU]

unused devices: <none>

I started this checking at 2:00 AM and I thought it could be already done. It isn't. When it was new (2 months ago) it tooks only about 150 - 300minutes to check raid. 

vmstat 1 shows:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  1      0 168108 278152 6649192    0    0   385   705    0    1 47 13 33  7
 1  1      0 168092 278152 6649236    0    0     0  1564 24426 42090 28 11 46 16
 3  1      0 173424 278152 6649236    0    0     0  1204 23750 41592 30  7 48 15
 1  2      0 173416 278160 6649228    0    0    24   592 23131 41252 25  5 47 23
 2  1      0 173424 278160 6649260    0    0     0  2340 24750 42888 29  8 45 18
 1  1      0 172928 278176 6649244    0    0     0  1408 23818 41362 30  8 42 21
 1  0      0 172696 278176 6649304    0    0     0   471 23144 40932 25  7 58 10
 1  0      0 172488 278176 6649304    0    0     0   275 26299 45241 27 17 52  5
 1  2      0 172612 278184 6649304    0    0     0  1806 24572 41288 40  6 44  9
 5  2      0 172752 278200 6649328    0    0     0   780 23541 41308 28  6 33 33

Iostat 1 shows:

Linux 2.6.32-5-amd64 ()       31.12.2012      _x86_64_        (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          47,02    0,34   12,94    6,95    0,00   32,74

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               7,68        17,48      3428,36  112980468 22156867512
sdb               0,00         0,00         0,00        748       2208
sdc              70,20      3506,35      1574,01 22660920204 10172547974
sdd              70,32      3528,74      1551,86 22805657128 10029430470
sde              71,11      3548,29      1538,53 22931965117 9943244782
md0               0,00         0,01         0,00      55936       5416
md1             356,51      3276,98      2594,09 21178557866 16765170392

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          24,88    0,00    6,47   18,41    0,00   50,25

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               0,00         0,00         0,00          0          0
sdb               0,00         0,00         0,00          0          0
sdc             164,00        80,00      2792,00         80       2792
sdd             116,00      1072,00      1248,00       1072       1248
sde             138,00         0,00      1864,00          0       1864
md0               0,00         0,00         0,00          0          0
md1             369,00         0,00      2952,00          0       2952

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          30,71    0,00    6,88   14,50    0,00   47,91

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               0,00         0,00         0,00          0          0
sdb               0,00         0,00         0,00          0          0
sdc             187,00      1040,00      1944,00       1040       1944
sdd             286,00        64,00      4616,00         64       4616
sde             231,00      1024,00      3056,00       1024       3056
md0               0,00         0,00         0,00          0          0
md1             601,00         0,00      4808,00          0       4808

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          22,03    0,00    6,68    3,71    0,00   67,57

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               0,00         0,00         0,00          0          0
sdb               0,00         0,00         0,00          0          0
sdc              30,00         8,00       716,00          8        716
sdd              10,00        40,00        44,00         40         44
sde              33,00         0,00       740,00          0        740
md0               0,00         0,00         0,00          0          0
md1              92,00         0,00       736,00          0        736

Let me give you following questions:

1) Is there problem in full capacity of my raid? (total size 916GB, used 505G, free 365G). 2) Is there problem with " 512k chunk" ?

3) Is EXT3 optimal for SW RAID 5?

4) Is there any possibility, how to increase speed of my SW raid 5?

5) Is it possible to add next disk to my RAID to be not spare, to increase speed of raid only?

6) How much CPU TIME consumes SW Raid 5? in top it shows me only: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 422 root 20 0 0 0 0 S 1 0.0 1743:38 md1_raid5 Server uptime is 74 days (since changing them for the old one).

TOP shows:

top - 10:18:43 up 74 days, 19:21,  3 users,  load average: 2.33, 2.86, 2.94
Tasks: 147 total,   2 running, 145 sleeping,   0 stopped,   0 zombie
Cpu(s): 22.6%us,  6.9%sy,  0.0%ni, 52.9%id, 17.1%wa,  0.1%hi,  0.4%si,  0.0%st
Mem:  24743684k total, 24598984k used,   144700k free,   270604k buffers
Swap:        0k total,        0k used,        0k free,  6664872k cached

I think, there is too high avarage load. But why?I don't see anything consuming too much CPU.

Power TOP Shows following:

Wakeups-from-idle per second : 6122,7   interval: 10,0s
Top causes for wakeups:
  74,0% (17652,4)               kvm : sys_timer_settime (posix_timer_fn)
  15,0% (3579,6)      <kernel IPI> : Rescheduling interrupts
   5,5% (1319,6)               kvm : apic_reg_write (kvm_timer_fn)
   1,8% (422,4)       <interrupt> : ahci
   1,0% (248,2)          events/0 : flush_to_ldisc (delayed_work_timer_fn)
   0,7% (178,6)       worldserver : __mod_timer (process_timeout)
   0,6% (153,4)       <interrupt> : eth0
   0,5% (118,4)       <interrupt> : pata_atiixp
   0,2% ( 43,6)               kvm : __kvm_migrate_timers (kvm_timer_fn)
   0,1% ( 20,0)         md1_raid5 : __mod_timer (blk_unplug_timeout)
   0,0% ( 11,2)        authserver : __mod_timer (process_timeout)
share|improve this question

2 Answers

up vote 3 down vote accepted

Some additional answers to your other Questions:

1) Is there problem in full capacity of my raid? (total size 916GB, used 505G, free 365G).

No, that's correct.

First, a 500 GB HDD isn't really 500 GB big, as HDD manufacturers decided that 1 GB is 1,000,000,000 bytes and not 1099511627776 (1024 * 1024 * 1024 * 1024) bytes.

So you have a raw capacity of ~931 GB. As your file system needs to save some meta information about structure etc., it needs some space for itself, which leads to ~916 GB usable space on a ext3 formatted partition.

As every file takes at least one block (mostly 512 bytes, but could also be 4k bytes or something else), a file with only 10 bytes also uses 512 bytes on disk. If you have lots of small files, you'll have a big difference between the size of all files and the occupied space on disk.

2) Is there problem with " 512k chunk" ?

No. But this also depends on workload and usage. You can find several performance comparisons of different chunk sizes on the internet.

3) Is EXT3 optimal for SW RAID 5?

Yes, as well as any other standard filesystem is. From my perspective, the better question would be "Is RAID5 and/or EXT3 good for my use/workload?"

For example: If you just have a fileserver with a few users, RAID5 and ext3 is fine. If you have a large database on ext3 with RAID5, it would be better to have a RAID10 and XFS.

5) Is it possible to add next disk to my RAID to be not spare, to increase speed of raid only?

Sure, you can have a RAID5 of 4 disks. This will probably increase read performance, but not write performance.

If you need more write performance, you need to get RAID10.

This is true only for sequential read/writes as mdraid on Linux is very bad at providing hight IOPS. So if you need it for a high random I/O load like databases or virtualization, you should get a hardware based RAID instead or use ZFS (which is not available in the standard linux kernel).

share|improve this answer
1  
Adding more disks increases both read and write throughput, at least for sequential IO. Raid10 will give better performance only for random IO ( and be worse for sequential ), and costs more space. There is nothing about mdadm that inherently limits IOPS. A true hardware raid will only help if you are saturating the PCI bus, since less has to go across it, and that isn't likely to happen with just 4 or 5 disks. – psusi Dec 31 '12 at 16:12
1  
After ~50 different load and performance tests on ~50 different servers, distributions, harddisks within the last years, I can truly say, mdraid raids are a performance disaster for setups which need IOPS. It just seems to be a problem with mdraid because a software raid on ZFS performs pretty good. – mgabriel Dec 31 '12 at 19:42
So you say, I should in the future rather think about ZFS? Are there any problems with ZFS under Debian linux? Does Wheezy support ZFS? If I want it for MySQL, and some KVM files, could be that better for me, when using RAID 5? – MIrra Jan 6 at 18:56
up vote 6 down vote

RAID resyncing/checking is done with a lower I/O priority than normal I/O. If there's a lot of I/O on that disk, it will run at the minimum speed which you can modify via /sys/block/md1/md/sync_speed_min

share|improve this answer
1  
cat /sys/block/md1/md/sync_speed_min shows me "1000 (system)" It is in Bytes or Kbytes? – MIrra Dec 31 '12 at 10:32
2  
Kbytes, and it matches your current reported sync speed in /proc/mdstat. You could raise that lower limit by writing the new value to the file. But note that it will affect the performance of your system. – Stephane Chazelas Dec 31 '12 at 12:57

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.