up vote 10 down vote favorite
4

I/O to my software RAID6 often freezes for around 30 seconds after which everything is back to normal.

After the freeze is over this is put into syslog:

Mar 14 18:43:57 server kernel: [35649.816060] sd 5:0:23:0: [sdy] CDB: Read(10): 28 00 6c 52 68 58 00 04 00 00
Mar 14 18:43:58 server kernel: [35651.149020] mptbase: ioc0: LogInfo(0x31140000): Originator={PL}, Code={IO Executed}, SubCode(0x0000) cb_idx mptscsih_io_done
Mar 14 18:43:58 server kernel: [35651.151962] mptscsih: ioc0: task abort: SUCCESS (rv=2002) (sc=ffff8807b02dfe80)
Mar 14 18:43:58 server kernel: [35651.151967] mptscsih: ioc0: attempting task abort! (sc=ffff88002a7f30c0)
Mar 14 18:43:58 server kernel: [35651.151972] sd 5:0:23:0: [sdy] CDB: Read(10): 28 00 6c 52 6c 58 00 04 00 00
Mar 14 18:43:58 server kernel: [35651.151981] mptscsih: ioc0: task abort: SUCCESS (rv=2002) (sc=ffff88002a7f30c0)
Mar 14 18:43:58 server kernel: [35651.151984] mptscsih: ioc0: attempting task abort! (sc=ffff8804120e5ec0)
Mar 14 18:43:58 server kernel: [35651.151988] sd 5:0:23:0: [sdy] CDB: Read(10): 28 00 6c 52 70 58 00 04 00 00
Mar 14 18:43:58 server kernel: [35651.151996] mptscsih: ioc0: task abort: SUCCESS (rv=2002) (sc=ffff8804120e5ec0)
Mar 14 18:43:58 server kernel: [35651.151999] mptscsih: ioc0: attempting task abort! (sc=ffff880154afb280)
Mar 14 18:43:58 server kernel: [35651.152020] sd 5:0:23:0: [sdy] CDB: Read(10): 28 00 6c 52 74 58 00 04 00 00
Mar 14 18:43:58 server kernel: [35651.152029] mptscsih: ioc0: task abort: SUCCESS (rv=2002) (sc=ffff880154afb280)

I have googled the error and someone suggested trying using 1.5Gbps instead of 3.0Gbps. Using lsiutil I changed the link speed:

# lsiutil -p 1 -i 

Firmware Settings
-----------------
SAS WWID:                       500605b002c0f680
Multi-pathing:                  Disabled
SATA Native Command Queuing:    Enabled
SATA Write Caching:             Enabled
SATA Maximum Queue Depth:       32
Device Missing Report Delay:    0 seconds
Device Missing I/O Delay:       0 seconds
Phy Parameters for Phynum:      0    1    2    3    4    5    6    7
  Link Enabled:                 Yes  Yes  Yes  Yes  Yes  Yes  Yes  Yes
  Link Min Rate:                1.5  1.5  1.5  1.5  1.5  1.5  1.5  1.5
  Link Max Rate:                1.5  1.5  1.5  1.5  1.5  1.5  1.5  1.5
  SSP Initiator Enabled:        Yes  Yes  Yes  Yes  Yes  Yes  Yes  Yes
  SSP Target Enabled:           No   No   No   No   No   No   No   No
  Port Configuration:           Auto Auto Auto Auto Auto Auto Auto Auto
Target IDs per enclosure:       1
Persistent mapping:             Enabled
Physical mapping type:          None
Target ID 0 reserved for boot:  No
Starting slot (direct attach):  0
Target IDs (physical mapping):  8
Interrupt Coalescing:           Enabled, timeout is 16 us, depth is 4

That did not help.

I tried changing 'Device Missing I/O Delay' to 32. That did not help either.

I tried changing /sys/class/scsi_device/*/device/timeout from 30 to 100 and then to 3. All failed.

$ uname -a
Linux server 3.2.0-0.bpo.1-amd64 #1 SMP Sat Feb 11 08:41:32 UTC 2012 x86_64 GNU/Linux
$ grep LSISAS1068E /var/log/messages
Mar 13 15:47:44 server kernel: [   21.082363] scsi5 : ioc0: LSISAS1068E B3, FwRev=01210000h, Ports=1, MaxQ=483, IRQ=45
$ modinfo mptscsih
filename:       /lib/modules/3.2.0-0.bpo.1-amd64/kernel/drivers/message/fusion/mptscsih.ko
version:        3.04.20
license:        GPL
description:    Fusion MPT SCSI Host driver
author:         LSI Corporation
srcversion:     85D42A00FEBA3C95555E3AF
depends:        scsi_mod,mptbase
intree:         Y
vermagic:       3.2.0-0.bpo.1-amd64 SMP mod_unload modversions 
$ cat /sys/block/sdae/device/model
ST3000DM001-9YN1
$ cat /sys/block/sdae/device/rev
CC4C

The problem happens extremely rarely if there are only read or write operations: I can read or write 1 TB with no problem. The problem seems to arise when there are both read and write operations. On a raid6 that happens if you write a file smaller than stripe size and you do not have the stripe cached already (in which case the stripe must be read to compute new checksum).

The system is not a virtual machine.

What is causing the problem? How do I get rid of the 30 seconds of freezing?

Edit: additional testing

I have found a nice test set that seems to provoke the problem. It contains files that are smaller than the stripe size thus forcing recomputation of parity thus forcing a lot of reads combined with the writes.

I must admit that I did not think that the queue scheduler would have any effect on this problem. I was wrong. It is clear that deadlineis much worse than the others. None of them solve the problem, though.

# cat /sys/block/sdaa/queue/scheduler
noop deadline [cfq]

Changing scheduler to noop causes the problem to arise after 100-120 secs.

parallel echo noop \> {} ::: /sys/block/sd*/queue/scheduler

Changing scheduler to deadline causes the problem to arise after 20-30 secs.

parallel echo deadline \> {} ::: /sys/block/sd*/queue/scheduler

Changing scheduler to cfq causes the problem to arise after 120-300 secs.

parallel echo cfq \> {} ::: /sys/block/sd*/queue/scheduler

Edit2

Since the scheduler has an effect I am thinking if the problem is caused by too many requests in a timeframe. Can I somehow throttle the number of requests sent per second?

share|improve this question
up vote 5 down vote

The MPTSCSIH-Driver Release Notes from LSI look interesting.

Major Changes For Version 2.06.75.00-1
Release Date:  12/10/2007

General Changes
Functionality
•   Task Aborts for commands to a Volume are returned as FAILED and not sent to FW.

Which version is your driver? (modinfo mptscsih)

Use this link for Seagate Firmware information about your Barracuda 3 TB drive. You have to enter the serial number to get details.

Update: Try smartctl -i /dev/sdaa on I just tested it on SCSI and SATA and got the serial number that way.

share|improve this answer
    
What parts of the driver release notes do you find relevant for this problem? How do I find the serial number using GNU/Linux on disks that are in production? And what would you expect to find from Seagate on this? Version of mptscsih is updated in the question. – Ole Tange Mar 25 '12 at 16:37
    
@OleTange I inserted the "interesting" section. Although your driver seems to be newer than that it might be an old problem reappearing here. As for the serial-number... Seagate only offers Windows-tools. On linux I would try an inq command - perhaps from some EMC-drivers (should be freely downloadable) - but this is just a guess. – Nils Mar 25 '12 at 19:22
2  
@OleTange RE:"How do I find the serial number using GNU/Linux on disks that are in production?" run dmidecode this will pull the description of hardware components from memory. Often on consumer level items you won't have entries for hard drives SN's, but, with enterprise gear it will typically have this added or the drives will have more intelligence. There are special --type codes to refer to MFR devices should they have made them available. Companies who supply arrays usually provide this information so that recalled drives can be located. – 2bc Mar 25 '12 at 19:55
    
@LinuxlyChallenged dmidecode sees no drives - neither internal nor external. I was unable to find inq for Debian. – Ole Tange Mar 26 '12 at 23:26
    
@OleTange use smartctl see my updated answer... – Nils May 10 '12 at 20:56
up vote 2 down vote

Have you tried changing your I/O schedulers?

   mccoy:/sys/block/sdb/queue # cat scheduler 
   noop anticipatory deadline [cfq] 
   mccoy:/sys/block/sdb/queue # echo noop > scheduler 
   mccoy:/sys/block/sdb/queue # cat scheduler 
   [noop] anticipatory deadline cfq

The default is CFQ typically for most systems "currently."

To compare I/O schedulers do the following:

Read testing:

# echo 3 > /proc/sys/vm/drop_caches

This will make sure you are testing the disk and not cached pages of RAM, this will flush the cache.

Write testing:

Copy your files multiple times simultaneously. Once the writes are complete issue a sync

If you are testing both you may want to drop_caches and call sync when the copy is done. In addition to the scheduler there are tunables for each scheduler. But, a quick test would be to change the scheduler and try again. If you have a good controller noop will offload the "I/O Scheduling' to it and not perform any OS level data scheduling.

Anyway, it is worth a try and it only takes an echo to set it back.

share|improve this answer
    
See updated question for the results. – Ole Tange Mar 24 '12 at 14:56
up vote 2 down vote accepted

I have solved the problem by buying a SAS2008 card. It still complains a little in the log, but it never blocks the disk I/O. Also I have tested it supports 4 TB SATA drives, whereas the LSI-SAS1068E only supports 2 TB.

As I will be returning the LSI-SAS1068E to the seller, I will not be able to try out other suggestions. Therefore I close the question here.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.