Reduce bad block retry / wait times in Ubuntu

Question

How can I reduce the IO wait time and retry times so that the OS doesn't continually try to write to a failing drive?

I have an system that I use to make copies of demo content that gets loaned out to customers on to regular SATA desktop hard drives. We connect many drives at once via SAS and copy content to them using a script.

Because the drives are loaned out, occasionally some come back damaged but I don't know that they are damaged, so the next time that drive gets reused in a copy operation, it slows down other drives as the system retries IO to that drive. Sometimes it can take hours before I notice the bad drive and remove it. After the drive is removed, the rest of the drives begin writing at normal speed.

I do not care about recovering the bad drives. I just need to weed them out so they don't slow everything else down.

I am also researching badblocks and smartmontools and considering writing a pre-check on the drives before I start writing.

OS: Ubuntu Linux (12.04 lts)

Answer 1

I've not used this tunable before but you probably want to adjust the eh_timeout (error handling timeout) for the drive in question:

[root@localhost device]# cat /sys/block/sda/device/eh_timeout
10
[root@localhost device]# 

The above shows sda set to 10 seconds. From Red Hat Knowledgebase:

In certain storage configurations (for example, configurations with many LUNs), the SCSI error handling code can spend a large amount of time issuing commands such as TEST UNIT READY to unresponsive storage devices. A new sysfs parameter, eh_timeout, has been added to the SCSI device object, which allows configuration of the timeout value for TEST UNIT READY and REQUEST SENSE commands used by the SCSI error handling code. This decreases the amount of time spent checking these unresponsive devices. The default value of eh_timeout is 10 seconds, which was the timeout value used prior to adding this functionality.

Answer 2

Monitor /sys/block/<dev>/stat for the devices you're interested in and compare the 10th parameter (io_ticks).

eg, ticks = io_ticks - prev_ticks / seconds_deltatime / 10

This is the percentage of available time that the disk has spent waiting for disk io.

Close to 100% would be worth checking of course, or else get really clever and compare it to the average of all your disks and pick on any disk(s) above the mean.

See the block layer statistics documentation.

Else use something like Munin and graph it. You can get Munin to alert if it goes above a threshold, eg, 90% or whatever your graphing shows is a good alert figure.

eg, see these two Munin graphs showing that /dev/sdi needs looking at. In this example if /dev/sdi is part of an array the whole array would suffer because of it.

If you look at the week graph you'll see that /dev/sdc might be slow as well.

I should add that /dev/sdi above isn't broken, it's just a slow disk (actually a green disk that somebody added to an array of enterprise grade sata disks) which slowed the array down. An actual failed disk would stick out like a sore thumb.

In summary, I'd probably go with a script if I had the time, but Munin if I just wanted a quick solution and connecting to the server was easy.