How can I reduce the IO wait time and retry times so that the OS doesn't continually try to write to a failing drive?
I have an system that I use to make copies of demo content that gets loaned out to customers on to regular SATA desktop hard drives. We connect many drives at once via SAS and copy content to them using a script.
Because the drives are loaned out, occasionally some come back damaged but I don't know that they are damaged, so the next time that drive gets reused in a copy operation, it slows down other drives as the system retries IO to that drive. Sometimes it can take hours before I notice the bad drive and remove it. After the drive is removed, the rest of the drives begin writing at normal speed.
I do not care about recovering the bad drives. I just need to weed them out so they don't slow everything else down.
I am also researching badblocks and smartmontools and considering writing a pre-check on the drives before I start writing.
OS: Ubuntu Linux (12.04 lts)