I recently got an IO Error on one of my RAID5 members; on a 3 disk fakeRAID array. I didn't notice at the time - there was only a message in /var/log/kern.log - so I had continued running the machine from that partition for a good while longer.
The BIOS / Intel RAID Manager didn't pick up any problems, but when I chose to boot from that partition, / was being mounted read-only.
Now I'm booted from a recovery partition, and have just run e2fsck -c -y on the partition, which has picked up quite a few errors, including: "Multiply-claimed blocks", "Unattached inode", "Free inodes count wrong for group #xxx", etc.. It has also reproduced the IO error in kern.log, so dmraid has degraded the array and I can now tell easily which drive has failed.
Now, what to do about the dodgy drive? I will get an advance RMA, but that takes about 5 days, so in the mean-time, I need a running machine, and a degraded RAID5 volume is not good karma!
The messages I'm seeing from dmesg are:-
ata3.00: exception Emask 0x10 SAct 0x1 SErr 0x280100 action 0x6 frozen
ata3.00: irq_stat 0x08000000, interface fatal error
ata3: SError: { UnrecovData 10B8B BadCRC }
ata3.00: failed command: READ FPDMA QUEUED
ata3.00: cmd 60/00:00:00:0b:0c/01:00:14:00:00/40 tag 0 ncq 131072 in
res 40/00:04:00:0b:0c/00:00:14:00:00/40 Emask 0x10 (ATA bus error)
ata3.00: status: { DRDY }
ata3: hard resetting link
ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata3.00: configured for UDMA/133
sd 2:0:0:0: [sdb]
Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 2:0:0:0: [sdb]
Sense Key : Aborted Command [current] [descriptor]
Descriptor sense data with sense descriptors (in hex):
72 0b 00 00 00 00 00 0c 00 0a 80 00 00 00 00 00
14 0c 0b 00
sd 2:0:0:0: [sdb]
Add. Sense: No additional sense information
sd 2:0:0:0: [sdb] CDB:
Read(10): 28 00 14 0c 0b 00 00 01 00 00
end_request: I/O error, dev sdb, sector 336333568
ata3: EH complete
device-mapper: dm-raid45: CRITICAL: io error on device /dev/sdb in region=336329728; DEGRADING RAID set
device-mapper: dm-raid45: further device error messages suppressed
Am I right in thinking that (from the above dmesg errors) that only some regions of the drive are bad? If so, can I continue to use the drive whilst avoiding those bad blocks? I'm inclined to format the drive and rebuild the array on to it, until a replacement arrives. Is that a bad idea?
Also, the SMART tests seem to be fine on all RAID disks...
