Worrisome HD related messages after power outage

Question

Had a power outage two weeks ago on my BSD server that my UPS decided then was a good time to fail. Just fired it up this evening only to have the following errors show during what I would assume is the rebuild process.

The port for the ECC error has shown up as either 2 or 1 in following errors, and while the rest of the data that you see on the image has not shown up since it stated the rebuild began, the Drive ECC error has popped once twice since that command was displayed.

While I have another system to backup critical data too, my questions really are what is this saying, and should I be shopping for hardware. The Drives are WD 250GB on a hardware RAID5 with a 3Ware 9650SE-4LPML 4-Port 3Gb/s RAID card. The OS is FreeBSD 6.2

EDIT: Something different now. Came up with an error that retries were exhausted, and produced the following line

g_vfs_dome():da0s1d[READ(offset=1155956736, length=16384)]error = 5

It then reset the controller and started a rebuild on unit -

Answer 1

What you should do here is trigger a SMART self-test on the bad drive(s). That will pull a lot of the controller/motherboard pieces out of here and give you a better reading on the underlying disks having issues. It's possible for non-drive failure to give wrong results there--power supply failure is the most common cause--but it's a good start. It will take a few hours to run an extended test, but you may get useful enough info with the short one. Drives that are failing tend to complain pretty fast even on that one.

The guide at Utilizing SMART to Monitor Drives in 3ware RAID should give you enough info to trigger a self-test and then view the logged results. Probably worthwhile to check the controller card logs for more info here before even running a new test. It would be interesting to know if errors were already increasing before the power outage or not. Sometimes RAID arrays can have hidden inconsistencies you just don't know about. Not impossible for the power cutting out to have damaged one sector via inconsistent writes, and then the problems you didn't know about cause the rebuild to go badly. If there were older sector repairs happening, you may find them in the controller logs, even if the controller fixed them quietly and didn't tell the Linux driver.