I'm pretty sure I'm dealing with at stuffed RAID 5 array. Two of the drives were listed as failed in /proc/mdstat and mdadmdetails:
0 0 8 33 0 active sync /dev/sdc1
1 1 0 0 1 faulty removed
2 2 0 0 2 faulty removed
3 3 8 1 3 active sync /dev/sda1
md127 : inactive sdd1[0](S) sdb1[3](S) sde1[2](S) sdc1[1](S)
3907039744 blocks
So sdb and sdd are failed. I then ran SMART diagnosis on my drives and confirmed that I've got current pending sectors, about 1000 of them on both of the failed disks.
Then running badblocks has confirmed that I do have many badblocks about. Trying to zero these out by hand would be a total nightmare, being in RAID5 which I can assemble but can't start anymore and then in LVM makes it a real pain in the ass. I think following something like this just isn't going to work.
I'm open to using MHDD if people have confirmed it to work (suggestions?), it was slow when I was trying it so I aborted and instead started backing up the drives instead.
http://hddguru.com/software/2005.10.02-MHDD/mhdd_manual.en.html#scanning
Currently doing a ddrescue and moving the data over to other disks, hopefully will be able to make a master copy - possibly zero out stuffed drives, and then dd copy back to those drives. Though not knowing much about ddrescue I feel I'm just going to be moving my problem about instead of fixing it.
http://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html
