mdadm RAID5 with bad sectors and bad blocks

Question

I have a 4 disk RAID5 setup with mdadm in Ubuntu 16.04.

sdb : good

sdc : 8 bad sectors according to SMART, 8 pending, 4 reported errors

sdd : bad blocks present according to mdadm --examine

sde : good

I was naive and assumed the system would notify me of a disk failure. Since the last update according to "mdadm --examine /dev/sdd" was November, I know that's when it was removed from the RAID. For 7 months, I've been unwittingly running in degraded mode. Until this past weekend, when it marked sdc as faulty and my RAID became inactive due to insufficient disks.

SMART (smartctl --xall) says all the disks passed but reports 8 uncorrectable sectors on sdc and 4 uncorrectable errors. So the disks are reasonably healthy.

The event count was obviously way off on sdd, but only 63 off on sdc. So I forced an assembly with three disks, and all my data is accessible. I am sure there are some corrupted files but it is not a huge concern for this data.

I then added the 4th disk back with "mdadm /dev/md0 -a /dev/sdd" and it started to restore but ran into the bad sectors on sdc which it could not reallocate (I assume because it is degraded?) and mdadm marked it as faulty again, and I'm back to where I started.

I found some good instructions on how to keep disks with bad blocks in an array here. https://stephane.lesimple.fr/blog/how-to-securely-keep-a-hard-drive-with-bad-blocks-in-a-raid-array/

However, I made another mistake. Since I wasn't sure I was going to stick with Linux, the array is formatted with NTFS in case I wanted to run Windows instead. This means none of the fs commands like fsck or debugfs work.

chkdsk might be able to help, but I'd need to install Windows, and I don't want to go through that pain unless I am sure it will work. I am hoping there is a way to do this with existing Linux utilities. I have the sectors from kern.log, but strangely it lists 10 sectors with "read error not correctable" messages, which is more than the 8 that SMART reports. If I can convert these sectors to logical blocks, then I could potentially zero them out with dd.

I've already backed up anything critical, so I can just buy new drives, but seems wasteful to toss 4TB drives for a handful of bad sectors. I am not even sure what is wrong with sdd since it has no bad sectors.

Note: When the RAID was trying to restore after adding the 4th disk, there are hundreds of "failed command: READ FPDMA QUEUED" messages in the kern.log Googling this suggests I could have a bad SATA cable or power supply. I have had my fair share of PSU's fail on me, but this would be the first time it causes hard drive failure if that is the case.

Answer 1

ddrescue did successfully copy from the drive with bad sectors to the one that was disabled from the array back in November, however, only after I replaced my power supply. In /var/log/kern.log I saw hundreds of failed WRITE FDMA QUEUE commands, so pulled the PSU from a newer machine and after the transplant, ddrescue worked fine. Took about 10 hours for the 4TB disks. It reported 15 errors totalling 80kb. Once it finished, sdd looked just like sdc as you'd expect, so I assembled the array with sdb sdd and sde, then I added sdd and let it restore, which it completed without error. In fact, SMART no longer reports any bad sectors on sdd. I assume this is because writing to those sectors caused the sectors to be reallocated by the disk. So all is well, just need to order a new PSU.