I have a 4 disk RAID5 setup with mdadm in Ubuntu 16.04.
sdb : good
sdc : 8 bad sectors according to SMART, 8 pending, 4 reported errors
sdd : bad blocks present according to mdadm --examine
sde : good
I was naive and assumed the system would notify me of a disk failure. Since the last update according to "mdadm --examine /dev/sdd" was November, I know that's when it was removed from the RAID. For 7 months, I've been unwittingly running in degraded mode. Until this past weekend, when it marked sdc as faulty and my RAID became inactive due to insufficient disks.
SMART (smartctl --xall) says all the disks passed but reports 8 uncorrectable sectors on sdc and 4 uncorrectable errors. So the disks are reasonably healthy.
The event count was obviously way off on sdd, but only 63 off on sdc. So I forced an assembly with three disks, and all my data is accessible. I am sure there are some corrupted files but it is not a huge concern for this data.
I then added the 4th disk back with "mdadm /dev/md0 -a /dev/sdd" and it started to restore but ran into the bad sectors on sdc which it could not reallocate (I assume because it is degraded?) and mdadm marked it as faulty again, and I'm back to where I started.
I found some good instructions on how to keep disks with bad blocks in an array here. https://stephane.lesimple.fr/blog/how-to-securely-keep-a-hard-drive-with-bad-blocks-in-a-raid-array/
However, I made another mistake. Since I wasn't sure I was going to stick with Linux, the array is formatted with NTFS in case I wanted to run Windows instead. This means none of the fs commands like fsck or debugfs work.
chkdsk might be able to help, but I'd need to install Windows, and I don't want to go through that pain unless I am sure it will work. I am hoping there is a way to do this with existing Linux utilities. I have the sectors from kern.log, but strangely it lists 10 sectors with "read error not correctable" messages, which is more than the 8 that SMART reports. If I can convert these sectors to logical blocks, then I could potentially zero them out with dd.
I've already backed up anything critical, so I can just buy new drives, but seems wasteful to toss 4TB drives for a handful of bad sectors. I am not even sure what is wrong with sdd since it has no bad sectors.
Note: When the RAID was trying to restore after adding the 4th disk, there are hundreds of "failed command: READ FPDMA QUEUED" messages in the kern.log Googling this suggests I could have a bad SATA cable or power supply. I have had my fair share of PSU's fail on me, but this would be the first time it causes hard drive failure if that is the case.