4

Context

I have a software RAID5 array (mdadm) on 3 disks. Last week, one disk started to get reading issue:

# dmesg
ata3.00: exception Emask 0x0 SAct 0x30000001 SErr 0x0 action 0x0
ata3.00: irq_stat 0x40000008
ata3.00: failed command: READ FPDMA QUEUED
ata3.00: cmd 60/08:e0:40:0b:c6/00:00:a1:00:00/40 tag 28 ncq dma 4096 in
         res 41/40:00:40:0b:c6/00:00:a1:00:00/40 Emask 0x409 (media error) <F>
ata3.00: status: { DRDY ERR }
ata3.00: error: { UNC }
ata3.00: configured for UDMA/133
sd 2:0:0:0: [sdc] tag#28 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=5s
sd 2:0:0:0: [sdc] tag#28 Sense Key : Medium Error [current] 
sd 2:0:0:0: [sdc] tag#28 Add. Sense: Unrecovered read error - auto reallocate failed
sd 2:0:0:0: [sdc] tag#28 CDB: Read(16) 88 00 00 00 00 00 a1 c6 0b 40 00 00 00 08 00 00
I/O error, dev sdc, sector 2714110784 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
ata3: EH complete

So I've formatted and added a new device to the array and then grow the array

# mdadm --add /dev/md0 /dev/sdd1
# mdadm --grow --raid-devices=4 /dev/md0

It seems that it wasn't the best idea. Due to reading issue of the faulty disk, reshape operation estimated duration is more or less 6 months. (below 12-hours progress)

$ cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] 
md0 : active raid5 sdd1[4] sdc1[0] sde1[1] sdb1[3]
      5860269184 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      [>....................]  reshape =  0.2% (6232960/2930134592) finish=265471.5min speed=183K/sec
      bitmap: 4/22 pages [16KB], 65536KB chunk

unused devices: <none>

So many events can occur meanwhile like power issue or second disk fail for example.

I would love telling mdadm to stop reading the faulty disk but it seems stopping reshaping operation may lead to data loss.

Questions

  1. Should I mark faulty the disk with reading issue while reshaping operation?
  2. Is there a clever way to speed up reshaping?
  3. Any other advices?

Thanks a lot for your ideas and help.

2
  • Do you have a good data backup ?
    – Criggie
    Commented Jun 17 at 20:55
  • 1
    @Criggie not recent enough indeed but the faulty disk prevents me from backup data now. My main hope is to follow frostschutz advices.
    – BiBzz
    Commented Jun 18 at 11:36

1 Answer 1

3

Caught between a rock and a hard place. Pick your poison.

You could

  • leave it running as is and hope for the best,
  • --fail /dev/sdc1 to kick the offending drive directly,
  • add another drive and --replace /dev/sdc1, (perhaps this is what you meant to do instead of --grow?)

Or --stop the array and then

  • --assemble --update=revert-reshape to undo --grow,
  • --assemble --freeze-reshape while you backup your files,
  • ddrescue to clone the failing drive externally,

The problem is that in theory many of these options should work just fine, but in practice there are bugs and so… it's somewhat impossible to predict how things are going to play out.

Do check smartctl -a if there are pending/uncorrectable/reallocated sectors or other log entries. If these values are rising it might mean that RAID is correcting read errors as it goes, and not just stuck on a bug. smartctl can set TLER / SCTERC for some drives, which can reduce the time the drive spends on read errors.

Also check mdadm --examine, --examine-badblocks since md might be recording bad blocks in metdata instead of kicking the drive. If there are bad blocks on multiple drives, you already lost data. This data is not replicated in subsequent reshape or replace operations.

Reverting the reshape is undocumented and rarely tested (moreso with a failing drive in the mix) but since your grow barely processed so far, it might be the fastest way to get you out of this situation.

ddrescue could clone your failing drive, but any read errors would not be corrected by the raid layer then. If you use a bad clone in a raid array, it will result in corrupt data.

10
  • Thanks a lot @frostschutz for your valuable inputs. Indeed, mdadm --examine-badblocks returns 2 disks affected. sdb with 14 items for 8 sectors each, sdc with 23 items (mainly 8 sectors but also 16, 24, 56 and 104 sectors implied).
    – BiBzz
    Commented Jun 17 at 15:12
  • 1
    If you fail the drive and rebuild w/o redundancy, you can also --assemble --update=force-no-bbl to maybe, use whatever was stored for these bad sectors, or at least continue without bad block lists after the drive is fixed. Regarding damage or recovery it depends on what was using these blocks... Commented Jun 17 at 15:31
  • Ok then I'll try first to add a new drive and --replace /dev/sdc1/ (I thought it wouldn't be allowed since reshaping is ongoing. You're totally right I should have done this instead of grow). If identical entries persist, I'll run --assemble --update=force-no-bbl to forcibly wipe Bad Block List like you explain it too on your awesome answer on raid5 bad blocks. Thanks a lot for those steps, I'll update this thread to give feedbacks.
    – BiBzz
    Commented Jun 17 at 15:53
  • @BiBzz you may already know, but you can ↑ Vote Up as many excellent answers like this one as you like. (You can also ✔ Accept the answer to your question that best answers it.) Commented Jun 17 at 16:37
  • 1
    --replace is a softer alternative to --fail which allows the to-be-replaced device to still help along with the rebuild. if you use --fail then --replace is no longer possible. Commented Jun 21 at 21:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .