0

I have a RAID1 array with one sligtly damaged disk /dev/sdb. I put a fresh hard disk /dev/sda into this array in order to mirror sdb (then I’m gonna replace sdb with a fresh HDD).

# cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sda2[2] sdb2[1]
      999872 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sda3[2] sdb3[1]
      1952380736 blocks super 1.2 [2/1] [_U]
      [>....................]  recovery =  2.7% (53083136/1952380736) finish=640.0min speed=49453K/sec

unused devices: <none>

But this “recovering” process stops at about 30%:

# cat /proc/mdstat 
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sda2[2] sdb2[1]
      999872 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sda3[2](S) sdb3[1]
      1952380736 blocks super 1.2 [2/1] [_U]

unused devices: <none>

I guess the problem here is that /dev/sdb has about 100 bad blocks (discovered with badblocks -v -b 512 /dev/sdb > sdb-bads.log). Reading/writing from/to them is not available:

# tail -n 2 sdb-bads.log
1039341414
1039341415

# dd if=/dev/sdb of=/dev/null bs=512 skip=1039341414 count=1
dd: error reading '/dev/sdb': Input/output error
0+0 records in
0+0 records out
0 bytes copied, 9.33531 s, 0.0 kB/s

# dd if=/dev/sdb of=/dev/null bs=512 skip=1039341415 count=1
dd: error reading '/dev/sdb': Input/output error
0+0 records in
0+0 records out
0 bytes copied, 3.36192 s, 0.0 kB/s

My questions are:

  • Is there any possibility to find out which files are affected (assuming RAID1)? As I understand tools like debugfs work with physical file systems, not virtual RAIDs.
  • Is there any chance to skip those blocks during sda syncing (suppose, I am okay with losing some data) in order to finish RAID1 mirroring process?
|improve this question
  • Two variants of the same rube-goldberg-esque last-resort, in case there are no other answers: a) modify a libfuse example to export a fake filesystem containing a single (block device) file pointing to your /dev/sdb, with code that ignores EIOs. b) patch QEMU's virtio device to ignore EIO (read all of ./configure --help and disable everything, QEMU will build quicker if you need to make-clean). In both cases you could achieve reasonable confidence of correctness by checksumming a file through your code paths and comparing hash with native read. – i336_ Aug 15 '19 at 14:30
  • Another approach.. Create a failed-RAID with the new disk. Mount old one read-only and new one read-write. Use something like rsync -avHPXX /old/path/ /new/path to copy filesystem from old to new. It will complain on files with IO errors but continue trying to process the remainder of the files in the filesystem. When you've got everything sorted, replace the old disk and add a replacement into the new RAID. – roaima Aug 15 '19 at 15:32
  • Oh, thanks, @roaima. Will try this approach if my current hdparm --write-sector way fails. – Sasha MaximAL Aug 16 '19 at 10:30
0

Finally I had to rewrite all bad sectors (discovered with badblocks -v -b 512 /dev/sdb > sdb-bads.log command) with hdparm --write-sector. I have lost some of my data, but, at least, now I’m able to replace sdb with the fresh HDD, and do the recovery from sda.

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.