16

I have my /boot partition in a RAID 1 array using mdadm. This array has degraded a few times in the past, and every time I remove the physical drive, add a new one, bring the array being to normal, it uses a new drive letter. Leaving the old one still in the array and failed. I can't seem to remove those all components that no longer exist.

[root@xxx ~]# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdg1[10] sde1[8](F) sdb1[7](F) sdd1[6](F) sda1[4] sdc1[5]
      358336 blocks super 1.0 [4/3] [UUU_]

Here's what I've tried to remove the non-existent drives and partitions. For example, /dev/sdb1.

[root@xxx ~]# mdadm /dev/md0 -r /dev/sdb1
mdadm: Cannot find /dev/sdb1: No such file or directory
[root@xxx ~]# mdadm /dev/md0 -r faulty
mdadm: Cannot find 8:49: No such file or directory
[root@xxx ~]# mdadm /dev/md0 -r detached
mdadm: Cannot find 8:49: No such file or directory

That 8:49 I believe refers to the major and minor number shown in --detail, but I'm not quite sure where to go from here. I'm trying to avoid a reboot or restarting mdadm.

[root@xxx ~]# mdadm --detail /dev/md0 
/dev/md0:
        Version : 1.0
  Creation Time : Thu Aug  8 18:07:35 2013
     Raid Level : raid1
     Array Size : 358336 (350.00 MiB 366.94 MB)
  Used Dev Size : 358336 (350.00 MiB 366.94 MB)
   Raid Devices : 4
  Total Devices : 6
    Persistence : Superblock is persistent

    Update Time : Sat Apr 18 16:44:20 2015
          State : clean, degraded 
 Active Devices : 3
Working Devices : 3
 Failed Devices : 3
  Spare Devices : 0

           Name : xxx.xxxxx.xxx:0  (local to host xxx.xxxxx.xxx)
           UUID : 991eecd2:5662b800:34ba96a4:2039d40a
         Events : 694

    Number   Major   Minor   RaidDevice State
       4       8        1        0      active sync   /dev/sda1
      10       8       97        1      active sync   /dev/sdg1
       5       8       33        2      active sync   /dev/sdc1
       6       0        0        6      removed

       6       8       49        -      faulty
       7       8       17        -      faulty
       8       8       65        -      faulty

Note: The array is legitimately degraded right now and I'm getting a new drive in there as we speak. However, as you can see above, that shouldn't matter. I should still be able to remove /dev/sdb1 from this array.

Improve this question
1
  • You don't need a --force to remove a disk in the "removed" state, and it might hide a more serious problem. Leave it off.
    – user149584
    Dec 31, 2015 at 22:16

3 Answers 3

Reset to default
18

It's because the device nodes no longer exist on your system (probably udev removed them when the drive died). You should be able to remove them by using the keyword failed or detached instead:

mdadm -r /dev/md0 failed     # all failed devices
mdadm -r /dev/md0 detached   # failed ones that aren't in /dev anymore

If your version of mdadm is too old to do that, you might be able to get it to work by mknod'ing the device to exist again. Or, honestly, just ignore it—it's not really a problem, and should go away the next time you reboot.

Improve this answer
4
  • I've tried using the keywords, you can see the output it gave me in the original post. I'll take a look at mknod. Yeah, it's probably not an issue, but I'm OCD, lol.
    – Sajan Parikh
    Apr 18, 2015 at 21:08
  • Yep, I used mknod to get this done and it worked.
    – Sajan Parikh
    Apr 18, 2015 at 21:57
  • @SajanParikh indeed, somehow I'd missed the lines showing you'd already tried failed and detached. Glad that mknod worked.
    – derobert
    Apr 18, 2015 at 22:35
  • 1
    Clarifying note: The OP did not state he used "failed". I got the same errors they did, but "failed" worked for me. They used "faulty".
    – Fmstrat
    Feb 19, 2022 at 14:27
7

What I ended up doing was using mknod like @derobert suggested to create the devices that mdadm was looking for. I tried the major/minor numbers mdadm was telling me it couldn't find with the different drive letters I was trying to remove until it worked.

mknod /dev/sde1 b 8 17

Then I had to use the --force option to get it remove the component.

mdadm /dev/md0 --remove --force /dev/sde1

Then, I removed that created block device.

rm /dev/sde1
Improve this answer
3
  • Hmm... 8:17 is sdb1. (8:65 is sde1) But that one was missing too, so I guess that works...
    – derobert
    Apr 18, 2015 at 22:37
  • This didn't worked for me, mdadm continued to say: "device or resource busy", but this made me try to feed him not with a fake block device, but with a "true" block device such as a loopback mounted image. At this point, I discovered that I had a stale /dev/loop that was still using a file on the degraded array. I detached it and finally mdadm let me stop the array. Horay! For everybody reading this, there is always a logical explanation for mdadm being so jerk, so look for a stale process/file/mountpoint/nfs handler/open bash/loopback device/etc. still using the degraded array. :)
    – Avio
    Feb 8, 2016 at 11:45
  • I was able to use exactly the same major and minor versions (8:18 in my case) to mknod a fake /dev/sdb2 device. After that, mdadm --remove deleted stale record of /dev/sdb2 from /proc/mdstat. Remember to rm /dev/sdb2 after successful --remove action.
    – ILIV
    Nov 29, 2018 at 15:07
6

You could also fix just by degreasing number of disks in array:

In my case I have raid-1 array /dev/md0 with /dev/sda1 and "removed". I just simply shrunk it to use one drive only:

mdadm -G /dev/md0 --raid-devices=1 --force

After that removed was really removed (no removed lines anymore in mdadm --detail)

Improve this answer
2
  • 2
    You have to be careful with this approach, though. Understand well what type of RAID you're dealing with before modifying --raid-devices.
    – ILIV
    Nov 29, 2018 at 15:08
  • This is exactly how to remove failed drive.
    – Scrooge McDuck
    Dec 31, 2021 at 17:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .