1

We have 2 disks in a RAID1 md array. MDADM detected a disk failure and "failed" the disk in mdadm, but then there was an unexpected reboot of the server. When I check /proc/mdstat or mdadm --detail /dev/md0, it shows "Removed".

Usually when the disk is marked "failed" by mdadm, it will show the disk name in /proc/mdstat with a (F) or in the mdadm --detail /dev/md0, it shows it. This makes it easy to identify the failed disk and remove the correct one from the server to replace.

But after the reboot , when mdadm auto "Removed" the drive from the mdadm array, the failed disk doesn't show up anyway in those commands. I understand you could potentially loop through a mdadm --examine /dev/dm-*) of every drive and find the mdadm header with the array UUID, but I can see in some cases, it shows mdadm: No md superblock detected on xxx, even on the correct drive (on another test server where I know for certain what the disk names were)

So my question is how do you consistently identify the failed disk name when mdadm auto removes it (only noticed this happens when mdadm "fails" it and then the server reboots somehow (or intentionally)).

1
  • 1
    You should have more mdadm information in the kernel log from the point of failure Dec 2, 2021 at 21:38

1 Answer 1

1

One of the possible failure modes of a disk is that it completely stops responding, which makes it undetectable in subsequent reboots: as far as the kernel is concerned, it will seem like the disk was already unplugged.

Because of this, the only fully generic and reliable method to identify the failing disk in such a situation is by exclusion: you list all the disks that are working (with e.g. lsblk -o +SERIAL,MODEL to include the disk serial numbers and model names to the listing), and then look for physical disks that are not on the list.

If the failed disk happens to be still detectable to the system, it should also be fairly obvious in the lsblk output, as a device that may or may not have partitions but no mountpoints associated with them, and no association to the RAID set (anymore).

Some disks even start reporting their size as 0 if they fail their internal self-tests, so the SIZE column of the lsblk output can be useful here too.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .