1

MD raid5 array appears to have stopped working suddenly. Symptoms are somewhat similar to this issue in that I'm getting errors talking about not enough devices to start the array, however in my case the event counts on all three drives are equal. It's a raid 5 array that should have 2 active drives and one parity, however mdadm --examine on each drive shows two of them having their role listed as spare and only one as an active drive.

I've tried mdadm --stop /dev/md1 followed by mdadm --assemble /dev/md1 (including attempts with the --force and --run flags).

SMART data doesn't indicate any issues with the drives (and current pending and reallocated sector counts are all zero), I've tried the raid.wiki.kernel.org guide linked below by frostschutz through the steps involving setting up mapped overlay devices.

I would have then assumed running the following command would create a raid array that I could then attempt to mount read-only and see if that resulted in a readable filesystem or just a garbled mess (i.e. to determine if my guess of sdf1 being the parity drive was correct or if I should try again with sde1) - but instead it gives the error show below (have also tried with the associated loop devices as per losetup --list, with the same result).

mdadm --create /dev/md2 --assume-clean --level=5 --chunk=64K --metadata=1.2 --data-offset=261888s --raid-devices=3 missing /dev/mapper/sdh1 /dev/mapper/sdf1

mdadm: super1.x cannot open /dev/mapper/sdh1: Device or resource busy
mdadm: /dev/mapper/sdh1 is not suitable for this array.
mdadm: super1.x cannot open /dev/mapper/sdf1: Device or resource busy
mdadm: /dev/mapper/sdf1 is not suitable for this array.
mdadm: create aborted

Also, while mdadm --detail /dev/md1 previously gave the output (further) below, it now gives:

/dev/md1:
           Version : 1.2
        Raid Level : raid0
     Total Devices : 3
       Persistence : Superblock is persistent

             State : inactive
   Working Devices : 3

              Name : bob:1  (local to host bob)
              UUID : 07ff9ba9:e8100e68:94c12c1a:3d7ad811
            Events : 373364

    Number   Major   Minor   RaidDevice

       -     253       11        -        /dev/dm-11
       -     253       10        -        /dev/dm-10
       -     253        9        -        /dev/dm-9

Also, I've noticed dmsetup status gives the same information for all three overlays, and has a number that looks suspiciously like it may refer to the size of original raid array (16TB) rather than an individual drive (8TB) - not sure if this is as it should be?

sde1: 0 15627528888 snapshot 16/16777216000 16
sdh1: 0 15627528888 snapshot 16/16777216000 16
sdf1: 0 15627528888 snapshot 16/16777216000 16

Not sure how to progress from this point as far as attempting to create the device, mount and inspect the filesystem to confirm whether I guessed the correct parity device or not, using the overlay to prevent anything being written to the actual drives.

UPDATE: As per frostschutz's suggestion below, the array was somehow in some kind of state where --stop needed to be issued prior to being able to do anything with the underlying drives. I'd discounted that possibility previously as cat /proc/mdstat was showing the array as inactive, which I'd assumed meant it could not possibly be what was tying the drives up, but that was not in fact the case (I'd also previously ran --stop, but it would seem something was done afterwards triggering it's return to a non-stopped state). After getting the drive order correct (not on the first try, glad I was using overlays) the array passed a fsck check with no errors reported and is now up and running as if nothing ever happened.

The result of running other diagnostic commands:

cat /proc/mdstat:

Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md1 : inactive sdh1[1](S) sde1[3](S) sdf1[0](S)
      23440900500 blocks super 1.2

mdadm --detail /dev/md1:

/dev/md1:
           Version : 1.2
        Raid Level : raid0
     Total Devices : 3
       Persistence : Superblock is persistent

             State : inactive
   Working Devices : 3

              Name : bob:1  (local to host bob)
              UUID : 07ff9ba9:e8100e68:94c12c1a:3d7ad811
            Events : 373364

    Number   Major   Minor   RaidDevice

       -       8      113        -        /dev/sdh1
       -       8       81        -        /dev/sdf1
       -       8       65        -        /dev/sde1

lines appearing in dmesg when trying to mdadm --assemble /dev/md1:

md/raid:md1: device sdh1 operational as raid disk 1
md/raid:md1: not enough operational devices (2/3 failed)
md/raid:md1: failed to run raid set.
md: pers->run() failed ..

and the mdadm --examines

/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 07ff9ba9:e8100e68:94c12c1a:3d7ad811
           Name : bob:1  (local to host bob)
  Creation Time : Mon Mar  4 22:10:29 2019
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 15627267000 (7451.66 GiB 8001.16 GB)
     Array Size : 15627266688 (14903.32 GiB 16002.32 GB)
  Used Dev Size : 15627266688 (7451.66 GiB 8001.16 GB)
    Data Offset : 261888 sectors
   Super Offset : 8 sectors
   Unused Space : before=261808 sectors, after=312 sectors
          State : clean
    Device UUID : e856f539:6a1b5822:b3b8bfb7:4d0f4741

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun May 30 00:22:45 2021
  Bad Block Log : 512 entries available at offset 40 sectors
       Checksum : 9b5703bc - correct
         Events : 373364

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : spare
   Array State : .AA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdf1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 07ff9ba9:e8100e68:94c12c1a:3d7ad811
           Name : bob:1  (local to host bob)
  Creation Time : Mon Mar  4 22:10:29 2019
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 15627267000 (7451.66 GiB 8001.16 GB)
     Array Size : 15627266688 (14903.32 GiB 16002.32 GB)
  Used Dev Size : 15627266688 (7451.66 GiB 8001.16 GB)
    Data Offset : 261888 sectors
   Super Offset : 8 sectors
   Unused Space : before=261800 sectors, after=312 sectors
          State : clean
    Device UUID : 7919e56f:2e08430e:95a4c4a6:1e64606a

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun May 30 00:22:45 2021
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : d54ff3e1 - correct
         Events : 373364

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : spare
   Array State : .AA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdh1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 07ff9ba9:e8100e68:94c12c1a:3d7ad811
           Name : bob:1  (local to host bob)
  Creation Time : Mon Mar  4 22:10:29 2019
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 15627267000 (7451.66 GiB 8001.16 GB)
     Array Size : 15627266688 (14903.32 GiB 16002.32 GB)
  Used Dev Size : 15627266688 (7451.66 GiB 8001.16 GB)
    Data Offset : 261888 sectors
   Super Offset : 8 sectors
   Unused Space : before=261800 sectors, after=312 sectors
          State : clean
    Device UUID : 0c9a8237:7e79a439:d4e35b31:659f3c86

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun May 30 00:22:45 2021
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 6ec2604b - correct
         Events : 373364

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 1
   Array State : .AA ('A' == active, '.' == missing, 'R' == replacing)
Improve this question
1
  • Re-make the array and restore your backup.
    – Marc Wilson
    May 30, 2021 at 0:50

1 Answer 1

Reset to default
2

It looks odd. You might have to mdadm --create with overlays for this one (with correct data offset, chunk size, and drive order). And perhaps with the first drive missing, as that seems to have failed first...

Basically there is no way to recover with conventional means once a drive no longer even remembers its Device Role. Both say they're "spare", so it's unknown whether either drive was role 0, or role 2, or nothing at all (some raid5 setups actually use spares for some reason). So it's unclear: whether there is useful data on them at all, and if so what order it would be in. You have to determine yourself.

While you're at it, also check SMART data and use ddrescue first if any of these drives actually have reallocated or pending sectors that might have contributed to raid failure.

Improve this answer
5
  • Thanks, your answer to the linked question is very comprehensive. With a copy-on-write overlay obviously the underlying device is protected from writes, just to be clear is there anything elsewhere on the system that may affect how it interacts with the raid array that could be affected by going through a process like that? The only thing I can think of is /etc/mdadm/mdadm.conf, but I know I don't have a whole heap of experience with this sort of thing. I also don't know what Device Role 0 or 2 is but I'm assuming active and parity respectively?
    – Blair
    May 30, 2021 at 5:47
  • Also having some issues now that I have tried to --create using mapped overlays (SMART didn't indicate any issues, reallocated and pending counts were all zero) - have updated the question to show my attempt to --create a new array using the overlays based on a guess of which drive is the parity drive and the resultant error message.
    – Blair
    May 30, 2021 at 9:00
  • 1
    Device Role 0 1 2 ... just represents the order of drives, what is on them (data, parity) is determined by the RAID layout (in your case raid5 with 3 drives with the default left-symmetric layout).
    – frostschutz
    May 30, 2021 at 11:58
  • 1
    Regarding device busy errors, check /proc/mdstat if it's already in there. You might have active udev rules for md auto-assembly. Perhaps disable those (see if there is md assembly rules in /lib/udev/rules.d/ then create empty file in /etc/udev/rules.d to mask them). Or just try mdadm --stop any existing md devices first.
    – frostschutz
    May 30, 2021 at 12:01
  • Thank you very (very) much again - as per edits I've just made to the question, issuing (another) --stop did indeed fix things and allow me to use the overlays in a test array and pin down the correct drive order - which I've now been able to use to recreate the real array using the drives themselves :) Very much appreciate your help, all the very best :)
    – Blair
    May 30, 2021 at 17:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .