mdadm RAID array gone after reshape and reboot

Question

My mdadm RAID5 array just underwent a 5>8 disk grow and reshape. This took several days and went uninterrupted. When cat /proc/mdstat said it was complete I rebooted the system and now the array no longer shows.

One potential problem I can see is that I used the full disk when adding the new drives (e.g. /dev/sda not /dev/sda1). However these drives had partitions on them that should span the entire drive.

I have tried:

$ sudo mdadm --assemble --scan
mdadm: No arrays found in config file or automatically

The three newly added drives do not appear to have md superblocks:

$ sudo mdadm --examine /dev/sd[kln]
/dev/sdk:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
/dev/sdl:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
/dev/sdn:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)

$ sudo mdadm --examine /dev/sd[kln]1
mdadm: No md superblock detected on /dev/sdk1.
mdadm: No md superblock detected on /dev/sdl1.
mdadm: No md superblock detected on /dev/sdn1.

the five others do and show the correct stats for the array:

$ sudo mdadm --examine /dev/sd[ijmop]1
/dev/sdi1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 7399b735:98d9a6fb:2e0f3ee8:7fb9397e
           Name : Freedom-2:127
  Creation Time : Mon Apr  2 18:09:19 2018
     Raid Level : raid5
   Raid Devices : 8

 Avail Dev Size : 15627795456 (7451.91 GiB 8001.43 GB)
     Array Size : 54697259008 (52163.37 GiB 56009.99 GB)
  Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
    Data Offset : 254976 sectors
   Super Offset : 8 sectors
   Unused Space : before=254888 sectors, after=7168 sectors
          State : clean
    Device UUID : ca3cd591:665d102b:7ab8921f:f1b55d62

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Jul 14 11:46:37 2020
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 6a1bca88 - correct
         Events : 401415

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)

...ect

Forcing the assembly does not work:

$ sudo mdadm /dev/md1 --assemble --force /dev/sd[ijmop]1 /dev/sd[kln]
mdadm: /dev/sdi1 is busy - skipping
mdadm: /dev/sdj1 is busy - skipping
mdadm: /dev/sdm1 is busy - skipping
mdadm: /dev/sdo1 is busy - skipping
mdadm: /dev/sdp1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sdk
mdadm: /dev/sdk has no superblock - assembly aborted

I'm at a loss for how to proceed.

Thank you so much for any and all help.

Answer 1

One potential problem I can see is that I used the full disk when adding the new drives (e.g. /dev/sda not /dev/sda1). However these drives had partitions on them that should span the entire drive.

It's dangerous to use full disk for anything other than a partition table. As soon as anything else writes a partition table, your full disk RAID / LUKS / LVM / filesystem metadata is gone. And user error aside, there are a lot of tools and circumstances out there that may write a partition table without really asking you.

And that's exactly what seems to have happened to you. Either you, or something else has overwritten metadata on three disks with a partition table. There is usually no way to restore the missing metadata. For example, parted's mklabel gpt would zero the mdadm 1.2 metadata (4K from start) completely.

Thus, your only hope here is to re-create the RAID to build new metadata from scratch.

And it must be re-created exactly the same way it was, so if you're sure you used full disk and not partitions, you must re-create it with full disks, and in the right order too. Think about migrating to partition instead of full disk devices when you have your data back.

---

Note that your drive order is not alphabetical, you showed mdadm --examine output for /dev/sdi1 only which happens to be the fourth drive (counting from 0, device role 3) in your array. For a successful re-create, read examine output carefully to deduce the correct settings. Also your data offset is unusual (since --grow changes it).

With copy-on-write overlays in place, the command you're looking for should be something like:

mdadm --create /dev/md100 --assume-clean \
      --level=5 --chunk=512 --data-offset=127488 --layout=left-symmetric \
      --raid-devices=8 /dev/mapper/sd{?,?,?,i,?,?}1 /dev/mapper/sd{k,l,n}

(I don't know your drive order so I put ? instead of the correct drive letters, also note the {c,b,a} syntax keeps the order intact where [cba] would not. If in doubt, write it out instead of using shell expansions.)

---

To make sure the existing GPT partition table does not interfere again, you should remove it with wipefs (from your full disk members only). This removes GPT at both start and end of disk so any software looking for GPT and finding it at end of disk should not feel compelled to restore it at start of disk, wiping your metadata in the process.

# wipefs --no-act --all --types gpt,PMBR /dev/loop0
/dev/loop0: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/loop0: 8 bytes were erased at offset 0x7ffffe00 (gpt): 45 46 49 20 50 41 52 54
/dev/loop0: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa

(Remove --no-act to actually perform the erase.)

Good luck... if the partition table is the only problem, you should have a good chance of success. If other data was changed too (something created partitions and formatted them) you will see data corruption on the RAID itself.

---

PS:

$ sudo mdadm /dev/md1 --assemble --force /dev/sd[ijmop]1 /dev/sd[kln]
mdadm: /dev/sdi1 is busy - skipping

This message (busy skipping) usually means the md device is already assembled (that happens with incomplete arrays due to incremental assembly method).

In that case you have to mdadm --stop the inactive array first before attempting to mdadm --assemble it again. (That or continue with incremental assemble if it was actually a missing drive before).