2

My mdadm RAID5 array just underwent a 5>8 disk grow and reshape. This took several days and went uninterrupted. When cat /proc/mdstat said it was complete I rebooted the system and now the array no longer shows.

One potential problem I can see is that I used the full disk when adding the new drives (e.g. /dev/sda not /dev/sda1). However these drives had partitions on them that should span the entire drive.

I have tried:

$ sudo mdadm --assemble --scan
mdadm: No arrays found in config file or automatically

The three newly added drives do not appear to have md superblocks:

$ sudo mdadm --examine /dev/sd[kln]
/dev/sdk:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
/dev/sdl:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
/dev/sdn:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)

$ sudo mdadm --examine /dev/sd[kln]1
mdadm: No md superblock detected on /dev/sdk1.
mdadm: No md superblock detected on /dev/sdl1.
mdadm: No md superblock detected on /dev/sdn1.

the five others do and show the correct stats for the array:

$ sudo mdadm --examine /dev/sd[ijmop]1
/dev/sdi1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 7399b735:98d9a6fb:2e0f3ee8:7fb9397e
           Name : Freedom-2:127
  Creation Time : Mon Apr  2 18:09:19 2018
     Raid Level : raid5
   Raid Devices : 8

 Avail Dev Size : 15627795456 (7451.91 GiB 8001.43 GB)
     Array Size : 54697259008 (52163.37 GiB 56009.99 GB)
  Used Dev Size : 15627788288 (7451.91 GiB 8001.43 GB)
    Data Offset : 254976 sectors
   Super Offset : 8 sectors
   Unused Space : before=254888 sectors, after=7168 sectors
          State : clean
    Device UUID : ca3cd591:665d102b:7ab8921f:f1b55d62

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Jul 14 11:46:37 2020
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 6a1bca88 - correct
         Events : 401415

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)

...ect

Forcing the assembly does not work:

$ sudo mdadm /dev/md1 --assemble --force /dev/sd[ijmop]1 /dev/sd[kln]
mdadm: /dev/sdi1 is busy - skipping
mdadm: /dev/sdj1 is busy - skipping
mdadm: /dev/sdm1 is busy - skipping
mdadm: /dev/sdo1 is busy - skipping
mdadm: /dev/sdp1 is busy - skipping
mdadm: Cannot assemble mbr metadata on /dev/sdk
mdadm: /dev/sdk has no superblock - assembly aborted

I'm at a loss for how to proceed.

Thank you so much for any and all help.

1
  • Add the output of cat /proc/mdstat to your question. Jul 14 '20 at 19:44
2

One potential problem I can see is that I used the full disk when adding the new drives (e.g. /dev/sda not /dev/sda1). However these drives had partitions on them that should span the entire drive.

It's dangerous to use full disk for anything other than a partition table. As soon as anything else writes a partition table, your full disk RAID / LUKS / LVM / filesystem metadata is gone. And user error aside, there are a lot of tools and circumstances out there that may write a partition table without really asking you.

And that's exactly what seems to have happened to you. Either you, or something else has overwritten metadata on three disks with a partition table. There is usually no way to restore the missing metadata. For example, parted's mklabel gpt would zero the mdadm 1.2 metadata (4K from start) completely.

Thus, your only hope here is to re-create the RAID to build new metadata from scratch.

And it must be re-created exactly the same way it was, so if you're sure you used full disk and not partitions, you must re-create it with full disks, and in the right order too. Think about migrating to partition instead of full disk devices when you have your data back.


Note that your drive order is not alphabetical, you showed mdadm --examine output for /dev/sdi1 only which happens to be the fourth drive (counting from 0, device role 3) in your array. For a successful re-create, read examine output carefully to deduce the correct settings. Also your data offset is unusual (since --grow changes it).

With copy-on-write overlays in place, the command you're looking for should be something like:

mdadm --create /dev/md100 --assume-clean \
      --level=5 --chunk=512 --data-offset=127488 --layout=left-symmetric \
      --raid-devices=8 /dev/mapper/sd{?,?,?,i,?,?}1 /dev/mapper/sd{k,l,n}

(I don't know your drive order so I put ? instead of the correct drive letters, also note the {c,b,a} syntax keeps the order intact where [cba] would not. If in doubt, write it out instead of using shell expansions.)


To make sure the existing GPT partition table does not interfere again, you should remove it with wipefs (from your full disk members only). This removes GPT at both start and end of disk so any software looking for GPT and finding it at end of disk should not feel compelled to restore it at start of disk, wiping your metadata in the process.

# wipefs --no-act --all --types gpt,PMBR /dev/loop0
/dev/loop0: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/loop0: 8 bytes were erased at offset 0x7ffffe00 (gpt): 45 46 49 20 50 41 52 54
/dev/loop0: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa

(Remove --no-act to actually perform the erase.)

Good luck... if the partition table is the only problem, you should have a good chance of success. If other data was changed too (something created partitions and formatted them) you will see data corruption on the RAID itself.


PS:

$ sudo mdadm /dev/md1 --assemble --force /dev/sd[ijmop]1 /dev/sd[kln]
mdadm: /dev/sdi1 is busy - skipping

This message (busy skipping) usually means the md device is already assembled (that happens with incomplete arrays due to incremental assembly method).

In that case you have to mdadm --stop the inactive array first before attempting to mdadm --assemble it again. (That or continue with incremental assemble if it was actually a missing drive before).

4
  • Thank you so much for your detailed reply. Does the order of the added drives matter as well? And if so, is it the order they were added to the raid before the grow command was issued? Also, you mentioned the data offset of the devices being odd, and in the command that you gave you use 127488, how did you determine that number? Jul 14 '20 at 20:12
  • Yes it matters, yes it should be. Otherwise you just have to try all possible orders... whoops, I forgot --assume-clean in my previous reply, just edited it in. Re-creating RAID is dangerous, make sure to use overlays or take other measures to prevent further data loss. Jul 14 '20 at 20:16
  • Sorry, two more questions before I go diving into this. When looking through my bash history to determine the order the new drives were added it looks like I may have added one drive twice(?) and not added one another. Would mdadm allow such a thing? Additionally, using the write overlays should prevent me from changing the data on the drives, so I could I try different orders until I get it right? Even though I know the order of most of the drives this would give me some freedom in the last three. Thank you again so much for helping me through this. :) Jul 14 '20 at 21:09
  • mdadm shouldn't allow it. with overlays you can re-try. you have the right order if you can verify the contents of a 4M (8*512K) file to be intact. Otherwise if you flip two drives out of order, it might mount first but then have filesystem corruption further down. Jul 14 '20 at 21:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.