0

I have a machine which had a mdadm RAID1 array, consisting of 2 8TB disks (/dev/sdc and /dev/sdd). This worked fine, and I added a bunch of data on it.

I did many dry-runs on a different machine growing a RAID1 with 2 disks into (eventually) a RAID5 with 5 disks, which was working as expected.

Tl;dr

What am I missing to grow a RAID1 array with 2 disks into a RAID5 array with 3 disks? The array is inactive and missing a device after reboot!

What I did:

  • Changing the RAID level to 5: mdadm --grow /dev/md0 -l 5
  • Adding a spare HDD: mdadm /dev/md0 --add /dev/sdb
  • Growing the RAID to use the new disk: mdadm --grow /dev/md0 -n 3
  • After this the synchronization starts

This is the output during the synchronization:

user@server:~$ sudo mdadm -D /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Tue Jul 19 17:56:28 2022
        Raid Level : raid5
        Array Size : 7813894464 (7.28 TiB 8.00 TB)
     Used Dev Size : 7813894464 (7.28 TiB 8.00 TB)
      Raid Devices : 3
     Total Devices : 3
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Thu Aug 25 18:25:21 2022
             State : clean, reshaping
    Active Devices : 3
   Working Devices : 3
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 64K

Consistency Policy : bitmap

    Reshape Status : 24% complete
     Delta Devices : 1, (2->3)

              Name : ubuntu-server:0
              UUID : 9d1e2e6e:14dc5960:011daf54:xxxxxxxx
            Events : 77556

    Number   Major   Minor   RaidDevice State
       0       8       32        0      active sync   /dev/sdc
       1       8       48        1      active sync   /dev/sdd
       2       8       16        2      active sync   /dev/sdb

And the disks:

user@server:~$ lsblk
NAME                   MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
loop0                    7:0    0    62M  1 loop  /snap/core20/1593
loop1                    7:1    0    62M  1 loop  /snap/core20/1611
loop2                    7:2    0  79.9M  1 loop  /snap/lxd/22923
loop3                    7:3    0   103M  1 loop  /snap/lxd/23541
loop4                    7:4    0  44.7M  1 loop  /snap/snapd/15534
loop5                    7:5    0    47M  1 loop  /snap/snapd/16292
sda                      8:0    0 931.5G  0 disk
├─sda1                   8:1    0     1M  0 part
├─sda2                   8:2    0     2G  0 part  /boot
└─sda3                   8:3    0 929.5G  0 part
  └─dm_crypt-1         253:0    0 929.5G  0 crypt
    └─ubuntu--vg-lv--0 253:1    0 929.5G  0 lvm   /
sdb                      8:16   0   7.3T  0 disk
└─md0                    9:0    0   7.3T  0 raid5
sdc                      8:32   0   7.3T  0 disk
└─md0                    9:0    0   7.3T  0 raid5
sdd                      8:48   0   7.3T  0 disk
└─md0                    9:0    0   7.3T  0 raid5

After the synchronization was finished, I was able to mount my array and access the data, although the array size was still 8TB (which I assume I have to increase manually).

Fast forward 24h to now (after a reboot):

root@server:~# mdadm -D /dev/md0
/dev/md0:
           Version : 1.2
        Raid Level : raid5
     Total Devices : 2
       Persistence : Superblock is persistent

             State : inactive
   Working Devices : 2

              Name : ubuntu-server:0
              UUID : 9d1e2e6e:14dc5960:011daf54:xxxxxxxx
            Events : 85828

    Number   Major   Minor   RaidDevice

       -       8       32        -        /dev/sdc
       -       8       48        -        /dev/sdd

root@server:~# lsblk
NAME                   MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
loop0                    7:0    0    62M  1 loop  /snap/core20/1593
loop1                    7:1    0    62M  1 loop  /snap/core20/1611
loop2                    7:2    0  79.9M  1 loop  /snap/lxd/22923
loop3                    7:3    0   103M  1 loop  /snap/lxd/23541
loop4                    7:4    0  44.7M  1 loop  /snap/snapd/15534
loop5                    7:5    0    47M  1 loop  /snap/snapd/16292
sda                      8:0    0 931.5G  0 disk
├─sda1                   8:1    0     1M  0 part
├─sda2                   8:2    0     2G  0 part  /boot
└─sda3                   8:3    0 929.5G  0 part
  └─dm_crypt-1         253:0    0 929.5G  0 crypt
    └─ubuntu--vg-lv--0 253:1    0 929.5G  0 lvm   /
sdb                      8:16   0   7.3T  0 disk
sdc                      8:32   0   7.3T  0 disk
└─md0                    9:0    0     0B  0 md
sdd                      8:48   0   7.3T  0 disk
└─md0                    9:0    0     0B  0 md

So, it seems to me the newly added HDD (/dev/sdb) somehow got lost! I tried adding the output of mdadm --detail --scan --verbose to my /etc/mdadm/mdadm.conf and running update-initramfs -u after, but to no avail...

Here's a little more info:

root@server:~# mdadm --examine /dev/sd[bcd]
/dev/sdb:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 9d1e2e6e:14dc5960:011daf54:403c80a6
           Name : ubuntu-server:0
  Creation Time : Tue Jul 19 17:56:28 2022
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 15627789488 sectors (7.28 TiB 8.00 TB)
     Array Size : 15627788928 KiB (14.55 TiB 16.00 TB)
  Used Dev Size : 15627788928 sectors (7.28 TiB 8.00 TB)
    Data Offset : 263680 sectors
   Super Offset : 8 sectors
   Unused Space : before=263600 sectors, after=560 sectors
          State : clean
    Device UUID : cce44b44:6be581c6:ed09e3e8:5a2f5735

Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Aug 26 19:56:25 2022
  Bad Block Log : 512 entries available at offset 40 sectors
       Checksum : 370fd1fa - correct
         Events : 85828

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 0
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 9d1e2e6e:14dc5960:011daf54:403c80a6
           Name : ubuntu-server:0
  Creation Time : Tue Jul 19 17:56:28 2022
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 15627789488 sectors (7.28 TiB 8.00 TB)
     Array Size : 15627788928 KiB (14.55 TiB 16.00 TB)
  Used Dev Size : 15627788928 sectors (7.28 TiB 8.00 TB)
    Data Offset : 263680 sectors
   Super Offset : 8 sectors
   Unused Space : before=263600 sectors, after=560 sectors
          State : clean
    Device UUID : 5744d817:29d6e7e7:30e536d7:16d43c13

Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Aug 26 19:56:25 2022
  Bad Block Log : 512 entries available at offset 40 sectors
       Checksum : f83ba242 - correct
         Events : 85828

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 1
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)

root@server:~# cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md0 level=raid5 num-devices=3 metadata=1.2 name=ubuntu-server:0 UUID=9d1e2e6e:14dc5960:011daf54:xxxxxx
   devices=/dev/sdb,/dev/sdc,/dev/sdd
root@server:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:        22.04
Codename:       jammy
4
  • Your mdadm.conf is too verbose (just use UUID, nothing else) and something appears to have written a partition table to your drive, wiping the mdadm metadata in the process (this is why you should use partitions, instead of full disks, in the first place - too many programs expect partition table to exist and even create it for you if it doesn't). Commented Aug 26, 2022 at 19:57
  • Thanks for your input! So I guess my best option would be to downgrade the array to two disks again? Due to lack of a spare HDD I would try to do the following now: 1) downgrade the array, 2) use the spare disk (/dev/sdb) to save/backup the data, 3) partition /dev/sdc and /dev/sdd, 4) recreate the array with 2 drives, 5) copy the data back onto it, and 6) add the third (partitioned) disk.
    – KenAdams
    Commented Aug 26, 2022 at 20:30
  • You should be able to fail sdb and still have the raid5 start. You can then wipe sdb and re-add it. But that doesn't address the issue of something overwriting the front (or end) of the disk with a partition table or something Commented Aug 26, 2022 at 21:02
  • I'd migrate to partitions. By adjusting mdadm data offset, this could be done even without moving data. If you want to proceed w/o partitions, use wipefs to wipe the partition table at both start and end of disk, then hope for the best or find out what is creating that partition table against your will. sometimes, even the bios. if you're dual booting, then windows. in linux, could be livecd installers, (g)parted, or any other tool. its too easy to create partition table by accident, which makes using raw drives risky Commented Aug 26, 2022 at 21:13

1 Answer 1

1

For anyone having the same issue:

It seems that my problem was indeed the missing partition on \dev\sdb, although I never found out what caused the partition table to be corrupt/gone/etc.

I wasn't too happy about doing something with no backup of my data, so I did what I proposed in my comment:

  1. Downgrade the array
  2. Use the spare ("failing") disk (/dev/sdb) to save/backup the data
  3. Partition the two remaining drives /dev/sdc and /dev/sdd (and destroying the array!)
  4. Recreate the array with 2 drives
  5. Copy the data back onto it
  6. Add the third (partitioned) disk

Thanks again for pointing me in the right direction!

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .