1

Yesterday I upgraded my CentOS 8 server from a

Raid 1, 2 disk setup

to a

Raid 5, 4 disk setup using mdadm

However after the final stage of the upgrade:

mdadm --grow /dev/md/pv00 -n 4

and some time after multiple times checking

cat /proc/mdstat

for progress on the Raid 5 setup (which was around 4/5%)

there was a power outage :(

When the power came back on, CentOS 8 wouldn't start anymore except after a long time to emergency mode.

The error given:

md: personality for level 5 is not loaded

failed to boot with md level 5 error

In this emergency mode, I cannot mdadm assemble the Raid as it errors with a wrong raid level issue. has wrong raid level

Trying to exit dracut emergency mode I see: dracut exit not all disks found

After a lot of Googling around, I finally created an Ubuntu 21.04 live bootable USB so I could maybe see more info.

Turns out, I finally could mdadm assembly the 4 drives again in the Ubuntu environment after which the raid system itself started to recover and continue the Raid5 process.

Finally after several hours this was done and after mounting the various volumes on the Raid system and seeing that everything was fine and finished, I rebooted.

Same problem..

Trying the same thing with a FedoraCore Workstation live boot usb, I could again simply mount the raid5 disks.

Then I went into CentOS Rescue mode, which I selected in Grub and there the Raid5 4 disks comes online just fine! very strange

I only see 1 error which is /boot won't mount (can't seem to find if this is normal because of Rescue mode?) EDIT: this is not normal! Error I get when trying to manually mount /boot:

mount /dev/sda1 /boot

mount: /boot: unknown filesystem type 'ext4'.

and lsmod does indeed show no ext4 loaded

but I can see the volumes as normal and cat /proc/mdstat outputs normal active information.

I don't know how to resolve this as Googling around shows me I might have to do some repair to the boot img, but don't know how to mount /boot in Repair mode and Emergency mode doesn't seem to have anything I can work with.

So basically I have a OS that won't start in normal mode, while the Raid disks seem to work just fine.

I tried manually editing the /etc/mdadm.conf file but that didn't work either on reboot and found out online it's just the assemble point.

Thank you very much for your help!

Details

CentOS Linux release 8.4.2105

Kernel version: 4.18.0-305.7.1

From Fedora Workstation Live 34:

cat /proc/mdstat

Personalities : [raid6] [raid5] [raid4] [raid0] [raid1]
md127 : active raid5 sdc[2] sdd[3] sdb1[1] sda2[0]
      2926740480 blocks super 1.2 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      bitmap: 0/8 pages [0KB], 65536KB chunk

unused devices: <none>

without any non-useful disks/volumes:

fdisk -l

Disk /dev/sda: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: ST1000NM0033-9ZM
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xd393f4b7

Device     Boot   Start        End    Sectors   Size Id Type
/dev/sda1  *       2048    2099199    2097152     1G 83 Linux
/dev/sda2       2099200 1953523711 1951424512 930.5G fd Linux raid autodetect


Disk /dev/sdb: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: ST1000NM0033-9ZM
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xf0f126b5

Device     Boot Start        End    Sectors   Size Id Type
/dev/sdb1        2048 1951426559 1951424512 930.5G fd Linux raid autodetect


Disk /dev/sdc: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: ST1000NM0033-9ZM
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdd: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: ST1000NM0033-9ZM
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/md127: 2.73 TiB, 2996982251520 bytes, 5853480960 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 196608 bytes

I commented out the ARRAY hoping the conf would pick up on the level and num-devices changes, but nope This is from the root volume of the RAID mount:

cat mdadm.conf

# mdadm.conf written out by anaconda
MAILADDR root
AUTO +imsm +1.x -all
#ARRAY /dev/md/pv00 level=raid1 num-devices=2 UUID=5b729889:1b231f26:6806a14c:71abe309
ARRAY /dev/md/pv00 level=raid5 num-devices=4 UUID=5b729889:1b231f26:6806a14c:71abe309

pvdisplay

  --- Physical volume ---
  PV Name               /dev/md127
  VG Name               cl_nas
  PV Size               <930.39 GiB / not usable 3.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              238178
  Free PE               0
  Allocated PE          238178
  PV UUID               RpepyF-BxMl-9BZT-Sebw-otBV-HLCq-7XSGWX

vgdisplay

  --- Volume group ---
  VG Name               cl_nas
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               930.38 GiB
  PE Size               4.00 MiB
  Total PE              238178
  Alloc PE / Size       238178 / 930.38 GiB
  Free  PE / Size       0 / 0
  VG UUID               NrW8lm-Xi70-hP73-Auis-wLVy-beHq-85TWZM

lvdisplay

  --- Logical volume ---
  LV Path                /dev/cl_nas/home
  LV Name                home
  VG Name                cl_nas
  LV UUID                VXGWg5-MnHI-1yg8-5rez-1tIa-Aa2f-s0h2K3
  LV Write Access        read/write
  LV Creation host, time nas.localdomain, 2020-10-28 10:45:39 +0100
  LV Status              available
  # open                 1
  LV Size                <892.51 GiB
  Current LE             228482
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     768
  Block device           253:2

  --- Logical volume ---
  LV Path                /dev/cl_nas/swap
  LV Name                swap
  VG Name                cl_nas
  LV UUID                49Aech-0lGd-TbVx-2Hy7-n18b-EG75-4tOBhd
  LV Write Access        read/write
  LV Creation host, time nas.localdomain, 2020-10-28 10:45:46 +0100
  LV Status              available
  # open                 0
  LV Size                <7.88 GiB
  Current LE             2016
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     768
  Block device           253:3

  --- Logical volume ---
  LV Path                /dev/cl_nas/root
  LV Name                root
  VG Name                cl_nas
  LV UUID                V5WLBp-lvkd-aGT4-isoE-U8UH-ehqG-i37BmE
  LV Write Access        read/write
  LV Creation host, time nas.localdomain, 2020-10-28 10:45:47 +0100
  LV Status              available
  # open                 1
  LV Size                30.00 GiB
  Current LE             7680
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     768
  Block device           253:4

(without /dev/sde which is the flash drive)

mdadm --examine /dev/sd*

/dev/sda:
   MBR Magic : aa55
Partition[0] :      2097152 sectors at         2048 (type 83)
Partition[1] :   1951424512 sectors at      2099200 (type fd)
mdadm: No md superblock detected on /dev/sda1.
/dev/sda2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 5b729889:1b231f26:6806a14c:71abe309
           Name : nas.localdomain:pv00
  Creation Time : Wed Oct 28 10:45:31 2020
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 1951160704 (930.39 GiB 998.99 GB)
     Array Size : 2926740480 (2791.16 GiB 2996.98 GB)
  Used Dev Size : 1951160320 (930.39 GiB 998.99 GB)
    Data Offset : 263808 sectors
   Super Offset : 8 sectors
   Unused Space : before=263728 sectors, after=384 sectors
          State : clean
    Device UUID : 33ff95f7:95ae0fb4:bf45c073:0e248f5f

Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Jul  9 21:45:11 2021
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 9c0e4536 - correct
         Events : 19835

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 0
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb:
   MBR Magic : aa55
Partition[0] :   1951424512 sectors at         2048 (type fd)
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 5b729889:1b231f26:6806a14c:71abe309
           Name : nas.localdomain:pv00
  Creation Time : Wed Oct 28 10:45:31 2020
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 1951160704 (930.39 GiB 998.99 GB)
     Array Size : 2926740480 (2791.16 GiB 2996.98 GB)
  Used Dev Size : 1951160320 (930.39 GiB 998.99 GB)
    Data Offset : 263808 sectors
   Super Offset : 8 sectors
   Unused Space : before=263728 sectors, after=384 sectors
          State : clean
    Device UUID : 5c027c56:0b6e4a5c:26a56c03:5dfb6c53

Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Jul  9 21:45:11 2021
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 26b93e8b - correct
         Events : 19835

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 1
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 5b729889:1b231f26:6806a14c:71abe309
           Name : nas.localdomain:pv00
  Creation Time : Wed Oct 28 10:45:31 2020
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 1953261360 (931.39 GiB 1000.07 GB)
     Array Size : 2926740480 (2791.16 GiB 2996.98 GB)
  Used Dev Size : 1951160320 (930.39 GiB 998.99 GB)
    Data Offset : 263808 sectors
   Super Offset : 8 sectors
   Unused Space : before=263728 sectors, after=2101040 sectors
          State : clean
    Device UUID : 1e731fa9:febb6375:f9f05cec:8dd1c8e5

Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Jul  9 21:45:11 2021
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : de22cf6 - correct
         Events : 19835

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 5b729889:1b231f26:6806a14c:71abe309
           Name : nas.localdomain:pv00
  Creation Time : Wed Oct 28 10:45:31 2020
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 1953261360 (931.39 GiB 1000.07 GB)
     Array Size : 2926740480 (2791.16 GiB 2996.98 GB)
  Used Dev Size : 1951160320 (930.39 GiB 998.99 GB)
    Data Offset : 263808 sectors
   Super Offset : 8 sectors
   Unused Space : before=263728 sectors, after=2101040 sectors
          State : clean
    Device UUID : 54ceea17:4b06f020:5a287029:9f7928dd

Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Jul  9 21:45:11 2021
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 5cacb1eb - correct
         Events : 19835

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 2
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
1

What ultimately did the trick was the following:

Made a bootable CentOS 8 USB so that I had about the same kernel loaded as was on the boot volume

Then start up and go to Troubleshooting -> Rescue

For me, it wouldn't automatically load/detect any drives/volumes so I ultimately dropped into the shell

there using

mdadm --assemble --scan -v

I was able to load the Raid system and then started to mount the filesystems as explained here:

https://wiki.centos.org/TipsAndTricks/CreateNewInitrd

First loaded the root from Raid to /mnt/sysimage Then the boot from ext4 disk to /mnt/sysimage/boot Then the other things as mentioned on the site and finally also mounted home to /mnt/sysimage/home

After this I dropped into chroot again as mentioned

and from there made a backup of the boot/initramfs*.img's and generated new ones using

for the bootable usb loaded kernel:

dracut -H -f

and specifically the latest kernel I normally run

dracut -H -f /boot/initramfs(kernel).img (kernel)

Got the command from: mdadm raid doesn't mount

Finally I rebooted and magically after 2 days of Googling and frustration it worked again :)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.