CentOS 7 created mdadm array disappears after reboot

Question

I created a raid1 using the below disks and command:

$ ls -l /dev/disk/by-id/ata-ST3000*
lrwxrwxrwx 1 root root 9 Sep 19 07:27 /dev/disk/by-id/ata-ST3000DM001-9YN166_Z1F04NR1 -> ../../sdc
lrwxrwxrwx 1 root root 9 Sep 19 07:27 /dev/disk/by-id/ata-ST3000DM001-9YN166_Z1F190E3 -> ../../sda

$ mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/disk/by-id/ata-ST3000DM001-9YN166_Z1F190E3 /dev/disk/by-id/ata-ST3000DM001-9YN166_Z1F04NR1

I added the pertinent information to mdadm.conf. I used 'mdadm --detail --scan >> /etc/mdadm.conf' for the ARRAY line:

$ cat /etc/mdadm.conf
DEVICE /dev/disk/by-id/ata-ST3000DM001-9YN166_Z1F190E3
DEVICE /dev/disk/by-id/ata-ST3000DM001-9YN166_Z1F04NR1
ARRAY /dev/md0 metadata=1.2 name=jaime.WORKGROUP:0 UUID=93f2cb73:2d124630:562f1dd9:bf189029
MAILADDR your@address

I created and mounted the filesystem:

$ mkfs -t xfs /dev/md0
$ mount -t xfs /dev/md0 /data

After rebooting, /dev/md0 no longer exists and I can't assemble the array:

$ mdadm --assemble /dev/md0 /dev/disk/by-id/ata-ST3000DM001-9YN166_Z1F190E3 /dev/disk/by-id/ata-ST3000DM001-9YN166_Z1F04NR1
mdadm: Cannot assemble mbr metadata on /dev/disk/by-id/ata-ST3000DM001-9YN166_Z1F190E3
mdadm: /dev/disk/by-id/ata-ST3000DM001-9YN166_Z1F190E3 has no superblock - assembly aborted



$ blkid
/dev/sda: PTTYPE="gpt"
/dev/sdb1: UUID="5c7d3f2b-c975-46a3-a116-e9fc156c1de5" TYPE="xfs"
/dev/sdb2: UUID="JhoqjI-N6R6-O9zt-Xumq-TnFX-OUCd-Lg9YHy" TYPE="LVM2_member"
/dev/sdc: PTTYPE="gpt"
/dev/mapper/centos-swap: UUID="3b882d4d-b900-4c59-9912-60a413699db4" TYPE="swap"
/dev/mapper/centos-root: UUID="08df953d-d4f4-4e83-bf4b-41f14a98a12e" TYPE="xfs"
/dev/mapper/centos-home: UUID="2358f723-5e7f-49ed-b207-f32fe34b1bbc" TYPE="xfs"

Answer 1

For all you folks, I have found a workaround that works for me.. May be this may help you guys.. Here it is:

Raid mount disappears because the system is not reading the mdadm.conf file during boot or startup. So, what I did is I edited the /etc/rc.d/rc.local file to include the below commands:

sleep 10
mdadm --assemble --scan
sleep 10
mount -a

Now, everytime I reboot the system, it reads this file, run the commands mentioned in it and mount the raid mount..

Answer 2

In theory, one can make raids out of "bare drives" (non-partitioned), but I noticed your disks are showing up as gpt-partitioned, not md drives. In general, I've found better success / stability by partitioning my disk, and then using partitions in my md arrays.

I'd try creating a partition table, setting the partition type as linux raid autodetect (fd in fdisk if I recall correctly). Then recreating your array.

Also, I found that if I did NOT use an mdadm.conf, I encountered better success. Modern versions of md tools will get all the information they need from the superblocks of the partitions involved.

Answer 3

I came across this question which helped me a lot during troubleshooting. But none of the answers could solve my problem.

So maybe this helps someone having the same issue. In my case I have two NVME drives on a RedHat Linux 7.2.

SELinux has a problem with nvme and prevents mdadm to work with them.

Symptoms:

Create software RAID as described in the question of this thread. After a reboot the RAID is gone:

• /proc/mdstat is empty and /dev/md0 does not exist
• mdadm --assemble --scan brings the RAID up.

Solution:

I disabled selinux in my case. I know this is not possible on every system. But maybe you get in the right direction from my post.

Answer 4

The output of blkid shows you your problem - the drives weren't clean when you created them md device, and the old information is confusing things.

Given that you created these without a partition table, the fact that blkid reports PTTYPE="gpt" for sda and sdc is the problem.

If you do not (yet) have any data on the drives that you need to save, you can wipe out the GPT partition table with 'dd':

$ sudo dd if=/dev/zero of=/dev/sda bs=1M count=10

This will zero out the first 10 megabytes of /dev/sda. Repeat for /dev/sdc. Recreate the array after that, and then /dev/md0 should come up on boot.

Answer 5

It seems to be a better way to format the disk partitions with type fd (Linux raid autodetect), but watch for alignment messages in fdisk! I had those warnings and resolved them with then parted tool for creating the partitions and then using fdisk to set them to type 'fd'. Then do exactly as you did and the RAID device will form automatically at boot time. With an entry in /etc/fstab it will even be mounted...

One caveat: Using /etc/mdadm.conf destroyed my RHEL7 3.10.0-514.6.1.el7.x86_6 system.

Answer 6

Usual caveats apply. Make sure your data was backed up. I'm not responsible for any data loss. This worked for me. Make sure to test your array before you put meaningful data on it. Make sure it can survive a reboot and device name changes, etc etc etc.

Now off to what I did.

I ran into the same issue today. After doing some searching I was able to fix this and get a stable raid device.

On two disks I was using for my array I created a partition using gdisk before I created the RAID device. I deleted the partitions using gdisk but when I created the array I got the informational message below:

mdadm: partition table exists on /dev/disk/by-id/ata-ST4000VN000-1H4168_Z30254QX but will be lost or meaningless after creating array

Each time I saw this message when I rebooted the array would be lost.

I figured I'd use Jim K's above suggestion and create a Linux raid partition on each drive. This worked but the sync rate went from 145MB/s on the raw drives to 35MB/s on the partitioned ones. I probably could have played with tuning parameters but why should I do that?

I went on a quest to figure out how to nuke the drive partition table.

The fix...

First make sure your raid device is stopped , then nuke the drives raid signature.

mdadm --stop /dev/md<number> 
mdadm --zero-superblock /dev/sd<x> (do this for all your raid disks)

Run blkid and see if any of the raid devices show up. If they do.

run:

dd if=/dev/zero of=/dev/sd<x> bs=512 count=1

This will hopefully nuke the partition tables. Run blkid again to make sure no disks used for the raid device are listed.

In my case gpt had a LVM label backed up on one disk. I cleaned this up using lvremove, vgremove, and finally pvremove.

blkid ran clean this time.

However when I tried creating the array again I got an error about an existing raid device so I did a dd one more time. After this the array created without notices like I listed above.

I recreated the array as it was originally, I used disk/by-id instead of sd. Once the array was recreated the test partition I had created on the array came back intact.

The array now survives reboots and changes to /dev/sd from when the array was originally created.

Answer 7

I know it's an old post, but I was struggling with this issue and this is my result:

My disks were "frozen" - Seagate disks. You can check, if you have same issue by entering command:

hdparm -I /dev/sdb

Which showed:

Security: 
Master password revision code = 65534
    supported
not enabled
not locked
    frozen
not expired: security count
    supported: enhanced erase

I wasn't able to change this setting. The disks worked fine with regular partitions, but when I was formatting them as linux raid, they lost they partition table and were "empty" after reboot.

I created raid on partitions, not on devices:

mdadm --create --verbose /dev/md1 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1

And now they are fine after reboot and everything works as expected.