Can't remount local file systems for read-write (RAID1)

Question

I assembled a new machine on Asus P9X79 motherboard, using its RAID controller to create a RAID1 array of two 500Gb drives.

When booting Arch from an external drive I am able to work fine with /dev/md126 which corresponds to the array. This way I created the partitions and filesystems on it, then chrooted and installed Arch Linux on the drive.

However, I am not able to boot from RAID successfully: /, /boot and /home cannot be remounted in read-write mode (mount returns 32), and I end up in emergency console. Trying to remount from there also fails, saying the drive is write-protected.

I figured that some necessary kernel modules were not loaded at boot and played with mkinitcpio.conf. I have mdadm_udev as a hook (after udev, before filesystems). To my understanding, this should be enough, but I also tried adding raid1, raid456, ext2 and ext4 to the MODULES array, this didn't change anything.

The RAID1 itself is recognized in the initial environment thanks to the mdadm_udev hook (the devices are there). raid1 is, I think, also loaded by this hook automatically.

I am still able to boot from external drive and mount the RAID1 device just fine; so I did lsmod on it and compared to lsmod on the "native" system. Nothing seems suspicious to me:

$ diff <(sort lsmod.old | cut -f1 -d ' ') <(sort lsmod.new | cut -f1 -d ' ')
7a8,12
> async_memcpy
> async_pq
> async_raid6_recov
> async_tx
> async_xor
13a19,20
> drm
> drm_kms_helper
24a32
> i2c_algo_bit
43c51
< nvidia
---
> nouveau
48a57,58
> raid456
> raid6_pq
64a75,76
> ttm
> uas
71a84
> xor

old is the one that works. As you can see, the only line starting with < is < nvidia. So all the necessary modules are loaded (some of the additional modules in new are dependencies of raid456 that I tried loading).

What am I missing here? What can be the possible differences between two systems? Kernel versions are the same: 3.6.8. (BTW the installation medium I first tried to use had 3.6.6 and didn't work with this array; all operations ended up hanging endlessly).

Answer 1

(Lev found the solution, I'm doing exegesis to explain why it works.)

using its RAID controller to create a RAID1 array

That's a bad sign: you're using fakeraid — a RAID implementation which is mostly implemented by the Windows driver with a little help from the firmware. You get all the downsides of hardware RAID (dependency on the firmware) with all the downsides of software RAID (no performance advantage).

The RAID metadata is handled by the firmware. (Metadata is the extra data that needs to be stored somewhere and that's not part of the filesystem or partition stored on the RAID device: things like where each block of data should be sorted, extra data to handle resynchronization and so on.) With Linux's implementation (at least for this driver), this is not handled by the kernel alone, the mdmon utility is also needed.

When your system boots, at first, there is only the kernel and an initial RAM drive (an initramfs). This initial RAM drive must contain all the loadable modules and programs that are necessary to mount the root filesystem. Because that has to fit in RAM, most distributions generate the initramfs on demand, based on the drivers that are necessary on your system. This is typically done on each kernel upgrade.

It seems that Arch Linux's initramfs generation scripts did not detect that you needed the mdmon program at boot time, and so generated a non-working initramfs. Forcing mdmon to be present in the initramfs made the initramfs work.

Answer 2

I got it working using the advice from here. Namely, I added /sbin/mdmon to the BINARIES array in mkinitcpio.conf.

This looks a little like black magic to me, so a detailed answer with explanations is by all means welcome.