LVM volume is inactive after reboot of CentOS

Question

I've reinstalled a Linux server from CentOS 6 to 7. The server has 3 drives - a system SSD drive (it hosts everything except /home) and two 4TB HDD drives that host /home. Everything uses LVM. The two 4TB drives are mirrored (using the raid option within LVM itself), and they are completely filled with the /home partition.

The problem is that although the 4TB disks are recognized fine, and LVM sees the volume without problems, it does not activate it automatically. Everything else is activated automatically. I can activate it manually, and it works.

I have an image of the old system drive in /home. That contains LVM volumes too. If I mount it with kpartx, and LVM picks those up and activates them. But I can see no difference between those volumes and the inactive ones.

The root filesystem is LVM too, and that activates just fine.

I see a peculiar thing though: executing lvchange -aay tells me that I need to specify which drives I want to activate. It doesn't do it automatically either. If I specify lvchange -ay lv_home - that works.

I cannot find anything that could be responsible for this behavior.

Added: I noticed that the old system (which used init) had vgchange -aay --sysinit in its startup scripts. The new one uses systemd, and I don't see the vgchange call in its scripts. But I also don't know where to put it.

Added 2: Starting to figure out systemd. I found where the scripts are located and started understanding how they are called. Also found that I could see the executed scripts with systemctl -al. This shows me that after starting lvmetad it calls pvscan for each known udev block device. However at that point there is just one registered udev block device, and that is one of the recognized lvm volumes. The hard drives are there too, but under different paths and much longer names. The recognized block device is something like 8:3, while the hard drives are like /device/something/. I'm not at the server anymore, so I cannot write it precisely (will fix this later).

I think that it has something to do with udev and device detection/mapping. I will continue in the evening and will study udev then.

If all else fails, I found the script that calls pvscan and checked that I can modify it to scan all the devices all the time. That fixes the problem, but it looks like a rather ugly hack, so I'll try to figure out the real root cause.

Added 3: OK, I still don't know why this happens, but at least I've made a fairly passable workaround. I made another systemd service that calls the pvscan once, right after starting lvmetad. The other call for the specific device is still there, and I think it's actually udev that calls it (that's the only place where I found reference to it). Why it doesn't call it for the other hard drives - I have no idea.

Answer 1

I did it! I did it! I fixed it properly (I think).

Here's the story:

After some time the server turned out to be faulty and had to be scrapped. I kept disks and got everything else new. Then I reinstalled CentOS again on the SSD and then I attached the HDDs. LVM worked nicely, the disks were recognized, the configuration kept. But the same problem came up again - after a reboot, the volume was inactive.

However this time I chanced to notice something else - the bootloader passes the following parameters to the kernel:

crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet

Hmm, wait a minute, those look FAMILIAR!

Quick google query, and there we are:

rd.lvm.lv=

only activate the logical volumes with the given name. rd.lvm.lv can be specified multiple times on the kernel command line.

Well now. THAT explains it!

So, the resolution was (gathered from several more google queries):

1. Modify /etc/defaults/grub to include the additional volume in the parameters: crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rd.lvm.lv=vg_home/lv_home rhgb quiet
2. Reconfigure grub with grub2-mkconfig -o /boot/grub2/grub.cfg
3. Reconfigure initramfs with mkinitrd -f -v /boot/initramfs-3.10.0-327.18.2.el7.x86_64.img 3.10.0-327.18.2.el7.x86_64. Note: your values may vary. Use uname -r to get that kernel version. Or just read up on mkinitrd. (Frankly, I don't know why this step is needed, but apparently it is - I tried without it and it didn't work)
4. And finally, reinstall grub: grub2-install /dev/sda
5. Reboot, naturally.

TA-DA! The volume is active on reboot. Add it to fstab and enjoy! :)

Answer 2

I had this problem as well. In my case it was a combination of iscsi, multipath and lvm and the ordering of session creation etc. I solved the problem by adding a call to /sbin/vgchange -a y to /etc/rc.local.

Answer 3

Minor update (for RHEL 7 on EFI (non-BIOS) machine):

I've got to success using these steps:

1. Modify /etc/defaults/grub to include the additional volume in the parameters: rd.lvm.lv=rhel/home (in addition to rhel/root and rhel/swap)

2. Reconfigure grub with

   grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
   

   (note: another path!)

3. Reconfigure initramfs with

   mkinitrd -f -v /boot/initramfs-$(uname -r).img $(uname -r)
   

4. Skip reinstall grub: grub2-install /dev/sda (because I have an empty dir /usr/lib/grub/)
5. Reboot, naturally.

Answer 4

During the installation and configuration of Gentoo Linux, I ran into a similar issue. The root LVM (disk1) was recognized without any issue, but all logical volumes from the other disk (disk2) were inactive after reboot.

I tried to change the parameters that are passed to the kernel (as described in the topic), a script that should activate the volumes by means of vgchange -a y or lvchange -a y vg1 on boot -- nothing helped.

After reading the comments (about activation) in the /etc/lvm/lvm.conf configuration file, I realized that lvm should do all these things automatically, i.e. it should activate the volumes, but it doesn't.

After this, I realized what is the issue: lvm wasn't configured to start on boot. The logical volumes started to initialize (to mount, etc) without any issue after I installed it in runlevel boot:

$ sudo rc-update add lvm boot

Answer 5

So I tried the rd.lvm.lv= setting in /etc/default/grub and that did not work

I needed both logical volumes on the ssd_vg volume group to be active on boot. As well as the logical volume home_lv on the kubuntu-vg to be active

What did work was to edit /etc/lvm/lvm.conf In the volume list section put this in volume_list = [ "ssd_vg", "kubuntu-vg/home_lv" ]

result after rebooting

$ sudo lvscan inactive Original '/dev/kubuntu-vg/root' [50.00 GiB] inherit

inactive          '/dev/kubuntu-vg/swap_1' [7.88 GiB] inherit

ACTIVE            '/dev/kubuntu-vg/home_lv' [1000.00 GiB] inherit

inactive Snapshot '/dev/kubuntu-vg/root_snap11' [50.00 GiB] inherit

inactive Snapshot '/dev/kubuntu-vg/root_snap12' [50.00 GiB] inherit

ACTIVE            '/dev/ssd_vg/root' [224.02 GiB] inherit

ACTIVE            '/dev/ssd_vg/swap_1' [7.88 GiB] inherit

Answer 6

For my part, I have comment this line in /etc/lvm/lvm.conf

auto_activation_volume_list = [ "vg00", "vg01" ]

Because if its active, only volume vg00 and vg01 are active on boot.

The documentation of lvm.conf :

If auto_activation_volume_list is defined, each LV that is to be
activated with the autoactivation option (--activate ay/-a ay) is
first checked against the list. There are two scenarios in which
the autoactivation option is used:

  - automatic activation of volumes based on incoming PVs. If all the
    PVs making up a VG are present in the system, the autoactivation
    is triggered. This requires lvmetad (global/use_lvmetad=1) and udev
    to be running. In this case, "pvscan --cache -aay" is called
    automatically without any user intervention while processing
    udev events. Please, make sure you define auto_activation_volume_list
    properly so only the volumes you want and expect are autoactivated.

  - direct activation on command line with the autoactivation option.
    In this case, the user calls "vgchange --activate ay/-a ay" or
    "lvchange --activate ay/-a ay" directly.

By default, the auto_activation_volume_list is not defined and all
volumes will be activated either automatically or by using --activate ay/-a ay.

N.B. The "activation/volume_list" is still honoured in all cases so even
if the VG/LV passes the auto_activation_volume_list, it still needs to
pass the volume_list for it to be activated in the end.

If auto_activation_volume_list is defined but empty, no volumes will be
activated automatically and --activate ay/-a ay will do nothing.

auto_activation_volume_list = []

If auto_activation_volume_list is defined and it's not empty, only matching
volumes will be activated either automatically or by using --activate ay/-a ay.

  "vgname" and "vgname/lvname" are matched exactly.
  "@tag" matches any tag set in the LV or VG.
  "@*" matches if any tag defined on the host is also set in the LV or VG


Only activate vg00 and vg01 automatically.
auto_activation_volume_list = [ "vg00", "vg01" ]