8

I've reinstalled a Linux server from CentOS 6 to 7. The server has 3 drives - a system SSD drive (it hosts everything except /home) and two 4TB HDD drives that host /home. Everything uses LVM. The two 4TB drives are mirrored (using the raid option within LVM itself), and they are completely filled with the /home partition.

The problem is that although the 4TB disks are recognized fine, and LVM sees the volume without problems, it does not activate it automatically. Everything else is activated automatically. I can activate it manually, and it works.

I have an image of the old system drive in /home. That contains LVM volumes too. If I mount it with kpartx, and LVM picks those up and activates them. But I can see no difference between those volumes and the inactive ones.

The root filesystem is LVM too, and that activates just fine.

I see a peculiar thing though: executing lvchange -aay tells me that I need to specify which drives I want to activate. It doesn't do it automatically either. If I specify lvchange -ay lv_home - that works.

I cannot find anything that could be responsible for this behavior.

Added: I noticed that the old system (which used init) had vgchange -aay --sysinit in its startup scripts. The new one uses systemd, and I don't see the vgchange call in its scripts. But I also don't know where to put it.

Added 2: Starting to figure out systemd. I found where the scripts are located and started understanding how they are called. Also found that I could see the executed scripts with systemctl -al. This shows me that after starting lvmetad it calls pvscan for each known udev block device. However at that point there is just one registered udev block device, and that is one of the recognized lvm volumes. The hard drives are there too, but under different paths and much longer names. The recognized block device is something like 8:3, while the hard drives are like /device/something/. I'm not at the server anymore, so I cannot write it precisely (will fix this later).

I think that it has something to do with udev and device detection/mapping. I will continue in the evening and will study udev then.

If all else fails, I found the script that calls pvscan and checked that I can modify it to scan all the devices all the time. That fixes the problem, but it looks like a rather ugly hack, so I'll try to figure out the real root cause.

Added 3: OK, I still don't know why this happens, but at least I've made a fairly passable workaround. I made another systemd service that calls the pvscan once, right after starting lvmetad. The other call for the specific device is still there, and I think it's actually udev that calls it (that's the only place where I found reference to it). Why it doesn't call it for the other hard drives - I have no idea.

  • Are the LVM system services running? This happened to me. – Naftuli Kay Jul 1 '15 at 0:34
  • @NaftuliTzviKay - Yes, the services are starting fine (at least lvmetad is - I haven't noticed any others). – Vilx- Jul 1 '15 at 5:46
  • There was another service whose name evades me at the moment. – Naftuli Kay Jul 1 '15 at 5:48
  • @NaftuliTzviKay - Hmm... there was also some kind of "monitoring" service too. That starts fine as well, though I remember reading about what it does and coming to the conclusion that it doesn't apply to me. I don't have access to the box at the moment however, so I'll check back later. – Vilx- Jul 1 '15 at 7:56
7

I did it! I did it! I fixed it properly (I think).

Here's the story:

After some time the server turned out to be faulty and had to be scrapped. I kept disks and got everything else new. Then I reinstalled CentOS again on the SSD and then I attached the HDDs. LVM worked nicely, the disks were recognized, the configuration kept. But the same problem came up again - after a reboot, the volume was inactive.

However this time I chanced to notice something else - the bootloader passes the following parameters to the kernel:

crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet

Hmm, wait a minute, those look FAMILIAR!

Quick google query, and there we are:

rd.lvm.lv=

only activate the logical volumes with the given name. rd.lvm.lv can be specified multiple times on the kernel command line.

Well now. THAT explains it!

So, the resolution was (gathered from several more google queries):

  1. Modify /etc/defaults/grub to include the additional volume in the parameters: crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swaprd.lvm.lv=vg_home/lv_homerhgb quiet
  2. Reconfigure grub with grub2-mkconfig -o /boot/grub2/grub.cfg
  3. Reconfigure initramfs with mkinitrd -f -v /boot/initramfs-3.10.0-327.18.2.el7.x86_64.img 3.10.0-327.18.2.el7.x86_64. Note: your values may vary. Use uname -r to get that kernel version. Or just read up on mkinitrd. (Frankly, I don't know why this step is needed, but apparently it is - I tried without it and it didn't work)
  4. And finally, reinstall grub: grub2-install /dev/sda
  5. Reboot, naturally.

TA-DA! The volume is active on reboot. Add it to fstab and enjoy! :)

2

Minor update (for RHEL 7 on EFI (non-BIOS) machine):

I've got to success using these steps:

  1. Modify /etc/defaults/grub to include the additional volume in the parameters: rd.lvm.lv=rhel/home (in addition to rhel/root and rhel/swap)
  2. Reconfigure grub with

    grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
    

    (note: another path!)

  3. Reconfigure initramfs with

    mkinitrd -f -v /boot/initramfs-$(uname -r).img $(uname -r)
    
  4. Skip reinstall grub: grub2-install /dev/sda (because I have an empty dir /usr/lib/grub/)
  5. Reboot, naturally.
  • Hmm, looks like you're using EFI. In my case it was a BIOS box, so that's probably why the difference. – Vilx- Aug 31 '16 at 10:49
  • Yes, it is a x240 blade. I've added a note on EFI. – jno Aug 31 '16 at 11:45
  • @Vilx- There is a vendor proposed workaround for this bug. But it's really ugly. They propose to add _netdev flag to fstab, enable chkconfig netfs on and even switch off use_lvmetad = 0 the lvmetad in /etc/lvm/lvm.conf just in hope the devices will re-polled and taken up... – jno Aug 31 '16 at 11:50
  • Sorry, can't see it, since I'm not a RedHat customer. But I don't see why our solutions aren't correct and a different workaround is needed. I mean, this isn't a bug - everything's working as supposed to. – Vilx- Aug 31 '16 at 12:33
  • I think, it's a bug: root and swap are listed explicitly in kernel cmdline while home is not. Hence, we have two LVM vols mounted and one dangling in "inactive" state. I've quoted their proposal here exactly to avoid the need in specific access credentials :) – jno Aug 31 '16 at 13:43
1

I had this problem as well. In my case it was a combination of iscsi, multipath and lvm and the ordering of session creation etc. I solved the problem by adding a call to /sbin/vgchange -a y to /etc/rc.local.

0

So I tried the rd.lvm.lv= setting in /etc/default/grub and that did not work

I needed both logical volumes on the ssd_vg volume group to be active on boot. As well as the logical volume home_lv on the kubuntu-vg to be active

What did work was to edit /etc/lvm/lvm.conf In the volume list section put this in volume_list = [ "ssd_vg", "kubuntu-vg/home_lv" ]

result after rebooting

$ sudo lvscan inactive Original '/dev/kubuntu-vg/root' [50.00 GiB] inherit

inactive          '/dev/kubuntu-vg/swap_1' [7.88 GiB] inherit

ACTIVE            '/dev/kubuntu-vg/home_lv' [1000.00 GiB] inherit

inactive Snapshot '/dev/kubuntu-vg/root_snap11' [50.00 GiB] inherit

inactive Snapshot '/dev/kubuntu-vg/root_snap12' [50.00 GiB] inherit

ACTIVE            '/dev/ssd_vg/root' [224.02 GiB] inherit

ACTIVE            '/dev/ssd_vg/swap_1' [7.88 GiB] inherit
0

For my part, I have comment this line in /etc/lvm/lvm.conf

auto_activation_volume_list = [ "vg00", "vg01" ]

Because if its active, only volume vg00 and vg01 are active on boot.

The documentation of lvm.conf :

If auto_activation_volume_list is defined, each LV that is to be
activated with the autoactivation option (--activate ay/-a ay) is
first checked against the list. There are two scenarios in which
the autoactivation option is used:

  - automatic activation of volumes based on incoming PVs. If all the
    PVs making up a VG are present in the system, the autoactivation
    is triggered. This requires lvmetad (global/use_lvmetad=1) and udev
    to be running. In this case, "pvscan --cache -aay" is called
    automatically without any user intervention while processing
    udev events. Please, make sure you define auto_activation_volume_list
    properly so only the volumes you want and expect are autoactivated.

  - direct activation on command line with the autoactivation option.
    In this case, the user calls "vgchange --activate ay/-a ay" or
    "lvchange --activate ay/-a ay" directly.

By default, the auto_activation_volume_list is not defined and all
volumes will be activated either automatically or by using --activate ay/-a ay.

N.B. The "activation/volume_list" is still honoured in all cases so even
if the VG/LV passes the auto_activation_volume_list, it still needs to
pass the volume_list for it to be activated in the end.

If auto_activation_volume_list is defined but empty, no volumes will be
activated automatically and --activate ay/-a ay will do nothing.

auto_activation_volume_list = []

If auto_activation_volume_list is defined and it's not empty, only matching
volumes will be activated either automatically or by using --activate ay/-a ay.

  "vgname" and "vgname/lvname" are matched exactly.
  "@tag" matches any tag set in the LV or VG.
  "@*" matches if any tag defined on the host is also set in the LV or VG


Only activate vg00 and vg01 automatically.
auto_activation_volume_list = [ "vg00", "vg01" ]

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.