lvm devices under /dev/mapper missing

Question

I'm using Debian squeeze, and running LVM on top of software RAID 1. I just accidentally just discovered that most of the links under /dev/mapper are missing, though my system seems to be still functioning correctly.

I'm not sure what happened. The only thing I can imagine that caused it was my failed attempt to get a LXC fedora container to work. I ended up deleting a directory /cgroup/laughlin, corresponding to the container, but I can't imagine why that should have caused the problem. /dev/mapper looked (I made some changes, see below) approximately like

orwell:/dev/mapper# ls -la
total 0
drwxr-xr-x  2 root root     540 Apr 12 05:08 .
drwxr-xr-x 22 root root    4500 Apr 12 05:08 ..
crw-------  1 root root  10, 59 Apr  8 10:32 control
lrwxrwxrwx  1 root root       7 Mar 29 08:28 debian-root -> ../dm-0
lrwxrwxrwx  1 root root       8 Apr 12 03:32 debian-video -> ../dm-23

debian-video corresponds to a LV I had just created.

However, I have quite a number of VGs on my system, corresponding to 4 VGs spread across 4 disks. vgs gives

orwell:/dev/mapper# vgs
  VG         #PV #LV #SN Attr   VSize   VFree  
  backup       1   2   0 wz--n- 186.26g  96.26g
  debian       1   7   0 wz--n- 465.76g 151.41g
  olddebian    1  12   0 wz--n- 186.26g  21.26g
  testdebian   1   3   0 wz--n- 111.75g  34.22g

I tried running

 /dev/mapper# vgscan --mknodes

and some devices were created (see output below), but they aren't symbolic links to the dm devices as they should be, so I'm not sure if this is useless or worse. Would they get in the way of recreation of the correct links? Should I delete these devices again?

I believe that udev creates these links, so would a reboot fix this problem, or would I get an unbootable system? What should I do to fix this? Are there any diagnostics/sanity checks I should run to make sure there aren't other problems I haven't noticed? Thanks in advance for any assistance.

orwell:/dev/mapper# ls -la
total 0
drwxr-xr-x  2 root root     540 Apr 12 05:08 .
drwxr-xr-x 22 root root    4500 Apr 12 05:08 ..
brw-rw----  1 root disk 253,  1 Apr 12 05:08 backup-local_src
brw-rw----  1 root disk 253,  2 Apr 12 05:08 backup-video
crw-------  1 root root  10, 59 Apr  8 10:32 control
brw-rw----  1 root disk 253, 15 Apr 12 05:08 debian-boot
brw-rw----  1 root disk 253, 16 Apr 12 05:08 debian-home
brw-rw----  1 root disk 253, 22 Apr 12 05:08 debian-lxc_laughlin
brw-rw----  1 root disk 253, 21 Apr 12 05:08 debian-lxc_squeeze
lrwxrwxrwx  1 root root       7 Mar 29 08:28 debian-root -> ../dm-0
brw-rw----  1 root disk 253, 17 Apr 12 05:08 debian-swap
lrwxrwxrwx  1 root root       8 Apr 12 03:32 debian-video -> ../dm-23
brw-rw----  1 root disk 253, 10 Apr 12 05:08 olddebian-etch_template
brw-rw----  1 root disk 253, 13 Apr 12 05:08 olddebian-fedora
brw-rw----  1 root disk 253,  8 Apr 12 05:08 olddebian-feisty
brw-rw----  1 root disk 253,  9 Apr 12 05:08 olddebian-gutsy
brw-rw----  1 root disk 253,  4 Apr 12 05:08 olddebian-home
brw-rw----  1 root disk 253, 11 Apr 12 05:08 olddebian-lenny
brw-rw----  1 root disk 253,  7 Apr 12 05:08 olddebian-msi
brw-rw----  1 root disk 253,  5 Apr 12 05:08 olddebian-oldchresto
brw-rw----  1 root disk 253,  3 Apr 12 05:08 olddebian-root
brw-rw----  1 root disk 253, 14 Apr 12 05:08 olddebian-suse
brw-rw----  1 root disk 253,  6 Apr 12 05:08 olddebian-vgentoo
brw-rw----  1 root disk 253, 12 Apr 12 05:08 olddebian-wsgi
brw-rw----  1 root disk 253, 20 Apr 12 05:08 testdebian-boot
brw-rw----  1 root disk 253, 18 Apr 12 05:08 testdebian-home
brw-rw----  1 root disk 253, 19 Apr 12 05:08 testdebian-root

Answer 1

I just had a similar problem to what you describe, though for me it happened when I was attempting to install the new Ubuntu 11.10 Oneiric Ozelot to an LVM volume. I had done the following to set lvm up on a live boot system (the logical volumes I needed were already present):

apt-get install lvm2
vgscan --mknodes -v

Now lvscan -v showed my volumes but they were not in /dev/mapper nor in /dev/<vg>/. I finally found that I needed to activate the volume group, like so:

vgchange -a y <name of volume group>

The above command created all the missing device files for me. Now I could start the install program and it would find the lvm volumes and allow me to install to them.

Finding this information on google was hard, so I write this answer in the hope that others will have an easier time of it -- hence the in-depth context and namedropping.

Though not part of the question, for completeness I'll add that in the above situation (Ubuntu LVM install) you need to add lvm2 to the initrd of the newly installed system once it's done installing, or it won't boot. Your new system should be set up for you in /target, but if it is not, do it manually like this:

mount /dev/vg/new_root /target
mount /dev/sdx1 /target/boot       # important
mount -o bind /proc /target/proc
mount -o bind /sys /target/sys
mount -o bind /dev /target/dev
mount -o bind /dev/pts /target/dev/pts

I needed to do this to make networking work in the chroot, which I'll come to next:

cp /etc/resolv.conf /target/etc/

Now chroot to the new system and install lvm2:

chroot /target
apt-get install lvm2

Notice that it runs update-initramfs. Now just type exit and reboot, and your system should boot properly.

Answer 2

These days /dev is on tmpfs and is created from scratch each boot by udev. You can safely reboot and these links will come back.

You should also find LVM symlinks to the /dev/dm-X nodes in the /dev/<vg> directories, one directory for each volume group. However, those nodes re-created by vgscan --mknodes will also work fine, assuming they have the right major/minor numbers - and it's a safe assumption they were created properly.

You can probably also get udev to re-create the symlinks using udevadm trigger with an appropriate match, testing with --dry-run until it is right. It hardly seems worth the effort though when a reboot will fix it too.

Answer 3

This also worked for me.

vgchange -a y -name of volume group-

Following a kernel patch my RHEL system failed to reboot. Complaining of a missing /dev/mapper/VG-lv file.

Booted to single-user and commented out of /etc/fstab. Once online I found my encrypted disk was showing as "unkown device" using pvs.

Fixed this but still no device files for the Volume group. Running the above command recreated the device mapper files and allowed me to mount.

Answer 4

I had a similar problem after upgrading my debian. During the reboot, this message appeared to me:

Unable to find LVM Volume. 
/dev/mapper/debian-root does not exist.

I found the solution here:

cryptsetup luksOpen /dev/sda5 lvmsys
lvm
lvm> vgscan        ## scan for volume groups
lvm> vgchange -a y ## activates the volume groups

And voilà, it rebooted nicely after this.

Answer 5

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=616689 is relevant here. It's to do with timeouts such that lvm root does not appear in time.