LVM recovery with physical disk failure on remote system

Question

Questions like this have been asked a lot here but I've not found any with the same situation as I have. I have a system with a failed LVM volume group and I have no physical access to it. More details below.

I have a Debian 10 linux server with two physical volumes, a 4TB HDD and a 750GB SSD. The SSD contains the boot volume. The HDD is configured as an LVM Volume Group with a single physical volume (/dev/sda, the 4 TB HDD) and a number of logical volumes.

The HDD has apparently failed. No diagnosis appears possible as /dev/sda simply doesn't exist anymore and any attempt to read from the disk results in an i/o error.

The data on the disk isn't critical and I do have a system backup. The issue is that the system is a headless linux box sitting in the cellar of my house in Switzerland, which is half a world away. My concern is that if I reboot it, it may well fail during boot and enter diagnostic mode and I will have no network access to it. In that case I have to ask a friend to collect it and mail it to me - which is what I'm really trying to avoid as it will be down for months. I can live without /dev/sda until the next time I visit the house and replace it.

The output I get from LVM is as follows:

sudo /sbin/pvdisplay -vd /dev/sda
  Error reading device /dev/Webcams/Birch-Snapshot at 0 length 512.
  Error reading device /dev/Webcams/Birch-Snapshot at 0 length 4096.
  Error reading device /dev/Webcams/Birch-Video at 0 length 512.
  Error reading device /dev/Webcams/Birch-Video at 0 length 4096.
  Error reading device /dev/Webcams/Roof-Snapshot at 0 length 512.
  Error reading device /dev/Webcams/Roof-Snapshot at 0 length 4096.
  Error reading device /dev/Webcams/Roof-Video at 0 length 512.
  Error reading device /dev/Webcams/Roof-Video at 0 length 4096.
  Error reading device /dev/Webcams/Tree-Snapshot at 0 length 512.
  Error reading device /dev/Webcams/Tree-Snapshot at 0 length 4096.
  Error reading device /dev/Webcams/Tree-Video at 0 length 512.
  Error reading device /dev/Webcams/Tree-Video at 0 length 4096.
  Error reading device /dev/Webcams/Archive at 0 length 512.
  Error reading device /dev/Webcams/Archive at 0 length 4096.
  Error reading device /dev/Webcams/Video2 at 0 length 512.
  Error reading device /dev/Webcams/Video2 at 0 length 4096.
  Error reading device /dev/Webcams/Backup at 0 length 512.
  Error reading device /dev/Webcams/Backup at 0 length 4096.
  Failed to find device for physical volume "/dev/sda".

vgs -vd and lvs -vd both execute and print just the single line:

sudo /sbin/vgs -vd
    No volume groups found.

/etc/fstab mounts the individuals logical volumes like so:

# <file system>             <mount point>  <type>  <options>  <dump>  <pass>
/dev/Webcams/Archive            /mnt/Webcams/Archive        ext4    defaults,nofail         1 2

/dev/Webcams looks like this:

ls -l /dev/Webcams
total 0
lrwxrwxrwx 1 root root 7 Aug 25 08:05 Archive -> ../dm-6
lrwxrwxrwx 1 root root 7 Aug 25 08:05 Backup -> ../dm-8
lrwxrwxrwx 1 root root 7 Aug 25 08:05 Birch-Snapshot -> ../dm-0
lrwxrwxrwx 1 root root 7 Aug 25 08:05 Birch-Video -> ../dm-1
lrwxrwxrwx 1 root root 7 Aug 25 08:05 Roof-Snapshot -> ../dm-2
lrwxrwxrwx 1 root root 7 Aug 25 08:05 Roof-Video -> ../dm-3
lrwxrwxrwx 1 root root 7 Aug 25 08:05 Tree-Snapshot -> ../dm-4
lrwxrwxrwx 1 root root 7 Aug 25 08:05 Tree-Video -> ../dm-5
lrwxrwxrwx 1 root root 7 Aug 25 08:05 Video2 -> ../dm-7

There is also a /dev/mapper directory:

ls -l /dev/mapper
total 0
crw------- 1 root root 10, 236 Aug  3 10:44 control
lrwxrwxrwx 1 root root       7 Aug 25 08:05 Webcams-Archive -> ../dm-6
lrwxrwxrwx 1 root root       7 Aug 25 08:05 Webcams-Backup -> ../dm-8
lrwxrwxrwx 1 root root       7 Aug 25 08:05 Webcams-Birch--Snapshot -> ../dm-0
lrwxrwxrwx 1 root root       7 Aug 25 08:05 Webcams-Birch--Video -> ../dm-1
lrwxrwxrwx 1 root root       7 Aug 25 08:05 Webcams-Roof--Snapshot -> ../dm-2
lrwxrwxrwx 1 root root       7 Aug 25 08:05 Webcams-Roof--Video -> ../dm-3
lrwxrwxrwx 1 root root       7 Aug 25 08:05 Webcams-Tree--Snapshot -> ../dm-4
lrwxrwxrwx 1 root root       7 Aug 25 08:05 Webcams-Tree--Video -> ../dm-5
lrwxrwxrwx 1 root root       7 Aug 25 08:05 Webcams-Video2 -> ../dm-7

So both /dev/Webcams and /dev/mapper have symlinks pointing to files in /dev:

ls -l /dev/*dm-*
brw-rw---- 1 root disk 253, 0 Aug 25 08:05 /dev/dm-0
brw-rw---- 1 root disk 253, 1 Aug 25 08:05 /dev/dm-1
brw-rw---- 1 root disk 253, 2 Aug 25 08:05 /dev/dm-2
brw-rw---- 1 root disk 253, 3 Aug 25 08:05 /dev/dm-3
brw-rw---- 1 root disk 253, 4 Aug 25 08:05 /dev/dm-4
brw-rw---- 1 root disk 253, 5 Aug 25 08:05 /dev/dm-5
brw-rw---- 1 root disk 253, 6 Aug 25 08:05 /dev/dm-6
brw-rw---- 1 root disk 253, 7 Aug 25 08:05 /dev/dm-7
brw-rw---- 1 root disk 253, 8 Aug 25 08:05 /dev/dm-8

My question: What steps should I take to ensure that the system does’t enter diagostic mode during reboot? Is it enough to simply comment out the relevant lines in /etc/fstab or is there something else I need to do?

My concern is that lvm itself might make checks during boot and throw off errors that stop the boot from completing. I am not using lvm for any other volumes so theoretically at least I suppose I could remove the package, although given the state the system is in I’m not sure that would be wise.

Any help much appreciated.

Answer 1

Comment out each /etc/fstab line that mentions /dev/Webcams. That should be enough, unless the system has other disks that are part of the same volume group.

Linux LVM (as it exists in kernel versions 2.6 and newer) will not normally store any VG configuration to /etc. All the configuration information is stored in the LVM headers of the respective PVs. If the root filesystem is on LVM, the initramfs usually only attempts to activate the specific LVs containing the root filesystem (and possibly the swap area, referenced by /etc/initramfs-tools/conf.d/resume in Debian).

If the sda disk is the only one in the Webcams VG, now that it has failed, after a reboot LVM will not even know the VG exists. And after commenting out the lines referring to it in /etc/fstab, there should not be any failing mount attempts that would interrupt the boot process.

If, however, the Webcams VG also included some other disk that is still present and working, LVM would detect a partial VG and might drop into emergency shell at boot because of this. You could prevent that from happening by running vgreduce --removemissing --force before boot. This will effectively update the VG configuration on the remaining disk(s) of that VG to forget the missing disk; the --force option will cause it to also forget/destroy any LVs that would have parts missing as a result.