Disk replacement in encrypted LUKS LVM without data loss

Question

so here's my setup of an off-site rsync backup server.

Ubuntu 20.10 with 9 hard disks.

Disks /dev/sd[a-h] belong to backup volume group.

System is on /dev/sdi

Server is:

• powered on via network-controlled switch (otherwise it is cut-off from power grid)
• configured with Wake-on-lan
• configured with dropbear that can be used to unlock the cryptfs over network and allow system boot

Initial LVM LUKS setup:

cryptsetup luksFormat --hash=sha512 --key-size=512 --cipher=aes-xts-plain64 --verify-passphrase /dev/sda
cryptsetup luksFormat --hash=sha512 --key-size=512 --cipher=aes-xts-plain64 --verify-passphrase /dev/sdb
cryptsetup luksFormat --hash=sha512 --key-size=512 --cipher=aes-xts-plain64 --verify-passphrase /dev/sdc
cryptsetup luksFormat --hash=sha512 --key-size=512 --cipher=aes-xts-plain64 --verify-passphrase /dev/sdd
cryptsetup luksFormat --hash=sha512 --key-size=512 --cipher=aes-xts-plain64 --verify-passphrase /dev/sde
cryptsetup luksFormat --hash=sha512 --key-size=512 --cipher=aes-xts-plain64 --verify-passphrase /dev/sdf
cryptsetup luksFormat --hash=sha512 --key-size=512 --cipher=aes-xts-plain64 --verify-passphrase /dev/sdg
cryptsetup luksFormat --hash=sha512 --key-size=512 --cipher=aes-xts-plain64 --verify-passphrase /dev/sdh

cryptsetup luksOpen /dev/sda luks_sda
cryptsetup luksOpen /dev/sdb luks_sdb
cryptsetup luksOpen /dev/sdc luks_sdc
cryptsetup luksOpen /dev/sdd luks_sdd
cryptsetup luksOpen /dev/sde luks_sde
cryptsetup luksOpen /dev/sdf luks_sdf
cryptsetup luksOpen /dev/sdg luks_sdg
cryptsetup luksOpen /dev/sdh luks_sdh

pvcreate /dev/mapper/luks_sda
pvcreate /dev/mapper/luks_sdb
pvcreate /dev/mapper/luks_sdc
pvcreate /dev/mapper/luks_sdd
pvcreate /dev/mapper/luks_sde
pvcreate /dev/mapper/luks_sdf
pvcreate /dev/mapper/luks_sdg
pvcreate /dev/mapper/luks_sdh

vgcreate tiburon_backup_vg /dev/mapper/luks_sda

Added other /dev/mapper/luks_sd* devices to created vg Created lv with a mountpoint

Updated /etc/crypttab for each luks_sd*:

luks_sd[a-h] /dev/sd[a-h] /etc/luks-keys/luks_sd[a-h] luks

Then updated initramfs:

update-initramfs -uv
reboot

It was all good for 7 years, till now when I need to replace /dev/sdf as it has more and more bad sectors.

Not sure how to proceed without copying 5TB of data, and without the data loss.

Here's what I found so far (in order not to lose the data):

cryptsetup status

cryptswap1
luks_sde
Tiburon2--vg-root
luks_sda
luks_sdf                 #problematic luks disk
Tiburon2--vg-swap_1
luks_sdb
luks_sdg
tiburon_backup_vg-tiburon_backup          #problematic vg-lv
luks_sdc
luks_sdh
luks_sdd
sdb5_crypt

cryptsetup status luks_sdf

/dev/mapper/luks_sdf is active and is in use.
  type:    LUKS1
  cipher:  aes-xts-plain64
  keysize: 512 bits
  key location: dm-crypt
  device:  /dev/sdf
  sector size:  512
  offset:  4096 sectors
  size:    3907025072 sectors
  mode:    read/write

umount /tiburon_backup

vgchange -a n tiburon_backup_vg

  0 logical volume(s) in volume group "tiburon_backup_vg" now active


pvmove /dev/mapper/luks_sdf

  Insufficient free space: 476931 extents needed, but only 1 available
  Unable to allocate mirror extents for tiburon_backup_vg/pvmove0.
  Failed to convert pvmove LV to mirrored.

#Therefore:
e2fsck -f /dev/mapper/tiburon_backup_vg-tiburon_backup


#FS/VG has 8TB, and 4TB is in use, therefore shrinking it to 5TB:

resize2fs -p /dev/mapper/tiburon_backup_vg-tiburon_backup  5T

Początkowy przebieg 2 (maksymalny = 262145)
Relokowanie bloków           XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Początkowy przebieg 3 (maksymalny = 40960)
Przeszukiwanie tablicy i-węzłówXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

lvreduce -L 5T /dev/mapper/tiburon_backup_vg-tiburon_backup

  WARNING: Reducing active logical volume to 5,00 TiB.
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce tiburon_backup_vg/tiburon_backup? [y/n]: y
  Size of logical volume tiburon_backup_vg/tiburon_backup changed from <7,80 TiB (2043653 extents) to 5,00 TiB.
  Logical volume tiburon_backup_vg/tiburon_backup successfully resized.



e2fsck -f /dev/mapper/tiburon_backup_vg-tiburon_backup

e2fsck 1.45.6 (20-Mar-2020)
Przebieg 1: Sprawdzanie i-węzłów, bloków i rozmiarów
Przebieg 2: Sprawdzanie struktury katalogów
Przebieg 3: Sprawdzanie łączności katalogów
Przebieg 4: Sprawdzanie liczników odwołań
Przebieg 5: Sprawdzanie sumarycznych informacji o grupach
/dev/mapper/tiburon_backup_vg-tiburon_backup: 11/176128 plików (0.0% nieciągłych), 281453/1408000 bloków

Now, pvscan shows that /dev/mapper/luks_sdf is empty:

PV /dev/mapper/luks_sdf VG tiburon_backup_vg lvm2 [<1,82 TiB / 1,82 TiB free]

Therefore, if I now run:

pvmove /dev/mapper/luks_sdf

it should mirror remaining blocks from this pv to some other free space within the vg, right? (or not?)

After that, I would plan to do:

vgchange -a n tiburon_backup_vg

cryptsetup close luks_sdf

vgreduce tiburon_backup_vg /dev/mapper/luks_sdf

pvremove /dev/sdf


#remove luks_sdf from /etc/crypttab

Will that work, or is there a better way to get rid of that faulty disk from a vm on LUKS?

Many thanks for any ideas that you might have.

Answer 1

The order of your operations needs a small correction.

Yes, pvmove would do exactly that if there are any allocated blocks left. If /dev/mapper/luks_sdf is actually already completely free of LVM data, it will do no harm.

If successful, pvdisplay /dev/mapper/luks_sdf should show the exact same value in both Total PE and Free PE fields, and Allocated PE should be 0.

At that point, you don't have to do a vgchange -a n tiburon_backup_vg; just do a vgreduce tiburon_backup_vg /dev/mapper/luks_sdf to remove it from the VG (since it's now empty of LVM data, you can do it on-line).

Since you have LVM on top of LUKS, it is important you do this before cryptsetup close luks_sdf, because after that the system will be only seeing the encrypted contents of /dev/sdf: if you try pvremove /dev/sdf it will tell you there is no LVM header to delete (because it will just see meaningless encrypted data).

In this case, running pvremove is not needed: as long as the disk has been removed from the VG, LVM will no longer require it to be present, and would not mind even if you hot-unplugged it. (Don't hot-unplug if your hardware is not actually hot-plug capable.)

Before shutting down, remember to remove or comment out /dev/sdf from /etc/crypttab and update the initramfs, or else the system will drop you into emergency mode at boot, as it would be trying to activate LUKS on /dev/sdf and no longer finding that disk (or would find a new disk with no existing LUKS header in its place).

Answer 2

I HOPE THIS TEST WILL BE USEFUL TO SOMEONE IN THE FUTURE.

root@Tiburon3:~# pvscan
  PV /dev/mapper/luks_sda     VG bck_vg          lvm2 [<232.87 GiB / 0    free]
  PV /dev/mapper/luks_sdb     VG bck_vg          lvm2 [<931.50 GiB / 0    free]
  PV /dev/mapper/luks_sdc     VG bck_vg          lvm2 [<1.82 TiB / 0    free]
  PV /dev/mapper/luks_sdd     VG bck_vg          lvm2 [149.03 GiB / 0    free]
  PV /dev/mapper/luks_sde     VG bck_vg          lvm2 [<1.82 TiB / 0    free]
  PV /dev/mapper/luks_sdf     VG bck_vg          lvm2 [<1.82 TiB / 0    free]
  PV /dev/mapper/luks_sdg     VG bck_vg          lvm2 [<931.50 GiB / 0    free]
  PV /dev/mapper/luks_sdh     VG bck_vg          lvm2 [149.03 GiB / 79.83 GiB free]
  PV /dev/mapper/sdb5_crypt   VG Tiburon3-vg     lvm2 [<148.53 GiB / <111.28 GiB free]
  Total: 9 [7.94 TiB] / in use: 9 [7.94 TiB] / in no VG: 0 [0   ]

root@Tiburon3:~# df -hP /tiburon_backup/
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/bck_vg-lv_tib_bck  7.7T   93M  7.3T   1% /tiburon_backup

#FILLING UP FS WITH DUMMY DATA TO REACH THE END OF VG:

cd /tiburon_backup/

FROMHERE=848
for ((i=FROMHERE; i>=1; i--))
do
    fallocate -l 10GB gentoo_root$i.img
done


root@Tiburon3:/tiburon_backup# df -hP /tiburon_backup/
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/bck_vg-lv_tib_bck  7.7T  7.3T   20G 100% /tiburon_backup

fallocate -l 10G gentoo_root000.img

root@Tiburon3:/tiburon_backup# pvscan
  PV /dev/mapper/luks_sda     VG bck_vg          lvm2 [<232.87 GiB / 0    free]
  PV /dev/mapper/luks_sdb     VG bck_vg          lvm2 [<931.50 GiB / 0    free]
  PV /dev/mapper/luks_sdc     VG bck_vg          lvm2 [<1.82 TiB / 0    free]
  PV /dev/mapper/luks_sdd     VG bck_vg          lvm2 [149.03 GiB / 0    free]
  PV /dev/mapper/luks_sde     VG bck_vg          lvm2 [<1.82 TiB / 0    free]
  PV /dev/mapper/luks_sdf     VG bck_vg          lvm2 [<1.82 TiB / 0    free]
  PV /dev/mapper/luks_sdg     VG bck_vg          lvm2 [<931.50 GiB / 0    free]
  PV /dev/mapper/luks_sdh     VG bck_vg          lvm2 [149.03 GiB / 79.83 GiB free]

  rm -f gentoo_root[1-9]*


root@Tiburon3:/tiburon_backup# df -hP .
  Filesystem                     Size  Used Avail Use% Mounted on
  /dev/mapper/bck_vg-lv_tib_bck  7.7T   10G  7.3T   1% /tiburon_backup


root@Tiburon3:/tiburon_backup# du -sh *
  10G   gentoo_root000.img

root@Tiburon3:/# 
btrfs filesystem resize -1T /tiburon_backup
Output:
Resize '/tiburon_backup' of '-1T'

Note, it took quite a lot of time comparing to resizing an empty fs.

I wonder if I could have used some --progress or --verbose parameter to see some more output.

root@Tiburon3:/tiburon_backup# df -hP .
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/bck_vg-lv_tib_bck  6.8T   12G  6.8T   1% /tiburon_backup


umount /tiburon_backup


root@Tiburon3:/# pvscan
  PV /dev/mapper/luks_sda     VG bck_vg          lvm2 [<232.87 GiB / <232.87 GiB free]
  PV /dev/mapper/luks_sdb     VG bck_vg          lvm2 [<931.50 GiB / <931.50 GiB free]
  PV /dev/mapper/luks_sdc     VG bck_vg          lvm2 [<1.82 TiB / 0    free]
  PV /dev/mapper/luks_sdd     VG bck_vg          lvm2 [149.03 GiB / 149.03 GiB free]
  PV /dev/mapper/luks_sde     VG bck_vg          lvm2 [<1.82 TiB / 0    free]
  PV /dev/mapper/luks_sdf     VG bck_vg          lvm2 [<1.82 TiB / 469.00 GiB free]
  PV /dev/mapper/luks_sdg     VG bck_vg          lvm2 [<931.50 GiB / <931.50 GiB free]

  PV /dev/mapper/luks_sdh     VG bck_vg          lvm2 [149.03 GiB / 149.03 GiB free] #Lets remove this one, for testing purposes, smallest size

  PV /dev/mapper/sdb5_crypt   VG Tiburon3-vg     lvm2 [<148.53 GiB / <111.28 GiB free]
  Total: 9 [7.94 TiB] / in use: 9 [7.94 TiB] / in no VG: 0 [0   ]

root@Tiburon3:/# pvdisplay /dev/mapper/luks_sdh
  --- Physical volume ---
  PV Name               /dev/mapper/luks_sdh
  VG Name               bck_vg
  PV Size               149.03 GiB / not usable <3.84 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              38152
  Free PE               20437

  Allocated PE          17715   **#allright!, this is the scenario I wanted to test. Data migration before removing the disk.**

  PV UUID               VRhdHD-5aam-9Wha-Qzwg-f8Iz-hosM-Wz0Q3q

root@Tiburon3:/# pvmove /dev/mapper/luks_sdh
    No extents available for allocation.

#To be on a safe side, I will reduce LV not by full 1TB, but by 800GB, should be enough to move remaining allocated PEs from luks_sdh...

root@Tiburon3:/# lvreduce -L -800G /dev/mapper/bck_vg-lv_tib_bck
      WARNING: Reducing active logical volume to <6.94 TiB.
      THIS MAY DESTROY YOUR DATA (filesystem etc.)
    Do you really want to reduce bck_vg/lv_tib_bck? [y/n]: y
      Size of logical volume bck_vg/lv_tib_bck changed from <7.72 TiB (2023191 extents) to <6.94 TiB (1818391 extents).
      Logical volume bck_vg/lv_tib_bck successfully resized.


root@Tiburon3:/# pvmove /dev/mapper/luks_sdh  --alloc anywhere
  /dev/mapper/luks_sdh: Moved: 0.06%
[...]
  /dev/mapper/luks_sdh: Moved: 82.00%
[...]
  /dev/mapper/luks_sdh: Moved: 99.99%
Done!

root@Tiburon3:/# pvdisplay /dev/mapper/luks_sdh
  --- Physical volume ---
  PV Name               /dev/mapper/luks_sdh
  VG Name               bck_vg
  PV Size               149.03 GiB / not usable <3.84 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              38152
  Free PE               38152
  
  Allocated PE          0       **#DATA WAS MOVED OUT OF THIS PV!**
  
  PV UUID               VRhdHD-5aam-9Wha-Qzwg-f8Iz-hosM-Wz0Q3q

root@Tiburon3:/# vgreduce bck_vg /dev/mapper/luks_sdh
    Removed "/dev/mapper/luks_sdh" from volume group "bck_vg"


root@Tiburon3:/# mount -a

root@Tiburon3:/# cd /tiburon_backup/

root@Tiburon3:/tiburon_backup# du -sh *
    10G gentoo_root000.img    **#DATA IS INTACT**

Now:

cryptsetup close luks_sdh

Now, as suggested by telcoM above, it is wise to remove (or comment out) luks_sdh from /etc/crypttab and update initramfs:

update-initramfs -uv

And reboot to test that it is working as expected.

Now I will run this on my Private Prod Env ;)

@telcoM, many thanks for your suggestions!