0

so here's my setup of an off-site rsync backup server.

Ubuntu 20.10 with 9 hard disks.

Disks /dev/sd[a-h] belong to backup volume group.

System is on /dev/sdi

Server is:

  • powered on via network-controlled switch (otherwise it is cut-off from power grid)
  • configured with Wake-on-lan
  • configured with dropbear that can be used to unlock the cryptfs over network and allow system boot

Initial LVM LUKS setup:

cryptsetup luksFormat --hash=sha512 --key-size=512 --cipher=aes-xts-plain64 --verify-passphrase /dev/sda
cryptsetup luksFormat --hash=sha512 --key-size=512 --cipher=aes-xts-plain64 --verify-passphrase /dev/sdb
cryptsetup luksFormat --hash=sha512 --key-size=512 --cipher=aes-xts-plain64 --verify-passphrase /dev/sdc
cryptsetup luksFormat --hash=sha512 --key-size=512 --cipher=aes-xts-plain64 --verify-passphrase /dev/sdd
cryptsetup luksFormat --hash=sha512 --key-size=512 --cipher=aes-xts-plain64 --verify-passphrase /dev/sde
cryptsetup luksFormat --hash=sha512 --key-size=512 --cipher=aes-xts-plain64 --verify-passphrase /dev/sdf
cryptsetup luksFormat --hash=sha512 --key-size=512 --cipher=aes-xts-plain64 --verify-passphrase /dev/sdg
cryptsetup luksFormat --hash=sha512 --key-size=512 --cipher=aes-xts-plain64 --verify-passphrase /dev/sdh

cryptsetup luksOpen /dev/sda luks_sda
cryptsetup luksOpen /dev/sdb luks_sdb
cryptsetup luksOpen /dev/sdc luks_sdc
cryptsetup luksOpen /dev/sdd luks_sdd
cryptsetup luksOpen /dev/sde luks_sde
cryptsetup luksOpen /dev/sdf luks_sdf
cryptsetup luksOpen /dev/sdg luks_sdg
cryptsetup luksOpen /dev/sdh luks_sdh
pvcreate /dev/mapper/luks_sda
pvcreate /dev/mapper/luks_sdb
pvcreate /dev/mapper/luks_sdc
pvcreate /dev/mapper/luks_sdd
pvcreate /dev/mapper/luks_sde
pvcreate /dev/mapper/luks_sdf
pvcreate /dev/mapper/luks_sdg
pvcreate /dev/mapper/luks_sdh
vgcreate tiburon_backup_vg /dev/mapper/luks_sda

Added other /dev/mapper/luks_sd* devices to created vg Created lv with a mountpoint

Updated /etc/crypttab for each luks_sd*:

luks_sd[a-h] /dev/sd[a-h] /etc/luks-keys/luks_sd[a-h] luks

Then updated initramfs:

update-initramfs -uv
reboot

It was all good for 7 years, till now when I need to replace /dev/sdf as it has more and more bad sectors.

Not sure how to proceed without copying 5TB of data, and without the data loss.

Here's what I found so far (in order not to lose the data):

cryptsetup status

cryptswap1
luks_sde
Tiburon2--vg-root
luks_sda
luks_sdf                 #problematic luks disk
Tiburon2--vg-swap_1
luks_sdb
luks_sdg
tiburon_backup_vg-tiburon_backup          #problematic vg-lv
luks_sdc
luks_sdh
luks_sdd
sdb5_crypt

cryptsetup status luks_sdf

/dev/mapper/luks_sdf is active and is in use.
  type:    LUKS1
  cipher:  aes-xts-plain64
  keysize: 512 bits
  key location: dm-crypt
  device:  /dev/sdf
  sector size:  512
  offset:  4096 sectors
  size:    3907025072 sectors
  mode:    read/write



umount /tiburon_backup

vgchange -a n tiburon_backup_vg

  0 logical volume(s) in volume group "tiburon_backup_vg" now active


pvmove /dev/mapper/luks_sdf

  Insufficient free space: 476931 extents needed, but only 1 available
  Unable to allocate mirror extents for tiburon_backup_vg/pvmove0.
  Failed to convert pvmove LV to mirrored.


#Therefore:
e2fsck -f /dev/mapper/tiburon_backup_vg-tiburon_backup


#FS/VG has 8TB, and 4TB is in use, therefore shrinking it to 5TB:

resize2fs -p /dev/mapper/tiburon_backup_vg-tiburon_backup  5T

Początkowy przebieg 2 (maksymalny = 262145)
Relokowanie bloków           XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Początkowy przebieg 3 (maksymalny = 40960)
Przeszukiwanie tablicy i-węzłówXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX


lvreduce -L 5T /dev/mapper/tiburon_backup_vg-tiburon_backup

  WARNING: Reducing active logical volume to 5,00 TiB.
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce tiburon_backup_vg/tiburon_backup? [y/n]: y
  Size of logical volume tiburon_backup_vg/tiburon_backup changed from <7,80 TiB (2043653 extents) to 5,00 TiB.
  Logical volume tiburon_backup_vg/tiburon_backup successfully resized.



e2fsck -f /dev/mapper/tiburon_backup_vg-tiburon_backup

e2fsck 1.45.6 (20-Mar-2020)
Przebieg 1: Sprawdzanie i-węzłów, bloków i rozmiarów
Przebieg 2: Sprawdzanie struktury katalogów
Przebieg 3: Sprawdzanie łączności katalogów
Przebieg 4: Sprawdzanie liczników odwołań
Przebieg 5: Sprawdzanie sumarycznych informacji o grupach
/dev/mapper/tiburon_backup_vg-tiburon_backup: 11/176128 plików (0.0% nieciągłych), 281453/1408000 bloków



Now, pvscan shows that /dev/mapper/luks_sdf is empty:

PV /dev/mapper/luks_sdf VG tiburon_backup_vg lvm2 [<1,82 TiB / 1,82 TiB free]

Therefore, if I now run:

pvmove /dev/mapper/luks_sdf

it should mirror remaining blocks from this pv to some other free space within the vg, right? (or not?)

After that, I would plan to do:

vgchange -a n tiburon_backup_vg

cryptsetup close luks_sdf

vgreduce tiburon_backup_vg /dev/mapper/luks_sdf

pvremove /dev/sdf


#remove luks_sdf from /etc/crypttab

Will that work, or is there a better way to get rid of that faulty disk from a vm on LUKS?

Many thanks for any ideas that you might have.

1
  • In case there are some used blocks on /dev/mapper/luks_sdf, I want them to be relocated to other pvs within that vg so that I do not ruin the vg config and not lose any data. Will that pvremove also take care of relocating the blocks? Jun 14 at 12:50
1

The order of your operations needs a small correction.

Yes, pvmove would do exactly that if there are any allocated blocks left. If /dev/mapper/luks_sdf is actually already completely free of LVM data, it will do no harm.

If successful, pvdisplay /dev/mapper/luks_sdf should show the exact same value in both Total PE and Free PE fields, and Allocated PE should be 0.

At that point, you don't have to do a vgchange -a n tiburon_backup_vg; just do a vgreduce tiburon_backup_vg /dev/mapper/luks_sdf to remove it from the VG (since it's now empty of LVM data, you can do it on-line).

Since you have LVM on top of LUKS, it is important you do this before cryptsetup close luks_sdf, because after that the system will be only seeing the encrypted contents of /dev/sdf: if you try pvremove /dev/sdf it will tell you there is no LVM header to delete (because it will just see meaningless encrypted data).

In this case, running pvremove is not needed: as long as the disk has been removed from the VG, LVM will no longer require it to be present, and would not mind even if you hot-unplugged it. (Don't hot-unplug if your hardware is not actually hot-plug capable.)

Before shutting down, remember to remove or comment out /dev/sdf from /etc/crypttab and update the initramfs, or else the system will drop you into emergency mode at boot, as it would be trying to activate LUKS on /dev/sdf and no longer finding that disk (or would find a new disk with no existing LUKS header in its place).

0

I HOPE THIS TEST WILL BE USEFUL TO SOMEONE IN THE FUTURE.

root@Tiburon3:~# pvscan
  PV /dev/mapper/luks_sda     VG bck_vg          lvm2 [<232.87 GiB / 0    free]
  PV /dev/mapper/luks_sdb     VG bck_vg          lvm2 [<931.50 GiB / 0    free]
  PV /dev/mapper/luks_sdc     VG bck_vg          lvm2 [<1.82 TiB / 0    free]
  PV /dev/mapper/luks_sdd     VG bck_vg          lvm2 [149.03 GiB / 0    free]
  PV /dev/mapper/luks_sde     VG bck_vg          lvm2 [<1.82 TiB / 0    free]
  PV /dev/mapper/luks_sdf     VG bck_vg          lvm2 [<1.82 TiB / 0    free]
  PV /dev/mapper/luks_sdg     VG bck_vg          lvm2 [<931.50 GiB / 0    free]
  PV /dev/mapper/luks_sdh     VG bck_vg          lvm2 [149.03 GiB / 79.83 GiB free]
  PV /dev/mapper/sdb5_crypt   VG Tiburon3-vg     lvm2 [<148.53 GiB / <111.28 GiB free]
  Total: 9 [7.94 TiB] / in use: 9 [7.94 TiB] / in no VG: 0 [0   ]

root@Tiburon3:~# df -hP /tiburon_backup/
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/bck_vg-lv_tib_bck  7.7T   93M  7.3T   1% /tiburon_backup

#FILLING UP FS WITH DUMMY DATA TO REACH THE END OF VG:

cd /tiburon_backup/

FROMHERE=848
for ((i=FROMHERE; i>=1; i--))
do
    fallocate -l 10GB gentoo_root$i.img
done


root@Tiburon3:/tiburon_backup# df -hP /tiburon_backup/
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/bck_vg-lv_tib_bck  7.7T  7.3T   20G 100% /tiburon_backup

fallocate -l 10G gentoo_root000.img

root@Tiburon3:/tiburon_backup# pvscan
  PV /dev/mapper/luks_sda     VG bck_vg          lvm2 [<232.87 GiB / 0    free]
  PV /dev/mapper/luks_sdb     VG bck_vg          lvm2 [<931.50 GiB / 0    free]
  PV /dev/mapper/luks_sdc     VG bck_vg          lvm2 [<1.82 TiB / 0    free]
  PV /dev/mapper/luks_sdd     VG bck_vg          lvm2 [149.03 GiB / 0    free]
  PV /dev/mapper/luks_sde     VG bck_vg          lvm2 [<1.82 TiB / 0    free]
  PV /dev/mapper/luks_sdf     VG bck_vg          lvm2 [<1.82 TiB / 0    free]
  PV /dev/mapper/luks_sdg     VG bck_vg          lvm2 [<931.50 GiB / 0    free]
  PV /dev/mapper/luks_sdh     VG bck_vg          lvm2 [149.03 GiB / 79.83 GiB free]

  rm -f gentoo_root[1-9]*


root@Tiburon3:/tiburon_backup# df -hP .
  Filesystem                     Size  Used Avail Use% Mounted on
  /dev/mapper/bck_vg-lv_tib_bck  7.7T   10G  7.3T   1% /tiburon_backup


root@Tiburon3:/tiburon_backup# du -sh *
  10G   gentoo_root000.img

root@Tiburon3:/# 
btrfs filesystem resize -1T /tiburon_backup
Output:
Resize '/tiburon_backup' of '-1T'

Note, it took quite a lot of time comparing to resizing an empty fs.

I wonder if I could have used some --progress or --verbose parameter to see some more output.

root@Tiburon3:/tiburon_backup# df -hP .
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/bck_vg-lv_tib_bck  6.8T   12G  6.8T   1% /tiburon_backup


umount /tiburon_backup


root@Tiburon3:/# pvscan
  PV /dev/mapper/luks_sda     VG bck_vg          lvm2 [<232.87 GiB / <232.87 GiB free]
  PV /dev/mapper/luks_sdb     VG bck_vg          lvm2 [<931.50 GiB / <931.50 GiB free]
  PV /dev/mapper/luks_sdc     VG bck_vg          lvm2 [<1.82 TiB / 0    free]
  PV /dev/mapper/luks_sdd     VG bck_vg          lvm2 [149.03 GiB / 149.03 GiB free]
  PV /dev/mapper/luks_sde     VG bck_vg          lvm2 [<1.82 TiB / 0    free]
  PV /dev/mapper/luks_sdf     VG bck_vg          lvm2 [<1.82 TiB / 469.00 GiB free]
  PV /dev/mapper/luks_sdg     VG bck_vg          lvm2 [<931.50 GiB / <931.50 GiB free]

  PV /dev/mapper/luks_sdh     VG bck_vg          lvm2 [149.03 GiB / 149.03 GiB free] #Lets remove this one, for testing purposes, smallest size

  PV /dev/mapper/sdb5_crypt   VG Tiburon3-vg     lvm2 [<148.53 GiB / <111.28 GiB free]
  Total: 9 [7.94 TiB] / in use: 9 [7.94 TiB] / in no VG: 0 [0   ]


root@Tiburon3:/# pvdisplay /dev/mapper/luks_sdh
  --- Physical volume ---
  PV Name               /dev/mapper/luks_sdh
  VG Name               bck_vg
  PV Size               149.03 GiB / not usable <3.84 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              38152
  Free PE               20437

  Allocated PE          17715   **#allright!, this is the scenario I wanted to test. Data migration before removing the disk.**

  PV UUID               VRhdHD-5aam-9Wha-Qzwg-f8Iz-hosM-Wz0Q3q



root@Tiburon3:/# pvmove /dev/mapper/luks_sdh
    No extents available for allocation.

#To be on a safe side, I will reduce LV not by full 1TB, but by 800GB, should be enough to move remaining allocated PEs from luks_sdh...

root@Tiburon3:/# lvreduce -L -800G /dev/mapper/bck_vg-lv_tib_bck
      WARNING: Reducing active logical volume to <6.94 TiB.
      THIS MAY DESTROY YOUR DATA (filesystem etc.)
    Do you really want to reduce bck_vg/lv_tib_bck? [y/n]: y
      Size of logical volume bck_vg/lv_tib_bck changed from <7.72 TiB (2023191 extents) to <6.94 TiB (1818391 extents).
      Logical volume bck_vg/lv_tib_bck successfully resized.


root@Tiburon3:/# pvmove /dev/mapper/luks_sdh  --alloc anywhere
  /dev/mapper/luks_sdh: Moved: 0.06%
[...]
  /dev/mapper/luks_sdh: Moved: 82.00%
[...]
  /dev/mapper/luks_sdh: Moved: 99.99%
Done!

root@Tiburon3:/# pvdisplay /dev/mapper/luks_sdh
  --- Physical volume ---
  PV Name               /dev/mapper/luks_sdh
  VG Name               bck_vg
  PV Size               149.03 GiB / not usable <3.84 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              38152
  Free PE               38152
  
  Allocated PE          0       **#DATA WAS MOVED OUT OF THIS PV!**
  
  PV UUID               VRhdHD-5aam-9Wha-Qzwg-f8Iz-hosM-Wz0Q3q



root@Tiburon3:/# vgreduce bck_vg /dev/mapper/luks_sdh
    Removed "/dev/mapper/luks_sdh" from volume group "bck_vg"


root@Tiburon3:/# mount -a

root@Tiburon3:/# cd /tiburon_backup/

root@Tiburon3:/tiburon_backup# du -sh *
    10G gentoo_root000.img    **#DATA IS INTACT**

Now:

cryptsetup close luks_sdh

Now, as suggested by telcoM above, it is wise to remove (or comment out) luks_sdh from /etc/crypttab and update initramfs:

update-initramfs -uv

And reboot to test that it is working as expected.

Now I will run this on my Private Prod Env ;)

@telcoM, many thanks for your suggestions!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.