2

I have an LVM VG with a single LV that is a raid5 volume with three PVs. I would like to add an additional PV to the volume group and extend the raid5 LV to use it.

Here I am using 4 100MB files as test disks to practice.

$ sudo vgs
  WARNING: Not using lvmetad because a repair command was run.
  VG     #PV #LV #SN Attr   VSize   VFree 
  testvg   4   1   0 wz--n- 384.00m 96.00m
$ sudo pvs
  WARNING: Not using lvmetad because a repair command was run.
  PV           VG     Fmt  Attr PSize  PFree 
  /dev/loop0p1 testvg lvm2 a--  96.00m     0 
  /dev/loop1p1 testvg lvm2 a--  96.00m     0 
  /dev/loop2p1 testvg lvm2 a--  96.00m     0 
  /dev/loop3p1 testvg lvm2 a--  96.00m 96.00m
$ sudo lvs
  WARNING: Not using lvmetad because a repair command was run.
  LV       VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  testraid testvg rwi-a-r--- 184.00m                                    100.00    

If I try changing the number of stripes to bring in the additional disk, the command returns errors but the new PV now shows subLVs, and the LV shows increased space available. But the subLVs show an out-of-sync attribute, and running a repair on the LV fails.

$ sudo lvconvert --stripes 3 /dev/testvg/testraid
  Using default stripesize 64.00 KiB.
  WARNING: Adding stripes to active logical volume testvg/testraid will grow it from 46 to 69 extents!
  Run "lvresize -l46 testvg/testraid" to shrink it or use the additional capacity.
Are you sure you want to add 1 images to raid5 LV testvg/testraid? [y/n]: y
  Insufficient free space: 4 extents needed, but only 0 available
  Failed to allocate out-of-place reshape space for testvg/testraid.
  Insufficient free space: 4 extents needed, but only 0 available
  Failed to allocate out-of-place reshape space for testvg/testraid.
  Reshape request failed on LV testvg/testraid.
$ sudo pvs -a -o +pv_pe_count,pv_pe_alloc_count
  PV                    VG     Fmt  Attr PSize  PFree PE  Alloc
  /dev/loop0p1          testvg lvm2 a--  96.00m    0   24    24
  /dev/loop1p1          testvg lvm2 a--  96.00m    0   24    24
  /dev/loop2p1          testvg lvm2 a--  96.00m    0   24    24
  /dev/loop3p1          testvg lvm2 a--  96.00m    0   24    24
$ sudo lvs -a
  LV                  VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  testraid            testvg rwi-a-r--- 276.00m                                    100.00          
  [testraid_rimage_0] testvg iwi-aor---  92.00m                                                    
  [testraid_rimage_1] testvg iwi-aor---  92.00m                                                    
  [testraid_rimage_2] testvg iwi-aor---  92.00m                                                    
  [testraid_rimage_3] testvg Iwi-aor---  92.00m                                                    
  [testraid_rmeta_0]  testvg ewi-aor---   4.00m                                                    
  [testraid_rmeta_1]  testvg ewi-aor---   4.00m                                                    
  [testraid_rmeta_2]  testvg ewi-aor---   4.00m                                                    
  [testraid_rmeta_3]  testvg ewi-aor---   4.00m 
$ sudo lvconvert --repair /dev/testvg/testraid 
  WARNING: Not using lvmetad because of repair.
  Active raid has a wrong number of raid images!
  Metadata says 4, kernel says 3.
Attempt to replace failed RAID images (requires full device resync)? [y/n]: y
  WARNING: Disabling lvmetad cache for repair command.
  Unable to repair testvg/testraid.  Source devices failed before the RAID could synchronize.
  You should choose one of the following:
    1) deactivate testvg/testraid, revive failed device, re-activate LV, and proceed.
    2) remove the LV (all data is lost).
    3) Seek expert advice to attempt to salvage any data from remaining devices.
  Failed to replace faulty devices in testvg/testraid.

What steps should I take to grow my LV by an additional identical disk?

2
  • have you found the solution?
    – rmalchow
    May 12 '20 at 17:07
  • @rmalchow yes, please see newly added answer and linked notes I took.
    – ctag
    May 12 '20 at 22:54
2

My notes taken while working on this are here: https://wiki.archlinux.org/index.php/User:Ctag/Notes#Growing_LVM_Raid5

I wound up adding a new disk, and migrating to Raid6.

If I recall correctly, the issue lies both in the new disk being a few sectors smaller than the others, and the overhead of necessary LVM/raid metadata increasing slightly with the addition of a new disk (so an identical disk also wouldn't work). The way to fix both problems is to underutilize all disks by a few sectors, leaving headroom for metadata and also future disk discrepancies.

# pvs -a -o +pv_pe_count,pv_pe_alloc_count
  PV                     VG      Fmt  Attr PSize  PFree    PE     Alloc 
  /dev/mapper/cryptslow1 cryptvg lvm2 a--  <1.82t   20.00m 476931 476931
  /dev/mapper/cryptslow2 cryptvg lvm2 a--  <1.82t   20.00m 476931 476931
  /dev/mapper/cryptslow3 cryptvg lvm2 a--  <2.73t <931.52g 715395 476931
  /dev/mapper/cryptslow4 cryptvg lvm2 a--  <1.82t   <1.82t 476927      0

See above how the new disk has only '476927' extents and not '476931'? That's the problem. We need to make LVM only allocate the smaller number (or less) extents for the RAID5 arrangement to be able to use this new disk.

# lvresize -r -l -10 /dev/cryptvg/raid
fsck from util-linux 2.34
/dev/mapper/cryptvg-raid: clean, 913995/240320512 files, 686703011/961280000 blocks
resize2fs 1.45.3 (14-Jul-2019)
Resizing the filesystem on /dev/mapper/cryptvg-raid to 976742400 (4k) blocks.
The filesystem on /dev/mapper/cryptvg-raid is now 976742400 (4k) blocks long.

  Size of logical volume cryptvg/raid changed from <3.64 TiB (953860 extents) to <3.64 TiB (953850 extents).
  Logical volume cryptvg/raid successfully resized.

# pvs -a -o +pv_pe_count,pv_pe_alloc_count
  PV                     VG      Fmt  Attr PSize  PFree    PE     Alloc 
  /dev/mapper/cryptslow1 cryptvg lvm2 a--  <1.82t   20.00m 476931 476926
  /dev/mapper/cryptslow2 cryptvg lvm2 a--  <1.82t   20.00m 476931 476926
  /dev/mapper/cryptslow3 cryptvg lvm2 a--  <2.73t <931.52g 715395 476926
  /dev/mapper/cryptslow4 cryptvg lvm2 a--  <1.82t   <1.82t 476927      0

Now we can continue with adding our new disk, and this time it will work.

5
  • thank you very much! this helped a lot ... i have an identical drive, but what you're saying makes sense. will shrink and resize. (my disks: /dev/mapper/sda1_crypt vgraid lvm2 a-- <7.28t 60.00m 1907716 1907701 /dev/mapper/sdb1_crypt vgraid lvm2 a-- <7.28t 60.00m 1907716 1907701 /dev/mapper/sdc1_crypt vgraid lvm2 a-- <7.28t 60.00m 1907716 1907701 /dev/mapper/sdd1_crypt vgraid lvm2 a-- <7.28t <7.28t 1907716 0 )
    – rmalchow
    May 13 '20 at 7:14
  • @rmalchow It looks like your setup already has a little bit of room for metadata, look at 1907716 PE on your drive, and only 1907701 is used. You may be able to skip the resize portion, but if you go ahead it shouldn't hurt anything.
    – ctag
    May 13 '20 at 23:10
  • 1
    thanks. i finally realized that my issue was actually the fact that my test VM was on debian 9, and lvm2 on debian9 is too old to be able to do this. thank you very much. what helped the most in the end were you notes, and lvs o +data_stripes ... this gave me an error, and from there i found the version issue (2.02 -> 2.03)
    – rmalchow
    May 17 '20 at 12:26
  • @ctag at the bottom of your wiki section it says Afterwards the scrub shows 100% complete with no mismatches, but the disk is still marked with attribute (I) which means out-of-sync. What was the fix for this? The final output seems to show an error.
    – Tom Hale
    Mar 8 at 14:57
  • @TomHale Apologies but I haven't found any additional notes on it. I believe it was something stupid like an additional scrub that corrected it.
    – ctag
    Mar 18 at 2:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.