0

This is the first time I'm posting here; usually, I always find someone who has had the same problem as me. But I'm facing an issue with my RAID5 setup using LVM and could really use some help.

Initially, I had a RAID5 array with three 20TB disks, providing a usable partition of 37TB. Everything was working perfectly. Recently, I decided to add an additional disk to increase my storage capacity while maintaining the benefits of RAID5.

Here is my current situation:

  • The new disk is successfully added to my Volume Group (VG), which now has a total size of 72.76TB.
  • I extended my Logical Volume (LV) to 54TB.
  • However, I'm unable to increase the size of my partition.

I'm not sure what I'm missing or doing wrong. And of course, I don't have the ability to make backups. Any advice or guidance would be greatly appreciated!

Thanks in advance!

 vgs
  VG      #PV #LV #SN Attr   VSize    VFree
  vg-raid   4   1   0 wz--n-   72,76t    0
lvs --all
  LV                  VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home                hyp1-vg -wi-ao---- 895,50g                                        

  root                hyp1-vg -wi-ao----  23,28g                                        

  swap_1              hyp1-vg -wi-ao---- 976,00m                                        

  tmp                 hyp1-vg -wi-ao----  <1,86g                                        

  var                 hyp1-vg -wi-ao----   9,31g                                        

  lv-media            vg-raid rwi-aor---  54,57t                                    100,00
  [lv-media_rimage_0] vg-raid iwi-aor---  18,19t                                        

  [lv-media_rimage_1] vg-raid iwi-aor---  18,19t                                        

  [lv-media_rimage_2] vg-raid iwi-aor---  18,19t                                        

  [lv-media_rimage_3] vg-raid Iwi-aor---  18,19t                                        

  [lv-media_rmeta_0]  vg-raid ewi-aor---   4,00m                                        

  [lv-media_rmeta_1]  vg-raid ewi-aor---   4,00m                                        

  [lv-media_rmeta_2]  vg-raid ewi-aor---   4,00m                                        

  [lv-media_rmeta_3]  vg-raid ewi-aor---   4,00m                                        

I haven't found any way to change the status of media_rimage_3. (I)mage out-of-sync

This status change to (i)mage, mirror or raid when I do a lvchange --rebuild /dev/sde vg-raid/lv-media -v but at the end of the process the status of media_rimage_3 go back to (I)mage out-of-sync.

 resize2fs  /dev/vg-raid/lv-media 
resize2fs 1.47.0 (5-Feb-2023)
Resizing the filesystem on /dev/vg-raid/lv-media to 14648862720 (4k) blocks.
resize2fs: Erreur d'entrée/sortie while trying to resize /dev/vg-raid/lv-media
 df -h
Sys. de fichiers               Taille Utilisé Dispo Uti% Monté sur
/dev/mapper/vg--raid-lv--media    37T     29T  6,4T  82% /raid5
lvdisplay -m vg-raid/lv-media
  --- Logical volume ---
  LV Path                /dev/vg-raid/lv-media
  LV Name                lv-media
  VG Name                vg-raid
  LV UUID                qXccHM-si1G-6n0D-Jrel-cvv9-KcKk-cfT6BY
  LV Write Access        read/write
  LV Creation host, time plex, 2023-05-30 12:15:44 +0200
  LV Status              available
  # open                 1
  LV Size                54,57 TiB
  Current LE             14305530
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     1024
  Block device           253:8

  --- Segments ---
  Logical extents 0 to 14305529:
    Type                raid5
    Monitoring          monitored
    Raid Data LV 0
      Logical volume    lv-media_rimage_0
      Logical extents   0 to 4768509
    Raid Data LV 1
      Logical volume    lv-media_rimage_1
      Logical extents   0 to 4768509
    Raid Data LV 2
      Logical volume    lv-media_rimage_2
      Logical extents   0 to 4768509
    Raid Data LV 3
      Logical volume    lv-media_rimage_3
      Logical extents   0 to 4768509
    Raid Metadata LV 0  lv-media_rmeta_0
    Raid Metadata LV 1  lv-media_rmeta_1
    Raid Metadata LV 2  lv-media_rmeta_2
    Raid Metadata LV 3  lv-media_rmeta_3

df -i /raid5
Sys. de fichiers                  Inœuds IUtil.    ILibre IUti% Monté sur
/dev/mapper/vg--raid-lv--media 610369536  26976 610342560    1% /raid5
resize2fs  /dev/vg-raid/lv-media  -p
resize2fs 1.47.0 (5-Feb-2023)
Resizing the filesystem on /dev/vg-raid/lv-media to 14648862720 (4k) blocks.

resize2fs: Erreur d'entrée/sortie while trying to resize /dev/vg-raid/lv-media
Please run 'e2fsck -fy /dev/vg-raid/lv-media' to fix the filesystem
after the aborted resize operation.
[ 9513.873574] EXT4-fs (dm-8): unmounting filesystem.
[ 9539.331537] Buffer I/O error on dev dm-8, logical block 14648862704, async page read
[ 9539.400112] Buffer I/O error on dev dm-8, logical block 14648862704, async page read
[ 9701.816300] Buffer I/O error on dev dm-8, logical block 14648862704, async page read
[ 9720.326547] Buffer I/O error on dev dm-8, logical block 14648862704, async page read
[10554.318976] perf: interrupt took too long (2503 > 2500), lowering kernel.perf_event_max_sample_rate to 79750
[11111.774521] Buffer I/O error on dev dm-8, logical block 9766436864, lost async page write
[11111.774613] Buffer I/O error on dev dm-8, logical block 9766961152, lost async page write
[11111.774699] Buffer I/O error on dev dm-8, logical block 9767485440, lost async page write
[11111.774792] Buffer I/O error on dev dm-8, logical block 9768009728, lost async page write
[11111.774882] Buffer I/O error on dev dm-8, logical block 9768534016, lost async page write
[11111.774898] Buffer I/O error on dev dm-8, logical block 9769058304, lost async page write
[11111.775052] Buffer I/O error on dev dm-8, logical block 9769582592, lost async page write
[11111.775151] Buffer I/O error on dev dm-8, logical block 9770106880, lost async page write
[11111.775264] Buffer I/O error on dev dm-8, logical block 9770631168, lost async page write
[11111.775376] Buffer I/O error on dev dm-8, logical block 9771155456, lost async page write
5
  • 1
    If you umount the filesystem and then resize it, does that work? Commented May 21 at 15:03
  • 1
    Thank you for your response, but no.
    – Tonnulus
    Commented May 21 at 18:05
  • df -i /raid5 to check the number of inodes, please Commented May 21 at 18:43
  • df -i /raid5 Sys. de fichiers Inœuds IUtil. ILibre IUti% Monté sur /dev/mapper/vg--raid-lv--media 610369536 26981 610342555 1% /raid5
    – Tonnulus
    Commented May 21 at 19:17
  • In your question please, formatted, and so everyone can see it easily Commented May 21 at 20:24

3 Answers 3

0

Before adding the extra disk you need to have at least 1 PE free on each PV.

So first shrink the original LV by 3 LE:

lvresize -r -l -3 vg-raid/lv-media

then add the extra disk to the RAID5 volume:

lvconvert --stripes 3 vg-raid/lv-media
2
  • ``` lvresize -r -l -3 vg-raid/lv-media fsadm: Cannot get FSTYPE of "/dev/mapper/vg--raid-lv--media". Filesystem check failed. ``` Same message with the parition mounted or not.
    – Tonnulus
    Commented May 21 at 19:19
  • -r flag supports only ext2, ext3, ext4, ReiserFS and XFS. Try shrinking the filesystem with "resize2fs" and then shrink the LV with lvresize without "-r" Commented May 22 at 10:48
0

You had a RAID5 set of 3 disks, and added 1 new disk.

Since 3 separate disks is the minimum for RAID5, the most straightforward extension would have required adding another set of 3 disks. Since you added just one disk, using it in a RAID5 will require a "reshape" operation to convert a 3-disk RAID5 into a 4-disk RAID5.

Such an operation is going to have to touch all the data on each disk on the set, so it is going to take a significant amount of time. See the chapter RAID RESHAPING in man 7 lvmraid.

Does /proc/mdstat indicate such an operation might already be happening automatically? If so, then you can monitor /proc/mdstat to see its progress; after the reshape operation is complete, the usable space in the LV should increase, and you could then resize the filesystem.

Otherwise, I think you would need to run a

lvconvert --stripes 3 vg-raid/lv-media

to manually start the appropriate reshape operation. The --stripes option is a bit of a misnomer here, as it is used to indicate the number of devices to use, with the added wrinkle that a RAID5 set with the --stripes value N actually needs N+1 separate PVs. Thus --stripes 3 to to reshape your RAID5 for 4 disks.


The LV seems to have been successfully extended/reshaped, and the problem is only in making the new capacity available to the filesystem.

You've never confirmed the actual filesystem type: the resize2fs command is appropriate for filesystems of type ext2, ext3 and ext4 only. Other filesystem types have their own extension tools.

Assuming that resize2fs is the appropriate tool, the error message could mean that since its original creation, the filesystem has been online-extended so far that it's run out of space reserved for extending certain critical metadata structures. If that's the case, it can be fixed by running an off-line extension operation:

umount /raid5
resize2fs /dev/vg-raid/lv-media
mount /raid5

With an off-line extension operation, resize2fs will be able to move any filesystem elements around if needed, to allow a further extension of the metadata structures.

(I don't remember the exact limit for the on-line extension, but I think it was something like "100x the filesystem size at the time of creation or last offline extension". So running one off-line extension now should open the way for further on-line extensions for quite a while.)

Another possible reason for the resize2fs: No space left on device While checking for on-line resizing support error message is that there is a problem with the disk device - check the dmesg output for disk I/O errors.

4
  • Hello, thank you for your response. I have added the result of lvdisplay -m vg-raid/lv-mediain my first post. lvconvert --stripes 3 vg-raid/lv-mediais done but nothing change. lv-media_rimage_3 have still this strange status. I did lvchange --syncaction repair vg-raid/lv-media without change
    – Tonnulus
    Commented May 23 at 8:06
  • After unmout this partition and a quick e2fsck -f /dev/vg-raid/lv-media the resize2fs /dev/vg-raid/lv-media give this result: The filesystem is already 9765908480 (4k) blocks long. Nothing to do! To me the problem is the state of lv-media_rimage_3 Iwi-aor---
    – Tonnulus
    Commented May 23 at 16:06
  • So, the filesystem still sees the old LV size... and the I indicates a RAID/mirror sub-image that is not in sync. The lvmraid man page says "LVM RAID uses both Device Mapper (DM) and Multiple Device (MD) drivers from the Linux kernel." Does /proc/mdstat indicate anything? Do you see any relevant messages if you run a sudo dmesg?
    – telcoM
    Commented May 23 at 17:06
  • Oh no... those "Buffer I/O error" messages don't seem good at all. You said you don't have a way to back up your data: getting such a way ASAP would be a very good idea.
    – telcoM
    Commented May 24 at 8:05
-1

after adding extra disk you need to run:

lvextend -l +100%FREE /dev/mapper/vg--raid-lv--media

then run the resize2fs as before.

1
  • That would use the entire space. The OP already said (and showed) they'd increased the LV to the appropriate 54TB size Commented May 21 at 16:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .