why resize a PV instead of adding a new PV?

Question

When you increase the size of a an existing block device (eg /dev/sdc was given 20 gigs from vmware, now it has been increased to 40 gigs), you need to do some work for LVM to use the space. All the guides I've seen (1, 2, 3) suggest deleting the existing partition and recreate it with the extra space added on to the end, then run pvresize.

Instead of resizing /dev/sdc1 then running pvresize /dev/sdc1, you could also create /dev/sdc2 and run pvcreate /dev/sdc2 and add it to the VG.

Is there a technical or performance reason why resizing is better than adding a new partition or is resizing just the way its always been done?

Answer 1

Summary: From a purely technical standpoint, it doesn't make much difference, but resizing is better. Once you add in practical aspects, adding a new partition is a clear winner.

From a strictly technical standpoint, new PVs have a few downsides:

• You get another copy of the LVM metadata, this costs some disk space. If you frequently do LVM operations, this also causes write amplification (as the metadata is mirrored to all PVs)
• There may be empty space left for alignment
• Increases amount of LVM metadata (per copy), slightly, to store info about the new PV.
• If you create a LV that spans both PVs, it's not contiguous (due to the extra metadata and any holes left for alignment). So, e.g, if you were using that LV for big sequential reads/writes, there is now a seek in there.

From a practical standpoint, none of those matter for any reasonable number of PVs. The extra copies of the metadata matter first, but there is an LVM option to keep fewer copies (VG --metadatacopies or PV --metadataignore).

Further, continuing the practical standpoint, deleting and recreating a partition is much more likely to suffer from admin error (typo, etc.) than creating a new partition due to better tools for the latter. And any admin errors that occur are likely to be much more destructive for resizing (because your data is on the resized partition, but there is no data on a new partition). This is even worse when you have multiple layers; e.g., mdraid below LVM. Depending on your -e option, the superblocks can be at the end of the array—fun when you resize a partition.

There is one exception that comes to mind: if for whatever reason you can't create a new partition. For example, maybe you're using DOS partition tables, and you already used all four primary partitions (without creating an extended partition). Then you don't have a choice.

Answer 2

Scaling storage laterally as you propose by continuing to add more PVs to the same VG to increase your VM's storage will get ugly.

Let's say you need to increase space on your VM 10 times- are you going to add a new PV to the VG 10 times? Your VMs would end up having a gazillion virtual disks and PVs attached to them.

And what happens when you do an lvextend and resize2fs each time you add a PV to the VG to use the extra space? Your LVs and the filesystems that live on them are going to span multiple PVs. If you execute a pvdisplay -m you'll see your LVs using extents from different PVs.

The more unduly complex a system is, the more likely it is to fail is some weird and unexpected way.

That's not to say "No more than ONE PV (virtual disk) per VM"! Use one PV for each VG, ie: vg_system and vg_data would be a tidy organization abstracting data into a separate pile from the OS. I'd only suggest avoiding using multiple PVs for the SAME VG in your VMs.