I have Centos 7

I have a 7 drive RAID6 Array created using mdadm. (This is composed on WD 4TB Red drives)

I have setup the RAID array as a LVM PV - all this has been working well for ages.

I have almost maxed out the drive space on the LVM - what i would like to do is to add additional space as follows

1) Purchase 2 x WD RED 6TB drives, create a RAID1 set from them under mdadm 2) Present that RAID set to LVM by creating a PV on it 3) Use that additional 6TB of disk space to extend the LV

I do not however want to make the LV 6TB bigger, the end result i want to achieve is to remove one of the 4TB drives from the RAID6 array, thus effectively increasing the total disk space by 2TB.

I believe the high level steps i need to take are as follows:-

1) Create a RAID1 array (call it /dev/md1 e.g.) composed of the two 6TB drives 2) partition the 6TB to have a LVM partition 3) create a PV with the 6TB of space 4) Add the PV to the existing VG 5) Move an appropriate number of extents to the 6TB PV to free up 4TB of extents in the RAID6 VG

This is now where i start to get a bit vague - i THINK i would

6) Shrink the RAID6 VG to ensure that 4TB of disk space is available 7) Reduce the number of disks in the RAID6 array 8) Shutdown the box and remove the retired drive ?

I need to flesh out 6), 7), 8) Please and let me know if i am on the right track

The RAID6 array is healthy - i will move the retired drive across to a secondary box that performs backup functions.


Your description is more or less correct.

However, this is quite a process: you have to pvmove extents currently located in the last 4TB of the PV, then reshape the entire RAID6 array (note there just was a RAID6 corruption bug in kernel 5.1.x prior to 5.1.5), and cross fingers and hope nothing will go wrong anywhere.

the end result i want to achieve is to remove one of the 4TB drives from the RAID6 array, thus effectively increasing the total disk space by 2TB.

Unless you have plans to get rid of all 4TB drives, I would do it a different way: partition the 6TB drives to participate in the 4TB RAID6 array.

Since you want to remove one 4TB drive, use mdadm --replace to replace it with (a 4TB partition on) one of your 6TB drives. Then you're free to move the 4TB drive to another box, the size of the RAID6 array stays the same.

Then you mdadm --grow --raid-devices=8 --add (a 4TB partition on) the other 6TB drive. Your RAID6 PV grows by 4TB in this step.

At this point only 4TB are used on your 6TB drives. The remaining 2TB partitions can be used to build a RAID1 array, adding another 2TB of useable storage.

Thus effectively increasing the total disk space by 6TB, and 4TB of those are covered by 2-way redundancy.

All my RAID arrays are built like this. I partition all my drives into same-sized segments so I can mix and match drive sizes and maximize storage efficiency without sacrificing redundancy; it also allows me to run dangerous reshape operations on a small segment of data first instead of risking it all in one shot; and if one drive goes bad in one place, only the affected segment degrades while other segments stay on full redundancy until you --replace the offending drive.

  • Thanks for taking the time to answer so quickly and effectively. Ultimately i do want to replace the whole array of 4TB drives (as funds permit) over the next two years or so. The drives in question all contain media for our Home entertainment, Movies, Music and TV Shows - all of this is backed up/replicated onto another system once per week - so whilst the thought of having to replicate it all back is not one i would want to entertain, it is one i can do. (in the event that some of the mdraid stuff goes horribly wrong.
    – Craig
    Jun 2 '19 at 8:11
  • Ultimately i would add another pair of RAID1 drives in 6TB at XMAS (thats what the 2TB of extra disk space will buy me in terms of time), i will then be able to convert the RAID10 (6TB) drives to RAID6 with the addition of 2 more 6TB drives. Correct me if i am wrong, but if i went down the path you have proposed i would end up with an issue when i got to the end as i would not be able to convert the left over 2TB partitions to be part of the RAID 6 - they would always be multiple RAID1 sets ?
    – Craig
    Jun 2 '19 at 8:22
  • They would be multiple RAID6 sets (RAID1 for 2 drives, turns into RAID5 for 3+ drives, eventually in RAID6) - which is fine for me as LVM doesn't care how many PVs it has in its VG (and I also happen to prefer several smaller filesystems over a single gigantic one). Going back to one large RAID array after replacing all 4TB drives with 6TB ones, would involve shrinking LVM down to 4TB base then growing partitions and 4TB RAID up. Jun 2 '19 at 8:36
  • Hmm, OK will give is some more thought and do a bit more research and see which way i will jump - will let you know - and thanks for your time once again.
    – Craig
    Jun 2 '19 at 9:34
  • Just to let everyone know this worked out fine. 1) Add a new pair of 6TB drives as a RAID1 pair 2) Added them as a LVM PV to the same VG as the existing disks 3) PVMove to move off enough extents from the old 4TB drives in the array to free up a whole disk (i added a bit of extra space to be sure i had it right) 4) Shrink the size of the PV on the 4TB array 5) Use MDADM to Resize the array to be less than the disk i want to retire (but leave a small buffer of additional size to house all the PV extents) - this was the scary part to make sure the equations we all correct
    – Craig
    Jun 21 '19 at 3:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.