5

I'm trying to add two new drives to grow a raid10 created with mdadm v3.4 on debian testing with kernel 4.4.6-1. The problem is that the new drives I have have just a few sectors less that the smallest drive that was used to create the array. Is there a way to shrink the partitions without destroying the array?

root@debian-nas:~# mdadm /dev/md0 --add /dev/sdb1 /dev/sdd1
mdadm: /dev/sdb1 not large enough to join array

Here is what the array looks like now.

root@debian-nas:~# mdadm --detail --verbose /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Sat Mar 26 21:13:23 2016
     Raid Level : raid10
     Array Size : 5860268032 (5588.79 GiB 6000.91 GB)
  Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Sun Mar 27 16:25:22 2016
          State : active
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=2
     Chunk Size : 512K

           Name : debian-nas:0  (local to host debian-nas)
           UUID : 86266375:bf94ae93:d449a171:aa746159
         Events : 13726

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync set-A   /dev/sda1
       1       8       33        1      active sync set-B   /dev/sdc1
       2       8       65        2      active sync set-A   /dev/sde1
       3       8       81        3      active sync set-B   /dev/sdf1

I'm trying to add /dev/sdb1 and /dev/sdd1 to /dev/md0

root@debian-nas:~# blockdev --report /dev/sda1
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  2048   512   512       2048   3000591916544   /dev/sda1

root@debian-nas:~# blockdev --report /dev/sdb1
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  2048   512  4096       2048   3000590934016   /dev/sdb1

root@debian-nas:~# blockdev --report /dev/sdc1
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  2048   512  4096       2048   3000591450112   /dev/sdc1

root@debian-nas:~# blockdev --report /dev/sdd1
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  2048   512  4096       2048   3000590934016   /dev/sdd1

root@debian-nas:~# blockdev --report /dev/sde1
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  2048   512  4096       2048   3000591450112   /dev/sde1

root@debian-nas:~# blockdev --report /dev/sdf1
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  2048   512   512       2048   3000591916544   /dev/sdf1

Am I better off just moving the data off of the array, destroying it, then creating it with smaller partitions so that I can include the new drives?

  • What's on the RAID? It should be possible to shrink a little to make the new partitions fit. – frostschutz Mar 28 '16 at 12:43
  • I have about 800GB free on the array right now. I guess that is my question: How would I go about shrinking the array in a non-destructive way so I can add a slightly smaller pair of drives? – rickbassham Mar 28 '16 at 14:00
  • So I think I have a way to do this now. I have tried it in a VM and I'm now ready to try it on my array. e2fsck -f /dev/md0 resize2fs /dev/md0 5000G e2fsck -f /dev/md0 mdadm --verbose --grow --size=14636698788954112 /dev/md0 e2fsck -f /dev/md0 mdadm --add /dev/md0 /dev/sdb1 /dev/sdd1 mdadm --verbose --grow --raid-devices=6 /dev/md0 e2fsck -f /dev/md0 mdadm --verbose --grow --size=max /dev/md0 resize2fs /dev/md0 e2fsck -f /dev/md0 mount /dev/md0 – rickbassham Mar 28 '16 at 19:47
  • IMO you'd be better off backing up the data, installing the zfsonlinux.org packages, rebuilding the array as a zfs pool with mirrored pairs (3 mirrored pairs of 2 disks each), and then restoring the data to it. That gives you RAID10-like mirroring+striping, plus snapshots, compression, volume management (including quotas and reservations), error-detection-and-correction, the ability to use an SSD for caching, and more. – cas Mar 29 '16 at 0:22
  • BTW, in the not-too-distant future (when the last few remaining licensing/copyright details are sorted out in the packages), zfsonlinux will be in debian unstable (and a few weeks later, in testing)....you can get them from alioth if you're impatient, otherwise the jessie packages from zfsonlinux.org will work on testing (up until kernel 4.3 IIRC). – cas Mar 29 '16 at 0:25
3

If you don't need a partition table on /dev/sd[bd] (basically, if you don't need to boot from either drive), you can add the whole drives to the array; that should provide enough sectors:

mdadm /dev/md0 --add /dev/sdb /dev/sdd

Alternatively, you could create a new, degraded array on the new partitions, migrate your data to it and progressively move drives from the old array to the new one.

2

So I figured out a way to do this, using /dev/sdb1 and /dev/sdd1, without rebuilding the array, and without losing data.

This only works with a modern kernel (3.5 or greater) and mdadm version 3.3 or greater.

Unmount the array.

umount /dev/md0

Run a filesystem check. Must be done before the next step.

e2fsck -f /dev/md0

Resize the filesystem to 5 TB.

resize2fs /dev/md0 5000G

Run another check on the filesystem. Make sure we didn't mess up so far.

e2fsck -f /dev/md0

Shrink the array to just over 5 TB. The size parameter here is how much space to use on each disk in the array. This should be (size of fs / number of mirrors) + some small extra. In my case, I made the filesystem 5 TB, I have 4 disks in a raid 10, so I have a stripe of 2 mirrors. THIS MUST GIVE A TOTAL LARGER THAN YOUR FILESYSTEM OR YOU WILL LOSE DATA.

mdadm --verbose --grow --size=2685403136 /dev/md0 # 2.5TB + 1MB

Check the filesystem again.

e2fsck -f /dev/md0

At this point, we can add our smaller drives to the raid 10, but first, do checks to make sure your data is good. Don't just rely on the filesystem check. Actually verify your data is good.

Now we can add the new drives!

mdadm --add /dev/md0 /dev/sdf1 /dev/sdg1

And grow the array to use them.

mdadm --verbose --grow --raid-devices=6 /dev/md0

At this point, doing a cat /proc/mdstat should show something like the following:

Personalities : [raid6] [raid5] [raid4] [raid0] [raid10]
md0 : active raid10 sdd1[5] sdb1[4] sdf1[3] sde1[2] sdc1[1] sda1[0]
      5370806272 blocks super 1.2 512K chunks 2 near-copies [6/6] [UUUUUU]
      [>....................]  reshape =  2.8% (151477952/5370806272) finish=458.7min speed=189604K/sec
      bitmap: 11/31 pages [44KB], 131072KB chunk

unused devices: <none>

Do another filesystem check after the reshape is complete. (I'm paranoid, I'm going to check it every step of the way.)

e2fsck -f /dev/md0

Now grow the size of the array back to the max that your drives will support.

mdadm --verbose --grow --size=max /dev/md0

Then resize the filesystem.

resize2fs /dev/md0

And check it once again.

e2fsck -f /dev/md0

Finally, we can mount the new expanded filesystem.

mount /dev/md0 /mnt/raid10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.