Adding new devices to an mdadm raid10 - new device has fewer sectors

Question

I'm trying to add two new drives to grow a raid10 created with mdadm v3.4 on debian testing with kernel 4.4.6-1. The problem is that the new drives I have have just a few sectors less that the smallest drive that was used to create the array. Is there a way to shrink the partitions without destroying the array?

root@debian-nas:~# mdadm /dev/md0 --add /dev/sdb1 /dev/sdd1
mdadm: /dev/sdb1 not large enough to join array

Here is what the array looks like now.

root@debian-nas:~# mdadm --detail --verbose /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Sat Mar 26 21:13:23 2016
     Raid Level : raid10
     Array Size : 5860268032 (5588.79 GiB 6000.91 GB)
  Used Dev Size : 2930134016 (2794.39 GiB 3000.46 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Sun Mar 27 16:25:22 2016
          State : active
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=2
     Chunk Size : 512K

           Name : debian-nas:0  (local to host debian-nas)
           UUID : 86266375:bf94ae93:d449a171:aa746159
         Events : 13726

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync set-A   /dev/sda1
       1       8       33        1      active sync set-B   /dev/sdc1
       2       8       65        2      active sync set-A   /dev/sde1
       3       8       81        3      active sync set-B   /dev/sdf1

I'm trying to add /dev/sdb1 and /dev/sdd1 to /dev/md0

root@debian-nas:~# blockdev --report /dev/sda1
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  2048   512   512       2048   3000591916544   /dev/sda1

root@debian-nas:~# blockdev --report /dev/sdb1
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  2048   512  4096       2048   3000590934016   /dev/sdb1

root@debian-nas:~# blockdev --report /dev/sdc1
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  2048   512  4096       2048   3000591450112   /dev/sdc1

root@debian-nas:~# blockdev --report /dev/sdd1
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  2048   512  4096       2048   3000590934016   /dev/sdd1

root@debian-nas:~# blockdev --report /dev/sde1
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  2048   512  4096       2048   3000591450112   /dev/sde1

root@debian-nas:~# blockdev --report /dev/sdf1
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  2048   512   512       2048   3000591916544   /dev/sdf1

Am I better off just moving the data off of the array, destroying it, then creating it with smaller partitions so that I can include the new drives?

Answer 1

If you don't need a partition table on /dev/sd[bd] (basically, if you don't need to boot from either drive), you can add the whole drives to the array; that should provide enough sectors:

mdadm /dev/md0 --add /dev/sdb /dev/sdd

Alternatively, you could create a new, degraded array on the new partitions, migrate your data to it and progressively move drives from the old array to the new one.

Answer 2

So I figured out a way to do this, using /dev/sdb1 and /dev/sdd1, without rebuilding the array, and without losing data.

This only works with a modern kernel (3.5 or greater) and mdadm version 3.3 or greater.

Unmount the array.

umount /dev/md0

Run a filesystem check. Must be done before the next step.

e2fsck -f /dev/md0

Resize the filesystem to 5 TB.

resize2fs /dev/md0 5000G

Run another check on the filesystem. Make sure we didn't mess up so far.

e2fsck -f /dev/md0

Shrink the array to just over 5 TB. The size parameter here is how much space to use on each disk in the array. This should be (size of fs / number of mirrors) + some small extra. In my case, I made the filesystem 5 TB, I have 4 disks in a raid 10, so I have a stripe of 2 mirrors. THIS MUST GIVE A TOTAL LARGER THAN YOUR FILESYSTEM OR YOU WILL LOSE DATA.

mdadm --verbose --grow --size=2685403136 /dev/md0 # 2.5TB + 1MB

Check the filesystem again.

e2fsck -f /dev/md0

At this point, we can add our smaller drives to the raid 10, but first, do checks to make sure your data is good. Don't just rely on the filesystem check. Actually verify your data is good.

Now we can add the new drives!

mdadm --add /dev/md0 /dev/sdf1 /dev/sdg1

And grow the array to use them.

mdadm --verbose --grow --raid-devices=6 /dev/md0

At this point, doing a cat /proc/mdstat should show something like the following:

Personalities : [raid6] [raid5] [raid4] [raid0] [raid10]
md0 : active raid10 sdd1[5] sdb1[4] sdf1[3] sde1[2] sdc1[1] sda1[0]
      5370806272 blocks super 1.2 512K chunks 2 near-copies [6/6] [UUUUUU]
      [>....................]  reshape =  2.8% (151477952/5370806272) finish=458.7min speed=189604K/sec
      bitmap: 11/31 pages [44KB], 131072KB chunk

unused devices: <none>

Do another filesystem check after the reshape is complete. (I'm paranoid, I'm going to check it every step of the way.)

e2fsck -f /dev/md0

Now grow the size of the array back to the max that your drives will support.

mdadm --verbose --grow --size=max /dev/md0

Then resize the filesystem.

resize2fs /dev/md0

And check it once again.

e2fsck -f /dev/md0

Finally, we can mount the new expanded filesystem.

mount /dev/md0 /mnt/raid10