mdadm raid 6 array decrease error Cannot set new_offset for /dev/sda1

Question

I tried to decrease the size of a mdadm raid6 array from 10 to 8.
So I followed these steps :

pi@raspbian-x64:~ $ sudo mdadm --grow /dev/md0 --raid-devices=8
mdadm: this change will decrease the size of the array.
   use --grow --array-size first to truncate array.
   e.g. mdadm --grow /dev/md0 --array-size 2928887808

pi@raspbian-x64:~ $ sudo mdadm --grow /dev/md0 --array-size 2928887808

pi@raspbian-x64:~ $ sudo mdadm --grow /dev/md0 --raid-devices=8
mdadm: Cannot set new_offset for /dev/sda1

Do you know why mdadm can't set new_offset for /dev/sda1 ?

Here is the detail of my array (After using the commands) :

pi@raspbian-x64:~ $ sudo mdadm --detail /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Tue May 11 07:47:04 2021
        Raid Level : raid6
        Array Size : 2928887808 (2793.21 GiB 2999.18 GB)
     Used Dev Size : 488147968 (465.53 GiB 499.86 GB)
      Raid Devices : 10
     Total Devices : 8
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Mon Jul 18 02:18:12 2022
             State : clean, degraded
    Active Devices : 8
   Working Devices : 8
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : bitmap

              Name : pi1:0
              UUID : 4314f81c:4b2f90a0:db6089d4:c5958ee0
            Events : 147618

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
       2       8       49        2      active sync   /dev/sdd1
       -       0        0        3      removed
      10       8       65        4      active sync   /dev/sde1
       9       8       82        5      active sync   /dev/sdf2
       8       8       81        6      active sync   /dev/sdf1
       -       0        0        7      removed
       7       8        2        8      active sync   /dev/sda2
       6       8        1        9      active sync   /dev/sda1

Right now, there is 46% free space in the raid array. So it should be OK to decrease the array to 8 devices.

UPDATE 1 :

I want to decrease the number of devices from 10 to 7 :

newSize = devSize * (totalWantedDeviceCount - raid6deviceIntegrityCount)
newSize = 488147968 * (7 - 2)
newSize = 2440739840 (instead of 3905183744)

So I managed to decrease the RAID array filesystem size :

sudo resize2fs /dev/md0 2440739840K

(I had to fix many errors with sudo e2fsck -f /dev/md0 -y)

Next I decreased the size of the RAID array in mdadm :

sudo mdadm --grow /dev/md0 --array-size 2440739840

It worked :

pi@raspbian-x64:~ $ sudo mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
    Creation Time : Tue May 11 07:47:04 2021
        Raid Level : raid6
        Array Size : 2440739840 (2327.67 GiB 2499.32 GB)
    Used Dev Size : 488147968 (465.53 GiB 499.86 GB)
    Raid Devices : 10
    Total Devices : 10
    Persistence : Superblock is persistent

    Intent Bitmap : Internal

    Update Time : Mon Aug 15 20:57:47 2022
            State : clean
    Active Devices : 10
Working Devices : 10
    Failed Devices : 0
    Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : bitmap

            Name : pi1:0
            UUID : 4314f81c:4b2f90a0:db6089d4:c5958ee0
            Events : 205009

    Number   Major   Minor   RaidDevice State
    0       8        1        0      active sync   /dev/sda1
    1       8       33        1      active sync   /dev/sdc1
    2       8       81        2      active sync   /dev/sdf1
    5       8      129        3      active sync   /dev/sdi1
    10       8      113        4      active sync   /dev/sdh1
    11       8       49        5      active sync   /dev/sdd1
    8       8       97        6      active sync   /dev/sdg1
    4       8       17        7      active sync   /dev/sdb1
    7       8       66        8      active sync   /dev/sde2
    6       8       65        9      active sync   /dev/sde1

And I tried to remove 3 devices, I get the same error :

pi@raspbian-x64:~ $ sudo mdadm --grow /dev/md0 --raid-devices=7
mdadm: Cannot set new_offset for /dev/sda1

UPDATE 2 :

I didn't manage to decrease the array size, so I had to destroy it and recreate a new array from scratch...

UPDATE 3 :

I created a script to help generate the necessary commands to run to decrease the size of the RAID array with Qbert-Pacman trick : script on github.
Let me know if it works for anybody.

Answer 1

I hope this information is useful. While trying to reduce a raid4 array from 8 SSD drives to 4 SSD drives, I followed the same steps as you have documented.

• Reduced the size of the ext4 filesystem. resize2fs /filesystem 2929890816

• Reduced the size for the mdadm raid4 array. mdadm -G /dev/md0 --array-size 2929890816

• Tried reducing the mdadm raid4 device from 8 SSD drives to 4 SSD drives and received the same error (mdadm: Cannot set new_offset for /dev/sdX). mdadm -G /dev/md0 -l4 -n4

I resolved this issue by adding the additional option to allocate a backup file to hold temporary data during the reshaping process. The previous command was replaced with the following which resolved the mdadm raid4 shrink issue.

mdadm -G /dev/md0 -l4 -n4 --backup-file /var/tmp/md0-backup

The backup file is created and uses around 10752k to hold the temporary data. The following additional output follows execution of the command to provide confirmation of backup file allocation and usage.

mdadm: Need to backup 10752K of critical section.

Hope this information proves useful for future mdadm device reshaping tasks.