0

I tried to decrease the size of a mdadm raid6 array from 10 to 8.
So I followed these steps :

pi@raspbian-x64:~ $ sudo mdadm --grow /dev/md0 --raid-devices=8
mdadm: this change will decrease the size of the array.
   use --grow --array-size first to truncate array.
   e.g. mdadm --grow /dev/md0 --array-size 2928887808

pi@raspbian-x64:~ $ sudo mdadm --grow /dev/md0 --array-size 2928887808

pi@raspbian-x64:~ $ sudo mdadm --grow /dev/md0 --raid-devices=8
mdadm: Cannot set new_offset for /dev/sda1

Do you know why mdadm can't set new_offset for /dev/sda1 ?

Here is the detail of my array (After using the commands) :

pi@raspbian-x64:~ $ sudo mdadm --detail /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Tue May 11 07:47:04 2021
        Raid Level : raid6
        Array Size : 2928887808 (2793.21 GiB 2999.18 GB)
     Used Dev Size : 488147968 (465.53 GiB 499.86 GB)
      Raid Devices : 10
     Total Devices : 8
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Mon Jul 18 02:18:12 2022
             State : clean, degraded
    Active Devices : 8
   Working Devices : 8
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : bitmap

              Name : pi1:0
              UUID : 4314f81c:4b2f90a0:db6089d4:c5958ee0
            Events : 147618

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
       2       8       49        2      active sync   /dev/sdd1
       -       0        0        3      removed
      10       8       65        4      active sync   /dev/sde1
       9       8       82        5      active sync   /dev/sdf2
       8       8       81        6      active sync   /dev/sdf1
       -       0        0        7      removed
       7       8        2        8      active sync   /dev/sda2
       6       8        1        9      active sync   /dev/sda1

Right now, there is 46% free space in the raid array. So it should be OK to decrease the array to 8 devices.

UPDATE 1 :

I want to decrease the number of devices from 10 to 7 :

newSize = devSize * (totalWantedDeviceCount - raid6deviceIntegrityCount)
newSize = 488147968 * (7 - 2)
newSize = 2440739840 (instead of 3905183744)

So I managed to decrease the RAID array filesystem size :

sudo resize2fs /dev/md0 2440739840K

(I had to fix many errors with sudo e2fsck -f /dev/md0 -y)

Next I decreased the size of the RAID array in mdadm :

sudo mdadm --grow /dev/md0 --array-size 2440739840

It worked :

pi@raspbian-x64:~ $ sudo mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
    Creation Time : Tue May 11 07:47:04 2021
        Raid Level : raid6
        Array Size : 2440739840 (2327.67 GiB 2499.32 GB)
    Used Dev Size : 488147968 (465.53 GiB 499.86 GB)
    Raid Devices : 10
    Total Devices : 10
    Persistence : Superblock is persistent

    Intent Bitmap : Internal

    Update Time : Mon Aug 15 20:57:47 2022
            State : clean
    Active Devices : 10
Working Devices : 10
    Failed Devices : 0
    Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : bitmap

            Name : pi1:0
            UUID : 4314f81c:4b2f90a0:db6089d4:c5958ee0
            Events : 205009

    Number   Major   Minor   RaidDevice State
    0       8        1        0      active sync   /dev/sda1
    1       8       33        1      active sync   /dev/sdc1
    2       8       81        2      active sync   /dev/sdf1
    5       8      129        3      active sync   /dev/sdi1
    10       8      113        4      active sync   /dev/sdh1
    11       8       49        5      active sync   /dev/sdd1
    8       8       97        6      active sync   /dev/sdg1
    4       8       17        7      active sync   /dev/sdb1
    7       8       66        8      active sync   /dev/sde2
    6       8       65        9      active sync   /dev/sde1

And I tried to remove 3 devices, I get the same error :

pi@raspbian-x64:~ $ sudo mdadm --grow /dev/md0 --raid-devices=7
mdadm: Cannot set new_offset for /dev/sda1

UPDATE 2 :

I didn't manage to decrease the array size, so I had to destroy it and recreate a new array from scratch...

UPDATE 3 :

I created a script to help generate the necessary commands to run to decrease the size of the RAID array with Qbert-Pacman trick : script on github.
Let me know if it works for anybody.

5
  • 1. Only do reshapes with optimal state arrays. You must first add replacements for two removed devices, properly sync an array, and only then perform the reshape. 2. You need to first reduce the "used device size"; did you do this properly? Commented Jul 18, 2022 at 6:10
  • I followed a guide, I set the devices to failed and removed them. And I am unable to reduce the array to 8 devices. If i readd the devices, I will be in the original state when I wanted to remove 2 devices.
    – Antoine
    Commented Jul 18, 2022 at 13:10
  • 1
    Which guide? It is very strange to see a guide which deliberately sets array into fully degraded mode before performing dangerous operations. Commented Jul 18, 2022 at 13:20
  • I got guides from these 2 links : delightlylinux.wordpress.com/2020/12/22/… documentation.suse.com/sles/12-SP4/html/SLES-all/…
    – Antoine
    Commented Jul 18, 2022 at 14:45
  • I am locked in the situation, you can look at the last update (3) where I reduced the file system size with success, I reduced the total RAID array size in mdadm with sucess, but reducing the RAID Array devices count still results to the same error.
    – Antoine
    Commented Aug 16, 2022 at 3:12

1 Answer 1

3

I hope this information is useful. While trying to reduce a raid4 array from 8 SSD drives to 4 SSD drives, I followed the same steps as you have documented.

  • Reduced the size of the ext4 filesystem. resize2fs /filesystem 2929890816

  • Reduced the size for the mdadm raid4 array. mdadm -G /dev/md0 --array-size 2929890816

  • Tried reducing the mdadm raid4 device from 8 SSD drives to 4 SSD drives and received the same error (mdadm: Cannot set new_offset for /dev/sdX). mdadm -G /dev/md0 -l4 -n4

I resolved this issue by adding the additional option to allocate a backup file to hold temporary data during the reshaping process. The previous command was replaced with the following which resolved the mdadm raid4 shrink issue.

mdadm -G /dev/md0 -l4 -n4 --backup-file /var/tmp/md0-backup

The backup file is created and uses around 10752k to hold the temporary data. The following additional output follows execution of the command to provide confirmation of backup file allocation and usage.

mdadm: Need to backup 10752K of critical section.

Hope this information proves useful for future mdadm device reshaping tasks.

2
  • Okay thnak you I can't test your proposition but I trust you. I put your answer as the solution. Hope it will help somebody here ;]
    – Antoine
    Commented May 16, 2023 at 22:59
  • When I had this problem I created a script to compute the size of the array and generate commands to run, I updated it with your remarks, you can take a look and let me know if it looks correct ^^ github.com/antlafarge/linux-scripts/blob/main/…
    – Antoine
    Commented May 16, 2023 at 23:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .