I originally created a software RAID using a controller that could only address 2TB per disk. The disks are 3 TB disks. It worked fine, but only used the first 2TB of each disk.

I have now changed the controller to see the full 3TB. I would therefore like /dev/md0 to use the last 1 TB, too.

I have tried:

# mdadm --grow /dev/md0 --size=max
mdadm: component size of /dev/md0 has been set to 2147479552K

But as you can see it only sees the 2TB. If I try forcing it higher:

# mdadm --grow /dev/md0 --size=2147483648
mdadm: Cannot set device size for /dev/md0: No space left on device

So somehow the system can see the disks are 3TB (in /proc/partitions), but the RAID cannot see them as 3TB.

mdadm details:

# mdadm --detail /dev/md0
    Version : 1.2
Creation Time : Fri Mar  2 15:14:46 2012
 Raid Level : raid6
 Array Size : 38654631936 (36863.93 GiB 39582.34 GB)
Used Dev Size : 2147479552 (2048.00 GiB 2199.02 GB)
Raid Devices : 20
Total Devices : 21
Persistence : Superblock is persistent

Update Time : Wed Apr 25 19:47:09 2012
      State : active
Active Devices : 20
Working Devices : 21
Failed Devices : 0
Spare Devices : 1

     Layout : left-symmetric
 Chunk Size : 4096K

       Name : node5:1
       UUID : 8603c3df:b740ba22:8c9c82fd:a18b3133
     Events : 845

Number   Major   Minor   RaidDevice State
  20      65       32        0      active sync   /dev/sds
   1      65       64        1      active sync   /dev/sdu
   2      65       80        2      active sync   /dev/sdv
   3      65       96        3      active sync   /dev/sdw
   4       8      192        4      active sync   /dev/sdm
   5       8       32        5      active sync   /dev/sdc
   6       8       48        6      active sync   /dev/sdd
   7       8       64        7      active sync   /dev/sde
   8       8       80        8      active sync   /dev/sdf
   9       8       96        9      active sync   /dev/sdg
  10       8      112       10      active sync   /dev/sdh
  11       8      128       11      active sync   /dev/sdi
  12       8      144       12      active sync   /dev/sdj
  13       8      160       13      active sync   /dev/sdk
  14       8      176       14      active sync   /dev/sdl
  15       8      208       15      active sync   /dev/sdn
  16       8      224       16      active sync   /dev/sdo
  17       8      240       17      active sync   /dev/sdp
  18      65        0       18      active sync   /dev/sdq
  19      65       16       19      active sync   /dev/sdr

  21      65       48        -      spare   /dev/sdt

Disk sizes:

# cat /proc/partitions |grep 2930266584
   8       48 2930266584 sdd
   8       32 2930266584 sdc
   8      112 2930266584 sdh
   8       96 2930266584 sdg
   8       80 2930266584 sdf
   8      128 2930266584 sdi
   8      176 2930266584 sdl
   8       64 2930266584 sde
  65       32 2930266584 sds
   8      192 2930266584 sdm
   8      144 2930266584 sdj
  65       80 2930266584 sdv
   8      224 2930266584 sdo
   8      208 2930266584 sdn
   8      160 2930266584 sdk
   8      240 2930266584 sdp
  65        0 2930266584 sdq
  65       64 2930266584 sdu
  65       16 2930266584 sdr
  65       48 2930266584 sdt
  65       96 2930266584 sdw


# mdadm --version
mdadm - v3.1.4 - 31st August 2010
# uname -a
Linux lemaitre 3.2.0-0.bpo.1-amd64 #1 SMP Sat Feb 11 08:41:32 UTC 2012 x86_64 GNU/Linux

The RAID6 uses the full disks (i.e. no partitions)

This morning the system crashed. After a reboot the system did not find the RAID (which was terrible). All disks showed up as spares (S):

cat /proc/mdstat 
Personalities : 
md0 : inactive sdr[20](S) sds[21](S) sdq[18](S) sdp[17](S) sdo[16](S) sdn[15](S) sdl[14](S) sdk[13](S) sdj[12](S) sdi[11](S) sdh[10](S) sdg[9](S) sdf[8](S) sde[7](S) sdd[6](S) sdc[5](S) sdm[4](S) sdv[3](S) sdu[2](S) sdt[1](S)
      42949652460 blocks super 1.2

Even here it is clear that mdadm did not find the 3 TB size.

I ran mdadm --stop /dev/md0. Removed the entry in /etc/mdadm/mdadm.conf. Ran mdadm -A --scan --force, which caused the RAID to come online and rebuild.

  • You are using 20 whole disks for RAID6 - correct? There is no partition on each disk? What does fdisk -l /dev/sdd show? What OS are you using, what is the mdadm-version? – Nils Apr 25 '12 at 20:45

I poked around /sys and got a lot closer to the answer.

# cd /sys/block/md0/md
# cat component_size

That agrees with what we have seen before. But this:

# grep . dev-sd*/size

seems to explain why the RAID sees the wrong size: Most of the drives shows up as 2TB while the 2 that have been replaced shows up as 3TB. All the drives are the same model, so let us see, if we can change the perceived size:

# parallel echo 2930265560 \> ::: dev-sd*/size
# grep . dev-sd*/size

Voila. Component_size is still small, though:

# cat component_size

Maybe it can be changed by mdadm:

# mdadm --grow /dev/md0 --size=max

This unfortunately locks up mdadm and subsequent access to /dev/md0 is blocked. And so is access to component_size:

# cat component_size   # This blocks

Sucks. But the good part is that in syslog it says:

Apr 27 20:45:50 server kernel: [124731.725019] md0: detected capacity change from 39582343102464 to 54010589478912

The filesystem on /dev/md0 still ran.

After a reboot I had to do 'mdadm --grow /dev/md0 --size=max' again. Then wait for resync to complete. Again access to /dev/md0 was blocked. So yet another reboot, followed by xfs_growfs /dev/md0 and then the resize was complete.

| improve this answer | |
  • I thoroughly liked the parallel(1) plug, Ole :) Thanks for the investigation, just hit a similar case. – Michael Shigorin Apr 22 '13 at 9:56
  • 1
    I guess "mdadm --update=devicesize" would do it too, see man page: "This will cause mdadm to determine the maximum usable amount of space on each device and update the relevant field in the metadata." – rudimeier Aug 8 '14 at 12:39

I think during creation the size of the device was registered somewhere in the metadata. Changing the controller doesn't change the metadata.

Remove the spare from the md, then re-add it to the RAID set as a new drive. You probably have to remove the metadata (check man page for --zero-superblock or wipe the whole disk). If that works for a single disk, repeat the procedure for all remainind drives too. Then finally do the --grow.

Don't remove additional disks until the sync finished!!

| improve this answer | |
  • It is unclear to me if you have tested that this actually works. A rebuild takes around 1.5 days. Replacing all 20 drives will cause running degraded for a full month. When you write "If that works for a single disk" how do you test that it worked for the single disk? – Ole Tange Apr 26 '12 at 8:53
  • No I haven't tested this, but I've seen some md problems before and I have some feeling about how it works ... and fails. I don't have the drives with bad metadata lying around to test it. I'm aware of the long sync time, that is why I advise to use the spare. You have RAID6, which means 2 drives can fail and you have a spare. That means a total of 3 drives that can fail before you have a problem. You could check with a single drive, at least you know if this is your issue. I have only limited information about your setup. Don't pull drives from your RAID set if you're not comfortable doing it – jippie Apr 26 '12 at 19:40
  • Oh, I understand your question. mdadm will warn you that you're trying to add a device that is much larger than the other ones. It doesn't mind about a few bytes, but it warns you when it the difference is relatively large. – jippie Apr 26 '12 at 19:42
  • 1
    You can fail a drive, then remove it, then zero-superblock it to remove the metadata. As your RAID set is in use you can either add it as a spare, which means it won't sync until another drive fails (that is normal behavior) but it only works for a single drive OR you add it and it requires a sync. I know it takes forever. Or you accept the fact that you cannot addres all storage available. One of the advantages of using a partition table which you didn't. Not using a partition table is not bad, but in this particular case would have helped you. – jippie Apr 26 '12 at 21:01
  • 1
    See my comment above: "mdadm will warn you that you're trying to add a device that is much larger than the other ones." – jippie Apr 27 '12 at 5:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.