4

I have 4x 3TB WD RED disks that are in mdadm RAID5, which should total an array size of 9TB usable space, however I only have 6TB.

I have confirmed all disks are in the array, the only strange thing is that fdisk returns different partition IDs, head sizes, and sector sizes, etc. for each of the drives. As these drives were added at different times it's possible I followed a different tutorial and this lead to some mishap.

I of course don't want to start trying things willy-nilly, as I don't want to lose data. So I'm hoping it's an easy fix and just something that I'm missing. Once I know which disk is at fault I can re-add this to the array correctly.

root@bobserv:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sdb1[5] sdc1[3] sdd1[1] sde1[4]
      6442053120 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]

unused devices: none
root@bobserv:~# mdadm --detail /dev/md/0
/dev/md/0:
        Version : 1.2
  Creation Time : Thu Oct 17 20:19:10 2013
     Raid Level : raid5
     Array Size : 6442053120 (6143.62 GiB 6596.66 GB)
  Used Dev Size : 2147351040 (2047.87 GiB 2198.89 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Tue Feb 11 09:36:33 2014
          State : active
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : bob:0
           UUID : aa43b63e:1c359341:077e94c8:36875aaf
         Events : 84216

    Number   Major   Minor   RaidDevice State
       4       8       65        0      active sync   /dev/sde1
       1       8       49        1      active sync   /dev/sdd1
       3       8       33        2      active sync   /dev/sdc1
       5       8       17        3      active sync   /dev/sdb1
root@bobserv:~# fdisk -l

Disk /dev/sda: 85.9 GB, 85899345920 bytes
255 heads, 63 sectors/track, 10443 cylinders, total 167772160 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00045481

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048   165675007    82836480   83  Linux
/dev/sda2       165677054   167770111     1046529    5  Extended
/dev/sda5       165677056   167770111     1046528   82  Linux swap / Solaris

WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdb: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x79c7d1c7

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1  4294967295  2147483647+  ee  GPT

Disk /dev/sdc: 3000.6 GB, 3000592982016 bytes
90 heads, 3 sectors/track, 21705678 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x6b7a81a1

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1            2048  4294967294  2147482623+  fd  Linux RAID autodetect

Disk /dev/sdf: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00083e19

   Device Boot      Start         End      Blocks   Id  System
/dev/sdf1            2048   123844607    61921280   83  Linux
/dev/sdf2       123846654   156248063    16200705    5  Extended
/dev/sdf5       123846656   156248063    16200704   82  Linux swap / Solaris

WARNING: GPT (GUID Partition Table) detected on '/dev/sde'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sde: 3000.6 GB, 3000592982016 bytes
255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1  4294967295  2147483647+  ee  GPT

Disk /dev/sdd: 3000.6 GB, 3000592982016 bytes
90 heads, 3 sectors/track, 21705678 cylinders, total 5860533168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x83a606dd

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1            2048  4294967294  2147482623+  fd  Linux RAID autodetect

Disk /dev/md0: 6596.7 GB, 6596662394880 bytes
2 heads, 4 sectors/track, 1610513280 cylinders, total 12884106240 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 524288 bytes / 1048576 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table
root@bobserv:~#

EDIT: I have also unmounted the array and tried both fsck.ext4 -f /dev/md0 & resize2fs /dev/md0 to no avail.

# parted -l
Model: VMware, VMware Virtual S (scsi)
Disk /dev/sda: 85.9GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type      File system     Flags
 1      1049kB  84.8GB  84.8GB  primary   ext4            boot
 2      84.8GB  85.9GB  1072MB  extended
 5      84.8GB  85.9GB  1072MB  logical   linux-swap(v1)


Model: VMware, VMware Virtual S (scsi)
Disk /dev/sdb: 3001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
 1      1049kB  3001GB  3001GB  ntfs         primary


Model: VMware, VMware Virtual S (scsi)
Disk /dev/sdc: 3001GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  2199GB  2199GB  primary               raid


Model: VMware, VMware Virtual S (scsi)
Disk /dev/sdd: 3001GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  2199GB  2199GB  primary               raid


Model: VMware, VMware Virtual S (scsi)
Disk /dev/sde: 3001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
 1      1049kB  3001GB  3001GB               primary


Model: VMware, VMware Virtual S (scsi)
Disk /dev/sdf: 500GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type      File system     Flags
 1      1049kB  63.4GB  63.4GB  primary   ext4
 2      63.4GB  80.0GB  16.6GB  extended
 5      63.4GB  80.0GB  16.6GB  logical   linux-swap(v1)


Model: Linux Software RAID Array (md)
Disk /dev/md0: 6597GB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End     Size    File system  Flags
 1      0.00B  6597GB  6597GB  ext4


root@kierserv:/media/store/TV Shows/Bob's Burgers#
4
  • can you give the output from parted -l? Also it is kind of strange to have some partitions with Id ee and some with fd (but that did not seem to confuse mdadm). Resizing does not help until the output from mdadm tells you have > 8Gb)
    – Anthon
    Feb 11, 2014 at 10:53
  • @Anthon, added above as per your request
    – Lemon
    Feb 11, 2014 at 11:01
  • Now you can see that the sdc1 and sdd1 are smaller than sdb1 and sde1, and mdadm can only work with raid5 using the biggest part of a partition availlabe on all contribution partitions (which is 2Gb hence 3x2Gb -> 6Gb space and 2Gb for spare)
    – Anthon
    Feb 11, 2014 at 11:07
  • Do you already have data on the array?
    – Anthon
    Feb 11, 2014 at 11:08

1 Answer 1

8

fdisk is the wrong tool for disks >2TB. Use parted or gdisk instead.

It appears that /dev/sdc1 and /dev/sdd1 are 2TB partitions, so that's what limits your array size. For the other disks, they have GPT so I assume they are 3TB already, but you should check.

Basically you have to stop the array, enlarge each partition to 3TB (without changing the starting offset), then start it again and follow it up with a grow:

mdadm --grow /dev/md0 --size=max

If you can't stop the array, you'll have to fail each 2TB partition individually, repartition and re-add it. This might go faster if you add a write-intent bitmap first.

mdadm --grow /dev/md0 --bitmap=internal

Then for each disk individually,

mdadm /dev/md0 --fail /dev/disk1 # check mdstat for [UUUU] first
mdadm /dev/md0 --remove /dev/disk1
parted /dev/disk -- mklabel gpt mkpart primary 1mib -1mib
mdadm /dev/md0 --re-add /dev/disk1
mdadm --wait /dev/md0 # must wait for sync

Once that's done you can remove the bitmap again (keeping it may harm performance).

mdadm --grow /dev/md0 --bitmap=none
mdadm --grow /dev/md0 --size=max

Finally do your resize2fs or whatever.

6
  • I've added fdisk -l output in my OP, can you confirm that your instructions are still valid before I proceed? I can indeed stop the array.
    – Lemon
    Feb 11, 2014 at 11:04
  • @Lemon: yes, just resize the [sdc/sdd] partitions. they start at 1MiB offset so the parted command from my answer should work. Oh, and you should probably add toggle 1 raid to the parted line, if you have a Windows where such flags might matter. Feb 11, 2014 at 11:09
  • @frostschutz +1 I am curious how you did see from the original output that sdc1 and sdd1 were at fault. I did suspect that at least one drive was only 2Tb, but could not determine that from the fdisk -l output (not even while comparing this with my own 4x3Tb Raid5 array). Did you see that in one of the other outputs?
    – Anthon
    Feb 11, 2014 at 11:13
  • @Anthon: the fdisk -l output was fine for those disks that were not using GPT (i.e. sdc/sdd). Feb 11, 2014 at 11:14
  • Ok, so you deduced it from them not having been GPT formatted, I thought you did see it from some number I overlooked. Makes sense, thanks.
    – Anthon
    Feb 11, 2014 at 11:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .