1

I'm trying to get a raid6 array to run, but it will not start.

Brief array history: This array was initially built with 6 disks (8TB each).

mdadm --create --verbose /dev/md1 --level=6 --raid-devices=6 /dev/sdb1 /dev/sde1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1

1 disk was added to grow the array

mdadm -v --grow --raid-devices=7 /dev/md1

then the partition was resized in gparted.

2 further disks were added to grow the array but partition has not be re-sized again. The array had been starting automatically on boot-up, but now will not start.

mdadm: failed to start array /dev/md1: Input/output error

Here are some other relevant outputs:

s:~$ mdadm --detail /dev/md1
/dev/md1:
           Version : 1.2
     Creation Time : Wed Aug 25 16:25:06 2021
        Raid Level : raid6
     Used Dev Size : 18446744073709551615
      Raid Devices : 9
     Total Devices : 8
       Persistence : Superblock is persistent

       Update Time : Wed Oct  6 16:45:06 2021
             State : active, FAILED, Not Started
    Active Devices : 8
   Working Devices : 8
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : unknown

              Name : Octavius:1  (local to host Octavius)
              UUID : 80bd1af7:20800c35:be64a577:8b62e937
            Events : 198308

    Number   Major   Minor   RaidDevice State
       -       0        0        0      removed
       -       0        0        1      removed
       -       0        0        2      removed
       -       0        0        3      removed
       -       0        0        4      removed
       -       0        0        5      removed
       -       0        0        6      removed
       -       0        0        7      removed
       -       0        0        8      removed

       -       8      177        5      sync   /dev/sdl1
       -       8      161        4      sync   /dev/sdk1
       -       8      145        3      sync   /dev/sdj1
       -       8      129        2      sync   /dev/sdi1
       -       8       97        1      sync   /dev/sdg1
       -       8       49        7      sync   /dev/sdd1
       -       8       33        0      sync   /dev/sdc1
       -       8       17        6      sync   /dev/sdb1

/dev/sda1 should be a member of this array but is missing. I don't know why there are all the removed devices.

s:~$ sudo mdadm --examine /dev/sda1
/dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 80bd1af7:20800c35:be64a577:8b62e937
           Name : Octavius:1  (local to host Octavius)
  Creation Time : Wed Aug 25 16:25:06 2021
     Raid Level : raid6
   Raid Devices : 9

 Avail Dev Size : 15627798528 (7451.92 GiB 8001.43 GB)
     Array Size : 54697251840 (52163.36 GiB 56009.99 GB)
  Used Dev Size : 15627786240 (7451.91 GiB 8001.43 GB)
    Data Offset : 251904 sectors
   Super Offset : 8 sectors
   Unused Space : before=251824 sectors, after=12288 sectors
          State : active
    Device UUID : 9bddd5dd:790156b1:7b8e38d3:37558974

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Oct  6 16:45:06 2021
  Bad Block Log : 512 entries available at offset 40 sectors
       Checksum : b23ecdf9 - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : spare
   Array State : AAAAAAAA. ('A' == active, '.' == missing, 'R' == replacing)
s:~$ sudo mdadm --examine /dev/sdb1
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 80bd1af7:20800c35:be64a577:8b62e937
           Name : Octavius:1  (local to host Octavius)
  Creation Time : Wed Aug 25 16:25:06 2021
     Raid Level : raid6
   Raid Devices : 9

 Avail Dev Size : 15627798528 (7451.92 GiB 8001.43 GB)
     Array Size : 54697251840 (52163.36 GiB 56009.99 GB)
  Used Dev Size : 15627786240 (7451.91 GiB 8001.43 GB)
    Data Offset : 251904 sectors
   Super Offset : 8 sectors
   Unused Space : before=251824 sectors, after=12288 sectors
          State : active
    Device UUID : ffa868e4:ee48f113:bd015c5c:7f92f378

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Oct  6 16:45:06 2021
  Bad Block Log : 512 entries available at offset 40 sectors
       Checksum : a8fd97fc - correct
         Events : 198308

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 6
   Array State : AAAAAAAA. ('A' == active, '.' == missing, 'R' == replacing)

The output from all other devices in the array is the same as for /dev/sdb1.

Any help or suggestions would be great and i can provide any other outputs that might be helpful.

0

So I think I worked out what was wrong, though i'm not sure how it happened.

I missed the fact that all dives were marked as spare:

:~$ sudo mdadm --stop /dev/md1
mdadm: stopped /dev/md1
:~$ cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md1 : inactive sdi1[2](S) sda1[9](S) sdc1[0](S) sdl1[5](S) sdb1[6](S) sdk1[4](S) sdj1[3](S) sdg1[1](S) sdd1[8](S)
      70325093376 blocks super 1.2

So i figured i'd bite the bullet (small risk of losing a small amount of non backed-up data), stop the array, then force assemly. The array is now re-building and is the data seems to be fine.

:~$ sudo mdadm --assemble --force /dev/md1 /dev/sda1 /dev/sdi1 /dev/sdc1 /dev/sdl1 /dev/sdb1 /dev/sdk1 /dev/sdj1 /dev/sdg1 /dev/sdd1 --verbose
mdadm: looking for devices for /dev/md1
mdadm: /dev/sda1 is identified as a member of /dev/md1, slot -1.
mdadm: /dev/sdi1 is identified as a member of /dev/md1, slot 2.
mdadm: /dev/sdc1 is identified as a member of /dev/md1, slot 0.
mdadm: /dev/sdl1 is identified as a member of /dev/md1, slot 5.
mdadm: /dev/sdb1 is identified as a member of /dev/md1, slot 6.
mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 4.
mdadm: /dev/sdj1 is identified as a member of /dev/md1, slot 3.
mdadm: /dev/sdg1 is identified as a member of /dev/md1, slot 1.
mdadm: /dev/sdd1 is identified as a member of /dev/md1, slot 7.
mdadm: Marking array /dev/md1 as 'clean'
mdadm: added /dev/sdg1 to /dev/md1 as 1
mdadm: added /dev/sdi1 to /dev/md1 as 2
mdadm: added /dev/sdj1 to /dev/md1 as 3
mdadm: added /dev/sdk1 to /dev/md1 as 4
mdadm: added /dev/sdl1 to /dev/md1 as 5
mdadm: added /dev/sdb1 to /dev/md1 as 6
mdadm: added /dev/sdd1 to /dev/md1 as 7
mdadm: no uptodate device for slot 8 of /dev/md1
mdadm: added /dev/sda1 to /dev/md1 as -1
mdadm: added /dev/sdc1 to /dev/md1 as 0
mdadm: /dev/md1 has been started with 8 drives (out of 9) and 1 spare.
:~$ cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md1 : active raid6 sdc1[0] sda1[9] sdd1[8] sdb1[6] sdl1[5] sdk1[4] sdj1[3] sdi1[2] sdg1[1]
      54697251840 blocks super 1.2 level 6, 512k chunk, algorithm 2 [9/8] [UUUUUUUU_]
      [>....................]  recovery =  0.0% (1019952/7813893120) finish=1276.6min speed=101997K/sec
      bitmap: 10/59 pages [40KB], 65536KB chunk

I had seen this as a suggestion to fix other similar issues, however the status of my array (active, FAILED, not started) was different to all examples i could find and i was not comfortable with the force option initially.

Hopefully this helps someone in the future...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.