2

I have an 8-disk 32TB RAID6 mdadm software RAID array. I started the computer today and it won't start the array. It has the common symptom of marking the array inactive and all 8 drives as spares. I can dd off all the array disks, so the hardware seems to be sound. smartctl also seems happy.

Having attempted due diligence and searched for existing answers, I noticed two things that I find peculiar.

  1. mdadm --examine --scan returns two devices named /dev/md/1
  2. what I think is the smoking gun. The first line of cat /proc/mdstat says Personalities: [raid1]. That's it. Both /dev/md0 and /dev/md1 are RAID1 arrays. I believe that this line should list a whole bunch of things, including RAID6. I'm hoping that this is why mdadm --scan --assemble is saying 1 drive is not enough to start the array.

Does this sound like it might actually be what's wrong, and if so, any ideas on hw to fix it? I haven't intentionally changed kernels, installed patches, rebuilt a ramdisk or anything else like that, and I don't think Debian will do that on me automatically. I have had a power fail or two that could have corrupted something, but the array has worked since then and everything (even the FS) was clean. Trying to assemble by UUID had the same effect.

Thanks for any suggestions. Here is some diagnostic output.

$ cat /proc/mdstat
Personalities : [raid1]
md2 : inactive sde1[0](S) sdd1[7](S) sdc1[6](S) sdb1[5](S) sda1[4](S) sdh1[3](S) sdg1[2](S) sdf1[1](S)
      31255091772 blocks super 1.2

md1 : active raid1 sdi3[0] sdj3[2]
      675801920 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sdi1[0] sdj1[2]
      48794496 blocks super 1.2 [2/2] [UU]

unused devices: <none>

$ mdadm --detail --scan
mdadm: cannot open /dev/md/2: No such file or directory
ARRAY /dev/md/0 metadata=1.2 name=ferret:0 UUID=1e5230d9:3f75c64e:d0885b53:0149f5aa
ARRAY /dev/md/1 metadata=1.2 name=ferret:1 UUID=663cf35c:b6cd2fe7:270e1729:bbf21ed2

$ mdadm --examine --scan
ARRAY /dev/md/0 metadata=1.2 UUID=1e5230d9:3f75c64e:d0885b53:0149f5aa name=ferret:0
ARRAY /dev/md/1 metadata=1.2 UUID=663cf35c:b6cd2fe7:270e1729:bbf21ed2 name=ferret:1
ARRAY /dev/md/1 metadata=1.2 UUID=67a454f1:4fe380f1:927069d6:2dc58580 name=ferret:1

$ cat /etc/mdadm/mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays

# This file was auto-generated on Sun, 08 Feb 2015 10:23:54 -0600
# by mkconf 3.2.5-5

ARRAY /dev/md/0 metadata=1.2 UUID=1e5230d9:3f75c64e:d0885b53:0149f5aa name=ferret:0
ARRAY /dev/md/1 metadata=1.2 UUID=663cf35c:b6cd2fe7:270e1729:bbf21ed2 name=ferret:1
ARRAY /dev/md/2 metadata=1.2 UUID=67a454f1:4fe380f1:927069d6:2dc58580 name=ferret:1

$ mdadm --stop /dev/md2
mdadm: stopped /dev/md2

$ mdadm --scan --assemble
mdadm: /dev/md/2 assembled from 1 drive - not enough to start the array.

Update: I ran a few things to see if they'd help. They changed things, but I wouldn't say helped. The additional personalities now appear in /proc/mdstat, but the array still comes up inactive.

$ modprobe raid456
$ update-initramfs -u
$ init 6

$ cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md2 : inactive sde1[0]
      3906886471 blocks super 1.2

md1 : active raid1 sdi3[0] sdj3[2]
      675801920 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sdi1[0] sdj1[2]
      48794496 blocks super 1.2 [2/2] [UU]

$ blkid
unused devices: <none>
/dev/md0: UUID="d913e7ec-96f5-4067-8255-8dd582646f2d" TYPE="ext4"
/dev/md1: UUID="233378f0-cefa-4197-a659-f8e7308a9270" TYPE="ext4"
/dev/sda1: UUID="67a454f1-4fe3-80f1-9270-69d62dc58580" UUID_SUB="5bb7ca2e-b9dd-7ce3-fa15-4f0f79536371" LABEL="ferret:1" TYPE="linux_raid_member"
/dev/sdb1: UUID="67a454f1-4fe3-80f1-9270-69d62dc58580" UUID_SUB="1150240c-c20f-603a-47d5-2547e603bd91" LABEL="ferret:1" TYPE="linux_raid_member"
/dev/sdc1: UUID="67a454f1-4fe3-80f1-9270-69d62dc58580" UUID_SUB="70296382-b69b-a059-db37-10604c4f9757" LABEL="ferret:1" TYPE="linux_raid_member"
/dev/sdd1: UUID="67a454f1-4fe3-80f1-9270-69d62dc58580" UUID_SUB="b0dd56bc-ca41-b4a0-97ae-18961c23ba5c" LABEL="ferret:1" TYPE="linux_raid_member"
/dev/sde1: UUID="67a454f1-4fe3-80f1-9270-69d62dc58580" UUID_SUB="0a044073-49b0-b90e-653b-6c9427ef3cd4" LABEL="ferret:1" TYPE="linux_raid_member"
/dev/sdf1: UUID="67a454f1-4fe3-80f1-9270-69d62dc58580" UUID_SUB="2a49c7be-dc87-0d33-3d49-055d4fd87c48" LABEL="ferret:1" TYPE="linux_raid_member"
/dev/sdg1: UUID="67a454f1-4fe3-80f1-9270-69d62dc58580" UUID_SUB="417c9661-aa46-6f3a-9019-65381c4deb6a" LABEL="ferret:1" TYPE="linux_raid_member"
/dev/sdh1: UUID="67a454f1-4fe3-80f1-9270-69d62dc58580" UUID_SUB="eb049051-cc6a-5910-290b-f1f5ffead2ac" LABEL="ferret:1" TYPE="linux_raid_member"
/dev/sdi1: UUID="1e5230d9-3f75-c64e-d088-5b530149f5aa" UUID_SUB="9fea4948-f42b-8386-9256-b6572b0adfa2" LABEL="ferret:0" TYPE="linux_raid_member"
/dev/sdj1: UUID="1e5230d9-3f75-c64e-d088-5b530149f5aa" UUID_SUB="57c47321-3c0c-f213-8c76-e4ed95961162" LABEL="ferret:0" TYPE="linux_raid_member"
/dev/sdi2: UUID="86fdd0b2-8ec9-4161-ad64-887ab5181066" TYPE="swap"
/dev/sdi3: UUID="663cf35c-b6cd-2fe7-270e-1729bbf21ed2" UUID_SUB="28bfe8c8-51a2-7fa1-9b17-28915145fed5" LABEL="ferret:1" TYPE="linux_raid_member"
/dev/sdj3: UUID="663cf35c-b6cd-2fe7-270e-1729bbf21ed2" UUID_SUB="9a47fa95-b416-53c0-5d5c-e2ad886154d3" LABEL="ferret:1" TYPE="linux_raid_member"

If I run mdadm --examine on all of the component disks, all except sde say:

Array State: AAAAAAAA ('A' == active, '.' == missing)

sde strangely shows

Array State: A....... ('A' == active, '.' == missing)

This is likely because /dev/md2 is started, though inactive, and thinks sde is inside. Stopping md2 and trying to assemble it via UUID still does not work.

Thanks!

Adding the --examine output.

/dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 67a454f1:4fe380f1:927069d6:2dc58580
           Name : ferret:1  (local to host ferret)
  Creation Time : Tue May  6 16:50:21 2014
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 7813772943 (3725.90 GiB 4000.65 GB)
     Array Size : 23441317632 (22355.38 GiB 24003.91 GB)
  Used Dev Size : 7813772544 (3725.90 GiB 4000.65 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 5bb7ca2e:b9dd7ce3:fa154f0f:79536371

    Update Time : Thu May 25 17:40:42 2017
       Checksum : 69790897 - correct
         Events : 71

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 4
   Array State : AAAAAAAA ('A' == active, '.' == missing)
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 67a454f1:4fe380f1:927069d6:2dc58580
           Name : ferret:1  (local to host ferret)
  Creation Time : Tue May  6 16:50:21 2014
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 7813772943 (3725.90 GiB 4000.65 GB)
     Array Size : 23441317632 (22355.38 GiB 24003.91 GB)
  Used Dev Size : 7813772544 (3725.90 GiB 4000.65 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 1150240c:c20f603a:47d52547:e603bd91

    Update Time : Thu May 25 17:40:42 2017
       Checksum : f5e64310 - correct
         Events : 71

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 5
   Array State : AAAAAAAA ('A' == active, '.' == missing)
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 67a454f1:4fe380f1:927069d6:2dc58580
           Name : ferret:1  (local to host ferret)
  Creation Time : Tue May  6 16:50:21 2014
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 7813772943 (3725.90 GiB 4000.65 GB)
     Array Size : 23441317632 (22355.38 GiB 24003.91 GB)
  Used Dev Size : 7813772544 (3725.90 GiB 4000.65 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 70296382:b69ba059:db371060:4c4f9757

    Update Time : Thu May 25 17:40:42 2017
       Checksum : 6a2a565f - correct
         Events : 71

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 6
   Array State : AAAAAAAA ('A' == active, '.' == missing)
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 67a454f1:4fe380f1:927069d6:2dc58580
           Name : ferret:1  (local to host ferret)
  Creation Time : Tue May  6 16:50:21 2014
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 7813772943 (3725.90 GiB 4000.65 GB)
     Array Size : 23441317632 (22355.38 GiB 24003.91 GB)
  Used Dev Size : 7813772544 (3725.90 GiB 4000.65 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : b0dd56bc:ca41b4a0:97ae1896:1c23ba5c

    Update Time : Thu May 25 17:40:42 2017
       Checksum : 265cfb41 - correct
         Events : 71

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 7
   Array State : AAAAAAAA ('A' == active, '.' == missing)
/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 67a454f1:4fe380f1:927069d6:2dc58580
           Name : ferret:1  (local to host ferret)
  Creation Time : Tue May  6 16:50:21 2014
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 7813772943 (3725.90 GiB 4000.65 GB)
     Array Size : 23441317632 (22355.38 GiB 24003.91 GB)
  Used Dev Size : 7813772544 (3725.90 GiB 4000.65 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 0a044073:49b0b90e:653b6c94:27ef3cd4

    Update Time : Sat Jun  3 22:07:36 2017
       Checksum : c11904ee - correct
         Events : 74

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 0
   Array State : A....... ('A' == active, '.' == missing)
/dev/sdf1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 67a454f1:4fe380f1:927069d6:2dc58580
           Name : ferret:1  (local to host ferret)
  Creation Time : Tue May  6 16:50:21 2014
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 7813772943 (3725.90 GiB 4000.65 GB)
     Array Size : 23441317632 (22355.38 GiB 24003.91 GB)
  Used Dev Size : 7813772544 (3725.90 GiB 4000.65 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 2a49c7be:dc870d33:3d49055d:4fd87c48

    Update Time : Thu May 25 17:40:42 2017
       Checksum : 6dd5fc9f - correct
         Events : 71

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 1
   Array State : AAAAAAAA ('A' == active, '.' == missing)
/dev/sdg1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 67a454f1:4fe380f1:927069d6:2dc58580
           Name : ferret:1  (local to host ferret)
  Creation Time : Tue May  6 16:50:21 2014
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 7813772943 (3725.90 GiB 4000.65 GB)
     Array Size : 23441317632 (22355.38 GiB 24003.91 GB)
  Used Dev Size : 7813772544 (3725.90 GiB 4000.65 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 417c9661:aa466f3a:90196538:1c4deb6a

    Update Time : Thu May 25 17:40:42 2017
       Checksum : 15d533a5 - correct
         Events : 71

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 2
   Array State : AAAAAAAA ('A' == active, '.' == missing)
/dev/sdh1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 67a454f1:4fe380f1:927069d6:2dc58580
           Name : ferret:1  (local to host ferret)
  Creation Time : Tue May  6 16:50:21 2014
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 7813772943 (3725.90 GiB 4000.65 GB)
     Array Size : 23441317632 (22355.38 GiB 24003.91 GB)
  Used Dev Size : 7813772544 (3725.90 GiB 4000.65 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : eb049051:cc6a5910:290bf1f5:ffead2ac

    Update Time : Thu May 25 17:40:42 2017
       Checksum : db2c6fee - correct
         Events : 71

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 3
   Array State : AAAAAAAA ('A' == active, '.' == missing)
  • lsmod? modprobe raid5 or something like it should provide personalities for raid4/5/6. If it's not just a missing module, also show mdadm --examine – frostschutz Jun 10 '17 at 22:21
  • What is the exact command and result when you try to start it? – psusi Jun 11 '17 at 2:23
  • mdadm --assemble /dev/md2 mdadm: /dev/md2 assembled from 1 drive - not enough to start the array. – byrontx Jun 11 '17 at 22:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.