Only half of my drives assemble during a RAID recovery

Question

I have an 8 drive RAID I am trying to recover. The assemble only found 4 of the 8 drives. Why are the other drives not found?

me@mjf-svr:~$ cat /proc/mdstat

Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] 
md1 : inactive sdn[7](S) sdk[4](S) sdm[6](S) sdl[5](S) sdi[1](S) sdj[0](S) sdh[2](S) sdg[3](S)
      15627068096 blocks super 1.2

md127 : active raid5 sde1[1] sdf1[3] sdd1[0]
      1953522688 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]

unused devices: <none>

me@mjf-svr:~$ sudo mdadm --stop /dev/md1

[sudo] password for me: 
mdadm: stopped /dev/md1

me@mjf-svr:~$ sudo mdadm --examine /dev/sd[g-n]

/dev/sdg:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : e2f38846:949163bc:c50d752a:069b9e1d
           Name : mjf-svr:1
  Creation Time : Sun Feb 22 21:45:59 2015
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
  Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=1200 sectors
          State : active
    Device UUID : a32f5f14:7b91eeb6:c5319782:419cdf3f

    Update Time : Mon Mar 28 18:47:10 2016
       Checksum : 673e7d4e - correct
         Events : 6597

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdh:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : e2f38846:949163bc:c50d752a:069b9e1d
           Name : mjf-svr:1
  Creation Time : Sun Feb 22 21:45:59 2015
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
  Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=1200 sectors
          State : active
    Device UUID : f7f68c18:300fb414:18a46a9a:05c55ecf

    Update Time : Mon Mar 28 18:47:10 2016
       Checksum : 70845d6d - correct
         Events : 6597

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdi:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : e2f38846:949163bc:c50d752a:069b9e1d
           Name : mjf-svr:1
  Creation Time : Sun Feb 22 21:45:59 2015
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
  Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=1200 sectors
          State : active
    Device UUID : e6f99981:b73578d2:77ad1b95:ee472611

    Update Time : Mon Mar 28 18:47:10 2016
       Checksum : d3ce132a - correct
         Events : 6597

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdj:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : e2f38846:949163bc:c50d752a:069b9e1d
           Name : mjf-svr:1
  Creation Time : Sun Feb 22 21:45:59 2015
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
  Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=1200 sectors
          State : active
    Device UUID : 0a12d1bd:223f4f49:5d60861c:de9f902a

    Update Time : Mon Mar 28 18:47:10 2016
       Checksum : 27b13f8e - correct
         Events : 6597

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdk:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : e2f38846:949163bc:c50d752a:069b9e1d
           Name : mjf-svr:1
  Creation Time : Sun Feb 22 21:45:59 2015
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
  Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=1200 sectors
          State : clean
    Device UUID : 5e1f443d:d22cf4df:501573f1:0b5d9e68

    Update Time : Sun Apr  3 00:57:01 2016
       Checksum : 50c49b69 - correct
         Events : 6732

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 4
   Array State : ....AAAA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdl:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : e2f38846:949163bc:c50d752a:069b9e1d
           Name : mjf-svr:1
  Creation Time : Sun Feb 22 21:45:59 2015
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
  Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=1200 sectors
          State : clean
    Device UUID : afc36449:e71aa993:c9a8c3f6:ee6c37a7

    Update Time : Sun Apr  3 00:57:01 2016
       Checksum : 5483d12c - correct
         Events : 6732

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 5
   Array State : ....AAAA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdm:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : e2f38846:949163bc:c50d752a:069b9e1d
           Name : mjf-svr:1
  Creation Time : Sun Feb 22 21:45:59 2015
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
  Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=1200 sectors
          State : clean
    Device UUID : 67bf80fb:dc2cab8b:a743a9f1:8a535392

    Update Time : Sun Apr  3 00:57:01 2016
       Checksum : e4a36054 - correct
         Events : 6732

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 6
   Array State : ....AAAA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdn:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : e2f38846:949163bc:c50d752a:069b9e1d
           Name : mjf-svr:1
  Creation Time : Sun Feb 22 21:45:59 2015
     Raid Level : raid6
   Raid Devices : 8

 Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
  Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=1200 sectors
          State : clean
    Device UUID : 02c9a435:14df0bca:7d3174de:540f924d

    Update Time : Sun Apr  3 00:57:01 2016
       Checksum : 531c5c8 - correct
         Events : 6732

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 7
   Array State : ....AAAA ('A' == active, '.' == missing, 'R' == replacing)

me@mjf-svr:~$ sudo mdadm --assemble /dev/md1 /dev/sd[g-n]

mdadm: /dev/md1 assembled from 4 drives - not enough to start the array.

me@mjf-svr:~$ sudo mdadm --assemble --scan --uuid=e2f38846:949163bc:c50d752a:069b9e1d

    mdadm: /dev/md/mjf-svr:1 assembled from 4 drives - not enough to start the array.

me@mjf-svr:~$

Answer 1

Four of your devices have an event count of 6732 and think they're part of an eight-device RAID-6 array with four missing devices. The other four have an event count of 6597 and think they're part of an intact eight-device RAID-6 array. I'm guessing that on Mar 28 at 18:47:10 you had a controller failure or other problem that knocked half your drives offline.

I hope you've got a backup. You can attempt the Recovering a failed software RAID procedure from the Linux RAID wiki (the key step is to assemble the array with --force; the purpose of the rest of the procedure is to figure out in advance if this will save your data, or destroy it), but with the significant difference in event counts, I wouldn't expect it to succeed.