Raid array 'clean, degraded'?

Question

Today I noticed that there are bunch of messages complaining about the RAID array (it's a software RAID10), so I started looking into it but need help because I'm unsure if I interpret the status output correctly (I've kinda forgotten the actual RAID set-up because the machine is at a remote location and I configured it about a year or two ago)... if I remember correctly the system was suppose to have 8x 2TB disks, but that's about all I can remember.

System mail:

 N 14 root@edmedia.loca  Wed May 25 21:30   32/1059  Fail event on /dev/md/0:EDMedia
 N 15 root@edmedia.loca  Thu May 26 06:25   30/1025  DegradedArray event on /dev/md/0:EDMedia
 N 16 root@edmedia.loca  Thu May 26 06:25   30/1025  SparesMissing event on /dev/md/0:EDMedia

The bit that's specifically confusing me, now that I'm looking at the outputs, is this:

Number   Major   Minor   RaidDevice State
   0       0        0        0      removed

Does it mean that a disk has been removed (or that it dropped from the array)? Should I try re-adding '/dev/sda1' to it? And is there any way I can tell that '/dev/sda1' was part of '/dev/md0' without adding a partitioned disk in-use by something, only to make things worse?

---

Status outputs:

---

'mdadm -D /dev/md0' output:

/dev/md0:
        Version : 1.2
  Creation Time : Mon Feb  8 23:15:33 2016
     Raid Level : raid10
     Array Size : 2197509120 (2095.71 GiB 2250.25 GB)
  Used Dev Size : 1465006080 (1397.14 GiB 1500.17 GB)
   Raid Devices : 3
  Total Devices : 2
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Thu Sep  1 19:54:05 2016
          State : clean, degraded
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

         Layout : near=2
     Chunk Size : 512K

           Name : EDMEDIA:0
           UUID : 6ebf98c8:d52a13f0:7ab1bffb:4dbe22b6
         Events : 4963861

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       17        1      active sync   /dev/sdb1
       2       8       33        2      active sync   /dev/sdc1

'lsblk' output:

NAME                       MAJ:MIN RM   SIZE RO TYPE   MOUNTPOINT
sda                          8:0    0   1.4T  0 disk
└─sda1                       8:1    0   1.4T  0 part
sdb                          8:16   0   1.4T  0 disk
└─sdb1                       8:17   0   1.4T  0 part
  └─md0                      9:0    0     2T  0 raid10
    ├─md0p1                259:0    0   1.5M  0 md
    ├─md0p2                259:1    0 244.5M  0 md     /boot
    └─md0p3                259:2    0     2T  0 md
      ├─EDMedia--vg-root   253:0    0     2T  0 lvm    /
      └─EDMedia--vg-swap_1 253:1    0    16G  0 lvm    [SWAP]
sdc                          8:32   0   1.4T  0 disk
└─sdc1                       8:33   0   1.4T  0 part
  └─md0                      9:0    0     2T  0 raid10
    ├─md0p1                259:0    0   1.5M  0 md
    ├─md0p2                259:1    0 244.5M  0 md     /boot
    └─md0p3                259:2    0     2T  0 md
      ├─EDMedia--vg-root   253:0    0     2T  0 lvm    /
      └─EDMedia--vg-swap_1 253:1    0    16G  0 lvm    [SWAP]
sdd                          8:48   0   1.4T  0 disk
└─sdd1                       8:49   0   1.4T  0 part
sdj                          8:144  0 298.1G  0 disk
└─sdj1                       8:145  0 298.1G  0 part
sr0                         11:0    1  1024M  0 rom

'df' output:

Filesystem      1K-blocks       Used Available Use% Mounted on
/dev/dm-0      2146148144 1235118212 801988884  61% /
udev                10240          0     10240   0% /dev
tmpfs             1637644      17124   1620520   2% /run
tmpfs             4094104          0   4094104   0% /dev/shm
tmpfs                5120          0      5120   0% /run/lock
tmpfs             4094104          0   4094104   0% /sys/fs/cgroup
/dev/md0p2         242446      34463    195465  15% /boot

'watch -n1 cat /proc/mdstat' output:

Every 1.0s: cat /proc/mdstat                                                                                                                                                           Thu Sep  1 21:26:22 2016

Personalities : [raid10]
md0 : active raid10 sdb1[1] sdc1[2]
      2197509120 blocks super 1.2 512K chunks 2 near-copies [3/2] [_UU]
      bitmap: 16/17 pages [64KB], 65536KB chunk

unused devices: <none>

Answer 1

Seems that your raid10 array was configured to have 2 active drives plus one spare. The spare is missing.

This can have several reasons:

1. Maybe you removed the spare disk from the server
2. Maybe one drive died and the existing hot spare became active now after a rebuild.
3. Maybe the hot spare died before it could be ever used.
4. Maybe one drive (or cable) "was" broken at one time in past and has been automatically removed from the array.

You may check if your server has one broken disk which you don't even see anymore in lsblk output. Could also be that one of your other drives (sda1 or sdd1) was part of your array in past but is broken now. (It can't be sdj1 because it's too small ).

Remove all broken drives from the server.

To avoid the warnings re-add a hot spare drive (maybe one of the unused, non-broken ones) or configure your array to not have a hot-spare anymore. Be aware that in case 4 the probability that the same drive will fail again is high.

BTW to see what exactly happened in past you could grep the old logfiles for relevant messages.

Answer 2

Inspected the system logs as rudimeier suggested and found out there had been a power outage event back at May, after which the RAID array errors started popping up. Since this is a software RAID10 (1+0), I'm thankful only the spare disk flew out of the array instead of the whole array irreversibly crashing. After doing a couple HDD tests with the trusty old Hiren's boot CD and just for variety - Partition Wizard bootable... all suspicious disks checked out with no errors/issues.

I erased (with Partition Wizard bootable, so that the disk would be unformatted and unpartitioned) and then re-added the spare using:

mdadm --add /dev/md0 /dev/sda1