mdadm RAID10 - change broken disk

Question

Tonight my server stopped answering to ssh, ping, so I drove to the server room. Server did not respond to keyboard (even Ctrl-Alt-Del), there was no data on screen (it was black, no symbols on it), so I decided to reboot him with button on the case. After several attempts to start the system (it does not boot more than GRUB and black screen right after it), I opened BIOS setup, and saw that one disk of array (one of 4 in software mdadm RAID10) is crashed. In BIOS it was:

SATA Port 1         INTEL SSDSC2CW060A3 (60.0GB)
SATA Port 2         ST1000DM003-1CH162 (1000.2GB)
SATA Port 3         ST1000DM003-1CH162 (1000.2GB)
SATA Port 4         Not Present
SATA Port 5         ST1000DM003-1CH162 (1000.2GB)

I've disconnect this disk, and system had started, but, without array. Now I get new same disk, that is installed on the place of faulted one. But RAID 10 is not rebuilds automatically (as I thought to). When booting I see

The disk drive for UUID=a3462d0b-7261-48da-8e13-6de332e4de33 is not ready yet or not present
keys:Continue to wait, or Press S to skip mounting or M for manual recovery

The disk drive for /var/log is not ready yet or not present
keys:Continue to wait, or Press S to skip mounting or M for manual recovery

The disk drive for /mnt/vm is not ready yet or not present
keys:Continue to wait, or Press S to skip mounting or M for manual recovery 

/etc/mdadm/mdadm.conf

CREATE owner=root group=disk mode=0660 auto=yes
HOMEHOST <system>
MAILADDR root
ARRAY /dev/md/0 metadata=1.2 name=rockshock:0 UUID=02f2f445:de9ba929:f6a3e9da:249f8e42

/etc/fstab

UUID=6760d8f4-2816-4b54-8938-ab11b329705b /               ext4    noatime,errors=remount-ro 0       1
UUID=7593-A7EC  /boot/efi       vfat    defaults        0       1
UUID=a3462d0b-7261-48da-8e13-6de332e4de33   none    swap    sw  0   0

/dev/raid/logs  /var/log    ext4    noatime,noexec  0   2
/dev/raid/vm    /mnt/vm     ext4    noatime,noexec  0   2

Here is some data that I habe (should I provide any more?):

$ cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : inactive sdb[3](S) sdd[0](S) sde[1](S)
   2929894536 blocks super 1.2

unused devices: <none>

$ blkid

/dev/sdb: UUID="02f2f445-de9b-a929-f6a3-e9da249f8e42" UUID_SUB="917412df-140c-dbc6-68ad-57e981cbcff9" LABEL="rockshock:0" TYPE="linux_raid_member" 
/dev/sdc1: UUID="7593-A7EC" TYPE="vfat" 
/dev/sdc2: UUID="6760d8f4-2816-4b54-8938-ab11b329705b" TYPE="ext4" 
/dev/sdd: UUID="02f2f445-de9b-a929-f6a3-e9da249f8e42" UUID_SUB="0904482c-1a9a-1174-6936-805c8a466493" LABEL="rockshock:0" TYPE="linux_raid_member" 
/dev/sde: UUID="02f2f445-de9b-a929-f6a3-e9da249f8e42" UUID_SUB="9badcfe0-f4a4-cd0c-baa8-3d4ebc46c56e" LABEL="rockshock:0" TYPE="linux_raid_member" 

$ sudo mdadm --detail /dev/md0

mdadm: md device /dev/md0 does not appear to be active.

$ sudo mdadm --examine --scan -v

ARRAY /dev/md/0 level=raid10 metadata=1.2 num-devices=4 UUID=02f2f445:de9ba929:f6a3e9da:249f8e42 name=rockshock:0
   devices=/dev/sde,/dev/sdd,/dev/sdb

$ sudo mdadm --examine /dev/sd[abde]

 /dev/sdb:
         Magic : a92b4efc
       Version : 1.2
   Feature Map : 0x0
    Array UUID : 02f2f445:de9ba929:f6a3e9da:249f8e42
          Name : rockshock:0  (local to host rockshock)
 Creation Time : Wed Jan  1 16:40:06 2014
    Raid Level : raid10
  Raid Devices : 4

Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
    Array Size : 1953262592 (1862.78 GiB 2000.14 GB)
 Used Dev Size : 1953262592 (931.39 GiB 1000.07 GB)
   Data Offset : 262144 sectors
  Super Offset : 8 sectors
         State : clean
   Device UUID : 917412df:140cdbc6:68ad57e9:81cbcff9

   Update Time : Sat Jul 25 04:18:34 2015
      Checksum : 1f73fe53 - correct
        Events : 727

        Layout : near=2
    Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing)

 /dev/sdd:
         Magic : a92b4efc
       Version : 1.2
   Feature Map : 0x0
    Array UUID : 02f2f445:de9ba929:f6a3e9da:249f8e42
          Name : rockshock:0  (local to host rockshock)
 Creation Time : Wed Jan  1 16:40:06 2014
    Raid Level : raid10
  Raid Devices : 4

Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
    Array Size : 1953262592 (1862.78 GiB 2000.14 GB)
 Used Dev Size : 1953262592 (931.39 GiB 1000.07 GB)
   Data Offset : 262144 sectors
  Super Offset : 8 sectors
         State : clean
   Device UUID : 0904482c:1a9a1174:6936805c:8a466493

   Update Time : Sat Jul 25 04:25:38 2015
      Checksum : 269e217e - correct
        Events : 731

        Layout : near=2
    Chunk Size : 512K

   Device Role : Active device 0
   Array State : AA.A ('A' == active, '.' == missing)

 /dev/sde:
         Magic : a92b4efc
       Version : 1.2
   Feature Map : 0x0
    Array UUID : 02f2f445:de9ba929:f6a3e9da:249f8e42
          Name : rockshock:0  (local to host rockshock)
 Creation Time : Wed Jan  1 16:40:06 2014
    Raid Level : raid10
  Raid Devices : 4

Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
    Array Size : 1953262592 (1862.78 GiB 2000.14 GB)

    Raid Level : raid10
  Raid Devices : 4

Avail Dev Size : 1953263024 (931.39 GiB 1000.07 GB)
    Array Size : 1953262592 (1862.78 GiB 2000.14 GB)
 Used Dev Size : 1953262592 (931.39 GiB 1000.07 GB)
   Data Offset : 262144 sectors
  Super Offset : 8 sectors
         State : clean
   Device UUID : 9badcfe0:f4a4cd0c:baa83d4e:bc46c56e

   Update Time : Sat Jul 25 04:25:38 2015
      Checksum : 4100486e - correct
        Events : 731

        Layout : near=2
    Chunk Size : 512K

   Device Role : Active device 1
   Array State : AA.A ('A' == active, '.' == missing)

$ sudo fdisk -l

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00092197

   Device Boot      Start         End      Blocks   Id  System
Disk /dev/sdc: 60.0 GB, 60022480896 bytes
255 heads, 63 sectors/track, 7297 cylinders, total 117231408 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

  Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1   117231407    58615703+  ee  GPT

Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x0003765c

  Device Boot      Start         End      Blocks   Id  System

Disk /dev/sde: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x000ca97d

  Device Boot      Start         End      Blocks   Id  System

/dev/sdc1               1   117231407    58615703+  ee  GPT

Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x0003765c

  Device Boot      Start         End      Blocks   Id  System

Disk /dev/sde: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x000ca97d

  Device Boot      Start         End      Blocks   Id  System

So I'm asking expirienced with mdadm: turn me in right direction, what should I do in this situation.

1. Can I get data from the array before doing something? ()
2. For what reason "mdadm --detail /dev/md0" tell me md0 is not active? Is it normal behaviour with RAID10?
3. How to say to mdadm to forget about old disk in array not by logical name?
4. And how to correctly add new disk as a replacement of failed one in array?

Thank you.

Answer 1

/dev/sdb: Events : 727

/dev/sdd: Events : 731

/dev/sde: Events : 731

It looks like you had a two-drive failure: /dev/sdb suffered a transient fault around 04:18:34 and was removed from the array, then /dev/sdc failed around 04:25:38 in a way that caused the entire array to freeze up. Since the event count for /dev/sdb is lower than that of /dev/sdd and /dev/sde, mdadm will not include it when starting the array, and the two remaining disks are not enough to form a complete RAID 10 array (presumably they're mirrors of each other).

The Linux RAID wiki procedure for recovering a failed array should work for RAID 10 just as it works for RAID 5/6. The key step is telling mdadm to ignore the event count mismatch using the --force flag; the purpose of the rest of the procedure is to find out in advance whether doing this will get your data back, or destroy it.