A friend of mine has a mdadm-raid5 with 9 disks which does not reassemble anymore.

After having a look at the syslog I found that the disk sdi was kicked from the array:

Jul  6 08:43:25 nasty kernel: [   12.952194] md: bind<sdc>
Jul  6 08:43:25 nasty kernel: [   12.952577] md: bind<sdd>
Jul  6 08:43:25 nasty kernel: [   12.952683] md: bind<sde>
Jul  6 08:43:25 nasty kernel: [   12.952784] md: bind<sdf>
Jul  6 08:43:25 nasty kernel: [   12.952885] md: bind<sdg>
Jul  6 08:43:25 nasty kernel: [   12.952981] md: bind<sdh>
Jul  6 08:43:25 nasty kernel: [   12.953078] md: bind<sdi>
Jul  6 08:43:25 nasty kernel: [   12.953169] md: bind<sdj>
Jul  6 08:43:25 nasty kernel: [   12.953288] md: bind<sda>
Jul  6 08:43:25 nasty kernel: [   12.953308] md: kicking non-fresh sdi from array!
Jul  6 08:43:25 nasty kernel: [   12.953314] md: unbind<sdi>
Jul  6 08:43:25 nasty kernel: [   12.960603] md: export_rdev(sdi)
Jul  6 08:43:25 nasty kernel: [   12.969675] raid5: device sda operational as raid disk 0
Jul  6 08:43:25 nasty kernel: [   12.969679] raid5: device sdj operational as raid disk 8
Jul  6 08:43:25 nasty kernel: [   12.969682] raid5: device sdh operational as raid disk 6
Jul  6 08:43:25 nasty kernel: [   12.969684] raid5: device sdg operational as raid disk 5
Jul  6 08:43:25 nasty kernel: [   12.969687] raid5: device sdf operational as raid disk 4
Jul  6 08:43:25 nasty kernel: [   12.969689] raid5: device sde operational as raid disk 3
Jul  6 08:43:25 nasty kernel: [   12.969692] raid5: device sdd operational as raid disk 2
Jul  6 08:43:25 nasty kernel: [   12.969694] raid5: device sdc operational as raid disk 1
Jul  6 08:43:25 nasty kernel: [   12.970536] raid5: allocated 9542kB for md127
Jul  6 08:43:25 nasty kernel: [   12.973975] 0: w=1 pa=0 pr=9 m=1 a=2 r=9 op1=0 op2=0
Jul  6 08:43:25 nasty kernel: [   12.973980] 8: w=2 pa=0 pr=9 m=1 a=2 r=9 op1=0 op2=0
Jul  6 08:43:25 nasty kernel: [   12.973983] 6: w=3 pa=0 pr=9 m=1 a=2 r=9 op1=0 op2=0
Jul  6 08:43:25 nasty kernel: [   12.973986] 5: w=4 pa=0 pr=9 m=1 a=2 r=9 op1=0 op2=0
Jul  6 08:43:25 nasty kernel: [   12.973989] 4: w=5 pa=0 pr=9 m=1 a=2 r=9 op1=0 op2=0
Jul  6 08:43:25 nasty kernel: [   12.973992] 3: w=6 pa=0 pr=9 m=1 a=2 r=9 op1=0 op2=0
Jul  6 08:43:25 nasty kernel: [   12.973996] 2: w=7 pa=0 pr=9 m=1 a=2 r=9 op1=0 op2=0
Jul  6 08:43:25 nasty kernel: [   12.973999] 1: w=8 pa=0 pr=9 m=1 a=2 r=9 op1=0 op2=0
Jul  6 08:43:25 nasty kernel: [   12.974002] raid5: raid level 5 set md127 active with 8 out of 9 devices, algorithm 2

Unfortunately this wasn't recognized and now another drive was kicked (sde):

Jul 14 08:02:45 nasty kernel: [   12.918556] md: bind<sdc>
Jul 14 08:02:45 nasty kernel: [   12.919043] md: bind<sdd>
Jul 14 08:02:45 nasty kernel: [   12.919158] md: bind<sde>
Jul 14 08:02:45 nasty kernel: [   12.919260] md: bind<sdf>
Jul 14 08:02:45 nasty kernel: [   12.919361] md: bind<sdg>
Jul 14 08:02:45 nasty kernel: [   12.919461] md: bind<sdh>
Jul 14 08:02:45 nasty kernel: [   12.919556] md: bind<sdi>
Jul 14 08:02:45 nasty kernel: [   12.919641] md: bind<sdj>
Jul 14 08:02:45 nasty kernel: [   12.919756] md: bind<sda>
Jul 14 08:02:45 nasty kernel: [   12.919775] md: kicking non-fresh sdi from array!
Jul 14 08:02:45 nasty kernel: [   12.919781] md: unbind<sdi>
Jul 14 08:02:45 nasty kernel: [   12.928177] md: export_rdev(sdi)
Jul 14 08:02:45 nasty kernel: [   12.928187] md: kicking non-fresh sde from array!
Jul 14 08:02:45 nasty kernel: [   12.928198] md: unbind<sde>
Jul 14 08:02:45 nasty kernel: [   12.936064] md: export_rdev(sde)
Jul 14 08:02:45 nasty kernel: [   12.943900] raid5: device sda operational as raid disk 0
Jul 14 08:02:45 nasty kernel: [   12.943904] raid5: device sdj operational as raid disk 8
Jul 14 08:02:45 nasty kernel: [   12.943907] raid5: device sdh operational as raid disk 6
Jul 14 08:02:45 nasty kernel: [   12.943909] raid5: device sdg operational as raid disk 5
Jul 14 08:02:45 nasty kernel: [   12.943911] raid5: device sdf operational as raid disk 4
Jul 14 08:02:45 nasty kernel: [   12.943914] raid5: device sdd operational as raid disk 2
Jul 14 08:02:45 nasty kernel: [   12.943916] raid5: device sdc operational as raid disk 1
Jul 14 08:02:45 nasty kernel: [   12.944776] raid5: allocated 9542kB for md127
Jul 14 08:02:45 nasty kernel: [   12.944861] 0: w=1 pa=0 pr=9 m=1 a=2 r=9 op1=0 op2=0
Jul 14 08:02:45 nasty kernel: [   12.944864] 8: w=2 pa=0 pr=9 m=1 a=2 r=9 op1=0 op2=0
Jul 14 08:02:45 nasty kernel: [   12.944867] 6: w=3 pa=0 pr=9 m=1 a=2 r=9 op1=0 op2=0
Jul 14 08:02:45 nasty kernel: [   12.944871] 5: w=4 pa=0 pr=9 m=1 a=2 r=9 op1=0 op2=0
Jul 14 08:02:45 nasty kernel: [   12.944874] 4: w=5 pa=0 pr=9 m=1 a=2 r=9 op1=0 op2=0
Jul 14 08:02:45 nasty kernel: [   12.944877] 2: w=6 pa=0 pr=9 m=1 a=2 r=9 op1=0 op2=0
Jul 14 08:02:45 nasty kernel: [   12.944879] 1: w=7 pa=0 pr=9 m=1 a=2 r=9 op1=0 op2=0
Jul 14 08:02:45 nasty kernel: [   12.944882] raid5: not enough operational devices for md127 (2/9 failed)

And now the array does not start anymore. However it seems that every disk contains the raid metadata:

/dev/sda:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : b8a04dbb:0b5dffda:601eb40d:d2dc37c9
           Name : nasty:stuff  (local to host nasty)
  Creation Time : Sun Mar 16 02:37:47 2014
     Raid Level : raid5
   Raid Devices : 9

 Avail Dev Size : 7814035120 (3726.02 GiB 4000.79 GB)
     Array Size : 62512275456 (29808.18 GiB 32006.29 GB)
  Used Dev Size : 7814034432 (3726.02 GiB 4000.79 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 8600bda9:18845be8:02187ecc:1bfad83a

    Update Time : Mon Jul 14 00:45:35 2014
       Checksum : e38d46e8 - correct
         Events : 123132

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAA.AAA.A ('A' == active, '.' == missing)


/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : b8a04dbb:0b5dffda:601eb40d:d2dc37c9
           Name : nasty:stuff  (local to host nasty)
  Creation Time : Sun Mar 16 02:37:47 2014
     Raid Level : raid5
   Raid Devices : 9

 Avail Dev Size : 7814035120 (3726.02 GiB 4000.79 GB)
     Array Size : 62512275456 (29808.18 GiB 32006.29 GB)
  Used Dev Size : 7814034432 (3726.02 GiB 4000.79 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : fe612c05:f7a45b0a:e28feafe:891b2bda

    Update Time : Mon Jul 14 00:45:35 2014
       Checksum : 32bb628e - correct
         Events : 123132

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAA.AAA.A ('A' == active, '.' == missing)


/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : b8a04dbb:0b5dffda:601eb40d:d2dc37c9
           Name : nasty:stuff  (local to host nasty)
  Creation Time : Sun Mar 16 02:37:47 2014
     Raid Level : raid5
   Raid Devices : 9

 Avail Dev Size : 7814035120 (3726.02 GiB 4000.79 GB)
     Array Size : 62512275456 (29808.18 GiB 32006.29 GB)
  Used Dev Size : 7814034432 (3726.02 GiB 4000.79 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 1d14616c:d30cadc7:6d042bb3:0d7f6631

    Update Time : Mon Jul 14 00:45:35 2014
       Checksum : 62bd5499 - correct
         Events : 123132

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAA.AAA.A ('A' == active, '.' == missing)


/dev/sde:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : b8a04dbb:0b5dffda:601eb40d:d2dc37c9
           Name : nasty:stuff  (local to host nasty)
  Creation Time : Sun Mar 16 02:37:47 2014
     Raid Level : raid5
   Raid Devices : 9

 Avail Dev Size : 7814035120 (3726.02 GiB 4000.79 GB)
     Array Size : 62512275456 (29808.18 GiB 32006.29 GB)
  Used Dev Size : 7814034432 (3726.02 GiB 4000.79 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : a2babca3:1283654a:ef8075b5:aaf5d209

    Update Time : Mon Jul 14 00:45:07 2014
       Checksum : f78d6456 - correct
         Events : 123123

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAAAAA.A ('A' == active, '.' == missing)


/dev/sdf:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : b8a04dbb:0b5dffda:601eb40d:d2dc37c9
           Name : nasty:stuff  (local to host nasty)
  Creation Time : Sun Mar 16 02:37:47 2014
     Raid Level : raid5
   Raid Devices : 9

 Avail Dev Size : 7814035120 (3726.02 GiB 4000.79 GB)
     Array Size : 62512275456 (29808.18 GiB 32006.29 GB)
  Used Dev Size : 7814034432 (3726.02 GiB 4000.79 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : e67d566d:92aaafb4:24f5f16e:5ceb0db7

    Update Time : Mon Jul 14 00:45:35 2014
       Checksum : 9223b929 - correct
         Events : 123132

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 4
   Array State : AAA.AAA.A ('A' == active, '.' == missing)


/dev/sdg:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : b8a04dbb:0b5dffda:601eb40d:d2dc37c9
           Name : nasty:stuff  (local to host nasty)
  Creation Time : Sun Mar 16 02:37:47 2014
     Raid Level : raid5
   Raid Devices : 9

 Avail Dev Size : 7814035120 (3726.02 GiB 4000.79 GB)
     Array Size : 62512275456 (29808.18 GiB 32006.29 GB)
  Used Dev Size : 7814034432 (3726.02 GiB 4000.79 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 2cee1d71:16c27acc:43e80d02:1da74eeb

    Update Time : Mon Jul 14 00:45:35 2014
       Checksum : 7512efd4 - correct
         Events : 123132

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 5
   Array State : AAA.AAA.A ('A' == active, '.' == missing)


/dev/sdh:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : b8a04dbb:0b5dffda:601eb40d:d2dc37c9
           Name : nasty:stuff  (local to host nasty)
  Creation Time : Sun Mar 16 02:37:47 2014
     Raid Level : raid5
   Raid Devices : 9

 Avail Dev Size : 7814035120 (3726.02 GiB 4000.79 GB)
     Array Size : 62512275456 (29808.18 GiB 32006.29 GB)
  Used Dev Size : 7814034432 (3726.02 GiB 4000.79 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : c239f0ad:336cdb88:62c5ff46:c36ea5f8

    Update Time : Mon Jul 14 00:45:35 2014
       Checksum : c08e8a4d - correct
         Events : 123132

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 6
   Array State : AAA.AAA.A ('A' == active, '.' == missing)


/dev/sdi:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : b8a04dbb:0b5dffda:601eb40d:d2dc37c9
           Name : nasty:stuff  (local to host nasty)
  Creation Time : Sun Mar 16 02:37:47 2014
     Raid Level : raid5
   Raid Devices : 9

 Avail Dev Size : 7814035120 (3726.02 GiB 4000.79 GB)
     Array Size : 62512275456 (29808.18 GiB 32006.29 GB)
  Used Dev Size : 7814034432 (3726.02 GiB 4000.79 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : d06c58f8:370a0535:b7e51073:f121f58c

    Update Time : Mon Jul 14 00:45:07 2014
       Checksum : 77844dcc - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : spare
   Array State : AAAAAAA.A ('A' == active, '.' == missing)


/dev/sdj:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : b8a04dbb:0b5dffda:601eb40d:d2dc37c9
           Name : nasty:stuff  (local to host nasty)
  Creation Time : Sun Mar 16 02:37:47 2014
     Raid Level : raid5
   Raid Devices : 9

 Avail Dev Size : 7814035120 (3726.02 GiB 4000.79 GB)
     Array Size : 62512275456 (29808.18 GiB 32006.29 GB)
  Used Dev Size : 7814034432 (3726.02 GiB 4000.79 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : f2de262f:49d17fea:b9a475c1:b0cad0b7

    Update Time : Mon Jul 14 00:45:35 2014
       Checksum : dd0acfd9 - correct
         Events : 123132

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 8
   Array State : AAA.AAA.A ('A' == active, '.' == missing)

But as you can see the two drives (sde, sdi) are in active state (but raid is stopped) and sdi is a spare. While sde has a slightly lower Events-count than most of the other drives (123123 instead of 123132) sdi has an Events-count of 0. So I think sde is almost up-to-date. But sdi not ...

Now we read online that a hard power-off could cause these "kicking non-fresh"-messages. And indeed my friend caused a hard power-off one or two times. So we followed the instructions we found online and tried to re-add sde to the array:

$ mdadm /dev/md127 --add /dev/sde
mdadm: add new device failed for /dev/sde as 9: Invalid argument

But that failed and now mdadm --examine /dev/sde shows an Events-count of 0 for sde too (+ it's a spare now like sdi):

/dev/sde:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : b8a04dbb:0b5dffda:601eb40d:d2dc37c9
           Name : nasty:stuff  (local to host nasty)
  Creation Time : Sun Mar 16 02:37:47 2014
     Raid Level : raid5
   Raid Devices : 9

 Avail Dev Size : 7814035120 (3726.02 GiB 4000.79 GB)
     Array Size : 62512275456 (29808.18 GiB 32006.29 GB)
  Used Dev Size : 7814034432 (3726.02 GiB 4000.79 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 689e0030:142122ae:7ab37935:c80ab400

    Update Time : Mon Jul 14 00:45:35 2014
       Checksum : 5e6c4cf7 - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : spare
   Array State : AAA.AAA.A ('A' == active, '.' == missing)

We know that 2 failed drives usually means the death for a raid5. However is there a way to add at least sde to the raid so that data can be saved?

up vote 6 down vote accepted

OK, it looks like we have now access to the raid. At least the first checked files looked good. So here is what we have done:


The raid recovery article on the kernel.org wiki suggests two possible solutions for our problem:

  1. using --assemble --force (also mentioned by derobert)
    The article says:

    [...] If the event count differs by less than 50, then the information on the drive is probably still ok. [...] If the event count closely matches but not exactly, use "mdadm --assemble --force /dev/mdX " to force mdadm to assemble the array [...]. If the event count of a drive is way off [...] that drive [...] shouldn't be included in the assembly.

    In our case the drive sde had an event difference of 9. So there was a good chance that --force would work. However after we executed the --add command the event count dropped to 0 and the drive was marked as spare.

    So we better desisted from using --force.

  2. recreate the array
    This solution is explicitly marked as dangerous because you can loose data if you do something wrong. However this seemed to be the only option we had.

    The idea is to create a new raid on the existing raid-devices (that is overwriting the device's superblocks) with the same configuration of the old raid and explicitly tell mdadm that the raid has already existed and should be assumed as clean.

    Since the event count difference was just 9 and the only problem was that we lost the superblock of sde there were good chances that writing new superblocks will get us access to our data... and it worked :-)


Our solution

Note: This solution was specially geared to our problem and may not work on your setup. You should take these notes to get an idea on how things can be done. But you need to research what's best in your case.

Backup
We already lost a superblock. So this time we saved the first and last gigabyte of each raid device (sd[acdefghij]) using dd before working on the raid. We did this for each raid device:

# save the first gigabyte of sda
dd if=/dev/sda of=bak_sda_start bs=4096 count=262144

# determine the size of the device
fdisk -l /dev/sda
# In this case the size was 4000787030016 byte.

# To get the last gigabyte we need to skip everything except the last gigabyte.
# So we need to skip: 4000787030016 byte - 1073741824 byte = 3999713288000 byte
# Since we read blocks auf 4096 byte we need to skip 3999713288000/4096=976492502 blocks.
dd if=/dev/sda of=bak_sda_end bs=4096 skip=976492502

Gather information
When recreating the raid it is important to use the same configration as the old raid. This is especially important if you want to recreate the array on another machine using a different mdadm version. In this case mdadm's default values may be different and could create superblocks that do not fit to the existing raid (see the wiki article).

In our case we use the same machine (and thus the same mdadm-version) to recreate the array. However the array was created by a 3rd party tool in the first place. So we didn't want to rely on default values here and had to gather some information about the existing raid.

From the output of mdadm --examine /dev/sd[acdefghij] we get the following information about the raid (Note: sdb was the ssd containing the OS and was not part of the raid):

     Raid Level : raid5
   Raid Devices : 9
  Used Dev Size : 7814034432 (3726.02 GiB 4000.79 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
         Layout : left-symmetric
     Chunk Size : 512K
   Device Role : Active device 0

The Used Dev Size is denominated in blocks of 512 byte. You can check this:
7814034432*512/1000000000 ~= 4000.79
But mdadm requires the size in Kibibytes: 7814034432*512/1024 = 3907017216

Important is the Device Role. In the new raid each device must have the same role as before. In our case:

device  role
------  ----
sda     0
sdc     1
sdd     2
sde     3
sdf     4
sdg     5
sdh     6
sdi     spare
sdj     8

Note: Drive letters (and thus the order) can change after reboot!

We also need the layout and the chunk size in the next step.

Recreate raid
We now can use the information of the last step to recreate the array:

mdadm --create --assume-clean --level=5 --raid-devices=9 --size=3907017216 \
    --chunk=512 --layout=left-symmetric /dev/md127 /dev/sda /dev/sdc /dev/sdd \
    /dev/sde /dev/sdf /dev/sdg /dev/sdh missing /dev/sdj

It is important to pass the devices in the correct order!
Moreover we did not add sdi as it's event count was too low. So we set the 7th raid slot to missing. Thus the raid5 contains 8 of 9 devices and will be assembled in degraded mode. And because it lacks a spare device no rebuild will automatically start.

Then we used --examine to check if the new superblocks fit to our old superblocks. And it did :-) We were able to mount the filesystem and read the data. The next step is to backup the data and then add back sdi and start the rebuild.

  • The used dev size is not important, unless the missing drive happens to be smaller than all the others or you were for some reason deliberately not using the full device size. It's odd that you specify this and not any of the other more important variables (metadata version, offset, ...). Glad it worked either way. – frostschutz Jul 28 '14 at 9:46
  • Hi frostschutz. That's strange. I was going to accept your answer, but it was gone for two days o0 But now it's back on this page again. I didn't remeber that you suggested also to set the metadata version and offset. But if I had seen that the data of the new superblock do not match the data of the old superblock I would have used the backups and used those options ;) Big THX to you for your help! – Biggie Jul 28 '14 at 12:02

mdadm --force should fix this. Note that you may suffer minor data corruption, as its going to pretend its in sync. Use it like this:

  1. mdadm --stop /dev/md127 (you need to stop what's currently running first)
  2. mdadm -v --assemble --run --force /dev/md127 /dev/sd[a-hl-z]. The key thing is leaving out /dev/sdi here, as we know that disk is the least recent. NOTE: If not all your disks are part of the array, it'd be safer to actually list out the disks here instead of using the shell wildcard.
  3. There should be a bunch of messages, including one telling its including the out-of-date disk /dev/sde. Check /proc/mdstat to make sure you have a running, but degraded, RAID5.
  4. Figure out why the disk was dropped. Depending on why, you may now need to immediately copy your data off (e.g., disk is on death's door) or alternatively can proceed to re-add /dev/sdi and let it rebuild.

mdadm can be configured to monitor the array and send alerts when bad things (like losing a disk) happen. You need to properly configure this.

  • The idea sounds good. But when we tried to add sde back to the array it's superblock seems to be overridden: the uuid and checksum changed, the Events-count dropped to 0 and it's role switched from "Active device 3" to "spare" (see logs above). But I think we need to restore the old superblock before we should try the --force option. Is that possible? – Biggie Jul 25 '14 at 21:13
  • @Biggie You could certainly set the device role back to what it should be (active device 3)... but I don't think mdadm has an easy way to do that. Could be done with a hex editor. You're right that will probably prevent assemble force from working. – derobert Jul 25 '14 at 21:21
  • @Biggie raid.wiki.kernel.org/index.php/RAID_superblock_formats has the superblock format, and where to find it on the disk (4k from the start for 1.2). – derobert Jul 25 '14 at 21:23
  • @Biggie Also, it's possible its array slot is actually still fine, just its role is wrong, and --force may well fix that for you. Just do the force with --read-only, and it should be pretty safe (since you already have superblock details backed up) – derobert Jul 25 '14 at 21:25

Your only option here, since you turned your drives spare and lost their metadata, is to re-create the RAID. This is very dangerous, a mistake will wipe your data.

When re-creating a RAID there are several things to consider. You must use --assume-clean so it won't sync. You should leave one drive out (preferably the one that is in the worst state) by specifying it as missing. Last but not least, you must get all the variables right: drive order, metadata version, raid level, chunksize, layout, data offset, etc. ...

You can not rely on defaults here since defaults tend to change a lot with mdadm. If your rescue environment is not using the same mdadm version as the environment that created your RAID in the first place, and you're not using the exact same parameters, etc. you are in trouble if you rely on defaults.

You should make at the very least a backup of each drive's first and last gigabyte or so, so you can at least undo mdadm experiments. Ideally you would do all of this on a full copy, or use dm-snapshots or nbd-server in copy-on-write mode to have a read-only view of things. See this Overlay how-to.

If I interpret the --examine output you posted correctly, provided your drive letters didn't change, this may be the correct command to recreate, but I can't guarantee anything:

mdadm --create /dev/md42 --assume-clean --metadata=1.2 --data-offset=1M \
      --level=5 --chunk=512 --layout=ls --raid-devices=9 \
      /dev/sda /dev/sdc /dev/sdd /dev/sde /dev/sdf \
      /dev/sdg /dev/sdh missing /dev/sdj

See also https://raid.wiki.kernel.org/index.php/RAID_Recovery however take the advice there with care. Recreation is a horrible business, very easy to make mistakes.

Once you created the RAID you should check it out in read-only mode:

mdadm --readonly /dev/md42
file -s /dev/md42
fsck -n /dev/md42
mount -o ro /dev/md42 /mnt/md42

If that works so far, you should find a large file (chunksize*drives) and see if it's all OK. It is entirely possible that you can mount the filesystem fine but have corrupt files if the wrong two drives have switched places.

You will also have to update your mdadm.conf and such since it will be a new raid with new uuid etc.

  • THX. A few questions: 1. How did you determine the data-offset? data-offset (sectors) multiplied with the chunk size? 2. Am I right that this will ignore the already existing superblocks (like the one of sde with the zero Events-count) and create new ones? 3. Is it safe to first test the --assemble --force option mentioned by derobert without risking any data loss before trying to re-create the array? – Biggie Jul 25 '14 at 21:52
  • 1) 2048 sectors = 1MiB. 2) It creates a new raid with new metadata, regardless of what was there before. 3) --force won't work. It only ignores the event count, it does not accept hard failed drives or drives that turned spare. Unfortunately mdadm does not have any smart recovery options. – frostschutz Jul 25 '14 at 22:03
  • to 1) So you assume a sector has 512 bytes? But most of the new drives use 4096 byte block size. Like the used 4TB drives. So I need to set it to 8MB? Or should I let it unchanged since I reassemble it on the same machine using same OS and mdadm as before (thus using the default params). to 3) But can a --force with --read-only cause any corruption? Or is it safe to test it before re-creating the array? I just don't want to keep possible solutions untested :) – Biggie Jul 25 '14 at 22:47
  • Well, good luck. – frostschutz Jul 25 '14 at 23:11
  • So is it safe or not? :) – Biggie Jul 26 '14 at 6:25

I tried this on a RAID5 setup (5 drives, 1TB each), where 2 drives got kicked out from the array. I took the risk and I added back both kicked off drives using: mdadm --create --assume-clean --level=5 --raid-devices=5 --chunk=512 --layout=left-symmetric /dev/md127 /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdb where sdd and sdf was the kicked off drive (also thanks to some meddling set to spare).

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.