2

I have a personal raid 5 array that is in rough shape. The failed array has eight 2 TB drives that were built up over time in my n-2 desktop machine.

The array is from an in-house computer that was my n-2 desktop machine. I put eight 2TB drives into it as my storage needs grew. It had an undersized processor and a meager amount of memory. After finding the raid in a degraded state, I tried a number of times to bring it back. This was late at night during a period when I was short on time and cash.

I have the 8 drives in a different machine now. However, the array will not activate. I have tried assembling it.

Is there anything I should try before attempting: --assemble --force or --create --assume-clean?

Here are details about the array:

$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md6 : inactive sde[0](S)
      1953514496 blocks

md1 : inactive sdf[0](S) sdl[5](S) sdg[1](S) sdh[2](S) sdi[4](S)
      9766917560 blocks super 1.2

unused devices: <none>

$ sudo mdadm --examine /dev/sd[e-l] |  egrep 'Event|/dev/sd'
/dev/sde:
         Events : 2266749
/dev/sdf:
         Events : 171
/dev/sdg:
         Events : 171
/dev/sdh:
         Events : 171
/dev/sdi:
         Events : 27
/dev/sdj:
         Events : 27
/dev/sdk:
         Events : 31
/dev/sdl:
         Events : 31

$ sudo mdadm --examine /dev/sd[e-l] |  egrep 'Array'
     Array Size : 13674601472 (13041.12 GiB 14002.79 GB)
     Array UUID : 5e410da5:ef0f83c4:aa4b35d6:28916966
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
   Array State : AAA.... ('A' == active, '.' == missing)
     Array UUID : 5e410da5:ef0f83c4:aa4b35d6:28916966
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
   Array State : AAA.... ('A' == active, '.' == missing)
     Array UUID : 5e410da5:ef0f83c4:aa4b35d6:28916966
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
   Array State : AAA.... ('A' == active, '.' == missing)
     Array UUID : 5e410da5:ef0f83c4:aa4b35d6:28916966
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
   Array State : AAAAAAA ('A' == active, '.' == missing)
     Array UUID : 5e410da5:ef0f83c4:aa4b35d6:28916966
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
   Array State : AAAAAAA ('A' == active, '.' == missing)
     Array UUID : 5e410da5:ef0f83c4:aa4b35d6:28916966
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
   Array State : AAAA.A. ('A' == active, '.' == missing)
     Array UUID : 5e410da5:ef0f83c4:aa4b35d6:28916966
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
   Array State : AAAA.A. ('A' == active, '.' == missing)

$ dmesg
...
[   33.532447] scsi7 : rr62x
[   33.532641] scsi 7:0:0:0: Direct-Access     ATA      WDC WD20EARX-00P 51.0 PQ: 0 ANSI: 5
[   33.532699] scsi 7:0:1:0: Direct-Access     ATA      WDC WD20EARX-00P 51.0 PQ: 0 ANSI: 5
[   33.532751] scsi 7:0:2:0: Direct-Access     ATA      WDC WD20EARX-00P 51.0 PQ: 0 ANSI: 5
[   33.532823] scsi 7:0:3:0: Direct-Access     ATA      WDC WD20EARX-00P 51.0 PQ: 0 ANSI: 5
[   33.532871] scsi 7:0:4:0: Direct-Access     ATA      WDC WD20EARX-00P 51.0 PQ: 0 ANSI: 5
[   33.532919] scsi 7:0:5:0: Direct-Access     ATA      WDC WD20EARX-00P 51.0 PQ: 0 ANSI: 5
[   33.532978] scsi 7:0:6:0: Direct-Access     ATA      WDC WD20EARX-00P 51.0 PQ: 0 ANSI: 5
[   33.533051] scsi 7:0:7:0: Direct-Access     ATA      WDC WD20EARX-00P 51.0 PQ: 0 ANSI: 5
...
[   33.535178]  sdg: unknown partition table
...
[   33.535928]  sdh: unknown partition table
[   33.536042]  sdf: unknown partition table
...
[   33.536481]  sdj: unknown partition table
[   33.536568]  sdk: unknown partition table
[   33.536624] sd 7:0:5:0: [sdj] Attached SCSI disk
[   33.536663]  sdl: unknown partition table
[   33.536775] sd 7:0:7:0: [sdl] Attached SCSI disk
[   33.536834] sd 7:0:6:0: [sdk] Attached SCSI disk
[   33.537042]  sdi: unknown partition table
[   33.537190] sd 7:0:4:0: [sdi] Attached SCSI disk
[   33.573418]  sde: sde1 sde2 < sde5 >
[   33.573753] sd 7:0:0:0: [sde] Attached SCSI disk
[   33.609751] md: bind<sdi>
[   33.612501] md: bind<sdh>
[   33.615419] md: bind<sdg>
...
[   34.032720] md: bind<sde>
...
[   39.140891] md: bind<sdl>
...
[   64.651514] systemd-udevd[484]: '/sbin/mdadm --incremental /dev/sdk' [2730] terminated by signal 9 (Killed)
[   64.651589] systemd-udevd[487]: '/sbin/mdadm --incremental /dev/sdj' [2466] terminated by signal 9 (Killed)
...
[   65.185503] systemd-udevd[488]: '/sbin/mdadm --incremental /dev/sdf' [2470] terminated by signal 9 (Killed)
...

$ sudo mdadm --examine /dev/sd[e-l]
/dev/sde:
        Version : 0.90.00
           UUID : d7b20e6c:1b319034:e5fb2d97:a1144fba
  Creation Time : Sun Apr 29 22:14:28 2012
     Raid Level : raid5
  Used Dev Size : 1953514496 (1863.02 GiB 2000.40 GB)
     Array Size : 13674601472 (13041.12 GiB 14002.79 GB)
   Raid Devices : 8
  Total Devices : 8
Preferred Minor : 1

    Update Time : Fri Dec 27 15:32:54 2013
          State : clean
 Active Devices : 8
Working Devices : 8
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 3a37d47e - correct
         Events : 2266749

         Layout : left-symmetric
     Chunk Size : 512K

      Number   Major   Minor   RaidDevice State
this     0       8        0        0      active sync   /dev/sda

   0     0       8        0        0      active sync   /dev/sda
   1     1       8       16        1      active sync   /dev/sdb
   2     2       8       32        2      active sync   /dev/sdc
   3     3       8       48        3      active sync   /dev/sdd
   4     4       8       64        4      active sync   /dev/sde
   5     5       8       80        5      active sync   /dev/sdf
   6     6       8       96        6      active sync   /dev/sdg
   7     7       8      112        7      active sync   /dev/sdh
/dev/sdf:
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 5e410da5:ef0f83c4:aa4b35d6:28916966
           Name : mjf-sysa:1
  Creation Time : Fri Dec 27 15:53:16 2013
     Raid Level : raid5
   Raid Devices : 7

 Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
  Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 9d365d82:23f568c5:46534949:3b2dff9f

    Update Time : Sat Mar 29 08:00:25 2014
       Checksum : 86b60aea - correct
         Events : 171

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAA.... ('A' == active, '.' == missing)
/dev/sdg:
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 5e410da5:ef0f83c4:aa4b35d6:28916966
           Name : mjf-sysa:1
  Creation Time : Fri Dec 27 15:53:16 2013
     Raid Level : raid5
   Raid Devices : 7

 Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
  Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : a21fde1e:07530215:0f854883:bc6484d6

    Update Time : Sat Mar 29 08:00:25 2014
       Checksum : e354bb1d - correct
         Events : 171

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAA.... ('A' == active, '.' == missing)
/dev/sdh:
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 5e410da5:ef0f83c4:aa4b35d6:28916966
           Name : mjf-sysa:1
  Creation Time : Fri Dec 27 15:53:16 2013
     Raid Level : raid5
   Raid Devices : 7

 Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
  Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : f0446a88:6a5d5026:37375e86:1bb5c6cf

    Update Time : Sat Mar 29 08:00:25 2014
       Checksum : 5a86ed57 - correct
         Events : 171

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAA.... ('A' == active, '.' == missing)
/dev/sdi:
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 5e410da5:ef0f83c4:aa4b35d6:28916966
           Name : mjf-sysa:1
  Creation Time : Fri Dec 27 15:53:16 2013
     Raid Level : raid5
   Raid Devices : 7

 Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
  Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : ec21c8aa:75a114e6:5cdb77ea:543c438b

    Update Time : Sun Jan 19 21:53:52 2014
       Checksum : 5bef1322 - correct
         Events : 27

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 4
   Array State : AAAAAAA ('A' == active, '.' == missing)
/dev/sdj:
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 5e410da5:ef0f83c4:aa4b35d6:28916966
           Name : mjf-sysa:1
  Creation Time : Fri Dec 27 15:53:16 2013
     Raid Level : raid5
   Raid Devices : 7

 Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
  Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : db1f64fd:5c0b94e8:6f717f78:c2348f0b

    Update Time : Sun Jan 19 21:53:52 2014
       Checksum : bf5e097a - correct
         Events : 27

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 6
   Array State : AAAAAAA ('A' == active, '.' == missing)
/dev/sdk:
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 5e410da5:ef0f83c4:aa4b35d6:28916966
           Name : mjf-sysa:1
  Creation Time : Fri Dec 27 15:53:16 2013
     Raid Level : raid5
   Raid Devices : 7

 Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
  Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 5bf385b8:5747c231:196e1bd5:286bb1ef

    Update Time : Sun Jan 19 21:58:51 2014
       Checksum : 46e4d24 - correct
         Events : 31

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAA.A. ('A' == active, '.' == missing)
/dev/sdl:
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 5e410da5:ef0f83c4:aa4b35d6:28916966
           Name : mjf-sysa:1
  Creation Time : Fri Dec 27 15:53:16 2013
     Raid Level : raid5
   Raid Devices : 7

 Avail Dev Size : 3906767024 (1862.89 GiB 2000.26 GB)
     Array Size : 11720297472 (11177.35 GiB 12001.58 GB)
  Used Dev Size : 3906765824 (1862.89 GiB 2000.26 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : a6a63833:7302c477:58127a50:b4204201

    Update Time : Sun Jan 19 21:58:51 2014
       Checksum : 52121556 - correct
         Events : 31

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 5
   Array State : AAAA.A. ('A' == active, '.' == missing)
  • 1
    You should make a backup of the drives if the data on it is worth it. – Anthon Feb 15 '15 at 3:37
2

In RAID recovery, you should never, never, ever --create unless you really really REALLY know EXACTLY what you're doing. However your output looks like you've already done the --create.

The RAID on your /dev/sde is a RAID5, created Sun Apr 29 22:14:28 2012, consisting of 8 disks as you stated in your question. This RAID was last updated on Fri Dec 27 15:32:54 2013.

However, your other disks claim to be RAID5, created Fri Dec 27 15:53:16 2013, consisting of 7 disks. So you already created a new RAID on that Friday; if you did it in the hope of recovery, you probably did it wrong, as it's not using the same settings shown by the /dev/sde 8 disk RAID metadata (wrong metadata version, offsets, layout etc. etc.).

So this sounds to me as though you already did a --create back in 2013. Starting from that point you were using it as a 7 disk RAID5.

The question now is, where is your data? Was it actually on the 7 disk RAID and lost sometime later? Or was it on the 8 disk RAID you created the other RAID over? Judging from the event counts it's more likely it was on 8 disks.

If the data was on 8 disks (i.e. it was lost in Dec 2013) recovery might be impossible. Recovery is hard enough after a botched --create, especially if the new/wrong RAID was allowed to sync. But with all sorts of things happening in the meantime as well... and that's not even considering that some of your disks might probably be faulty.

Your situation seems chaotic so it's impossible to advise what steps to take...

No matter what you do, DO NOT WRITE TO THESE DISKS ANYMORE. That means, don't assemble, don't sync, don't fsck, don't mount.

If you must, use this method: https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file

But first I'd buy another set of 8 disks (or half the number of 4TB disks) and ddrescue all of it, as the above method assumes none of your disks are faulty.

  • Thanks for the detailed analysis of this problem and the URI reference. I will read it over. – Michael Feb 15 '15 at 19:44
0

you say "After finding the raid in a degraded state, I tried a number of times to bring it back."

what did you do exactly? if you find a degraded array you need to replace the failed disk. if you did not do that and you got another drive failure, your RAID is gone! RAID5 tolerates only one failed drive.

you say you have eight 2TB drives in your RAID5. your /proc/mdstat contains only six. where are the two remaining disks? you need minimum of 7 to start your RAID5

additionally you have two RAID devices md1 and md6. should't sde also belong to md1? however even with sde in md1 you are missing one drive to start RAID5 with 8 drives...

for next time remember: "RAID substitutes no BACKUP"

  • I agree that raid is not in and of itself a backup. – Michael Feb 15 '15 at 19:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.