0

I'm looking to recover data from 4 old HDDs set up in a software raid5 and it looks like a disk has failed. What I want to do is recover the raid so I can copy its data somewhere else. I have done some research and I believe I want to use mdadm to perform a resync, but at the end of the day I do not want to mess it up and I would appreciate if someone could explain what needs to be done in order to get that data to safety. Also I am on ubuntu 16.04, here is what I see when I run mdadm --detail /dev/md0

/dev/md0:
        Version : 1.1
  Creation Time : Thu Feb 13 09:03:27 2014
     Raid Level : raid5
     Array Size : 4395016704 (4191.41 GiB 4500.50 GB)
  Used Dev Size : 1465005568 (1397.14 GiB 1500.17 GB)
   Raid Devices : 4
  Total Devices : 3
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Sun Dec 23 12:51:56 2018
          State : clean, FAILED 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 1
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : CentOS-01:0
           UUID : 1cf7d605:8b0ef6c5:bccd8c1e:3e841f24
         Events : 4178728

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       33        1      active sync   /dev/sdc1
       2       8       65        2      active sync   /dev/sde1
       6       0        0        6      removed

       0       8       49        -      faulty   /dev/sdd1

Also, I ran mdadm --examine on each device:

    /dev/sdb1:
          Magic : a92b4efc
        Version : 1.1
    Feature Map : 0x1
     Array UUID : 1cf7d605:8b0ef6c5:bccd8c1e:3e841f24
           Name : CentOS-01:0
  Creation Time : Thu Feb 13 09:03:27 2014
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 2930012160 (1397.14 GiB 1500.17 GB)
     Array Size : 4395016704 (4191.41 GiB 4500.50 GB)
  Used Dev Size : 2930011136 (1397.14 GiB 1500.17 GB)
    Data Offset : 262144 sectors
   Super Offset : 0 sectors
   Unused Space : before=262072 sectors, after=1024 sectors
          State : clean
    Device UUID : 252a74c1:fae726d9:179963f2:e4694a65

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Mar 15 07:05:19 2015
       Checksum : 53cae08e - correct
         Events : 130380

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

    /dev/sdc1:
          Magic : a92b4efc
        Version : 1.1
    Feature Map : 0x1
     Array UUID : 1cf7d605:8b0ef6c5:bccd8c1e:3e841f24
           Name : CentOS-01:0
  Creation Time : Thu Feb 13 09:03:27 2014
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 2930012160 (1397.14 GiB 1500.17 GB)
     Array Size : 4395016704 (4191.41 GiB 4500.50 GB)
  Used Dev Size : 2930011136 (1397.14 GiB 1500.17 GB)
    Data Offset : 262144 sectors
   Super Offset : 0 sectors
   Unused Space : before=262072 sectors, after=1024 sectors
          State : clean
    Device UUID : dc8c18bd:e92ba6d3:b303ee86:01bd6451

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Dec 23 14:18:53 2018
       Checksum : d1ed82ce - correct
         Events : 4178730

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : .AA. ('A' == active, '.' == missing, 'R' == replacing)

    /dev/sdd1:
          Magic : a92b4efc
        Version : 1.1
    Feature Map : 0x1
     Array UUID : 1cf7d605:8b0ef6c5:bccd8c1e:3e841f24
           Name : CentOS-01:0
  Creation Time : Thu Feb 13 09:03:27 2014
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 2930012160 (1397.14 GiB 1500.17 GB)
     Array Size : 4395016704 (4191.41 GiB 4500.50 GB)
  Used Dev Size : 2930011136 (1397.14 GiB 1500.17 GB)
    Data Offset : 262144 sectors
   Super Offset : 0 sectors
   Unused Space : before=262072 sectors, after=1024 sectors
          State : active
    Device UUID : 03a2de27:7993c129:23762f07:f4ba7ff8

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Dec 23 12:48:03 2018
       Checksum : ba2a5a95 - correct
         Events : 4178721

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)

    /dev/sde1:
          Magic : a92b4efc
        Version : 1.1
    Feature Map : 0x1
     Array UUID : 1cf7d605:8b0ef6c5:bccd8c1e:3e841f24
           Name : CentOS-01:0
  Creation Time : Thu Feb 13 09:03:27 2014
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 2930012160 (1397.14 GiB 1500.17 GB)
     Array Size : 4395016704 (4191.41 GiB 4500.50 GB)
  Used Dev Size : 2930011136 (1397.14 GiB 1500.17 GB)
    Data Offset : 262144 sectors
   Super Offset : 0 sectors
   Unused Space : before=262072 sectors, after=1024 sectors
          State : clean
    Device UUID : c00a8798:51804c50:3fe76211:8aafd9b1

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Dec 23 14:18:53 2018
       Checksum : 14ec2b30 - correct
         Events : 4178730

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : .AA. ('A' == active, '.' == missing, 'R' == replacing)

EDIT: Following @frostschutz advice, I have run:

server:~$ sudo mdadm --stop /dev/md0

That successfully stopped the raid. After that I ran:

server:~$ sudo mdadm --assemble --force /dev/md0 /dev/sdc1 /dev/sdd1 /dev/sde1`
mdadm: forcing event count in /dev/sdd1(0) from 4178721 upto 4178730
mdadm: Marking array /dev/md0 as 'clean'
mdadm: /dev/md0 assembled from 3 drives - not enough to start the array.

That didn't look so good, but I still tried the following:

server:~$ sudo mdadm --assemble --scan
mdadm: /dev/md/0 has been started with 3 drives (out of 4).

After that the array is now 'Active,Degraded' with 3 disks in 'active sync' and the last one removed. I am pleased to report that I have successfully started to copy the data into a safer place (at least so far the rsync command shows no error message, but I guess we shall see).

Improve this question
0

1 Answer 1

Reset to default
3

So... /dev/sdb1 hasn't been active in this array since 2015 (Update Time). The data on it should be outdated to the point of uselessness. Essentially you've been running a RAID-0 ever since.

That leaves you the three other devices /dev/sd{c,d,e}1. Out of these, /dev/sdd1 failed recently. Since you already lost redundancy years ago, this failure effectively stopped your RAID from working at all.

Now it depends. Are these three drives still readable? Then you can probably recover data. Otherwise, it's game over. So check smartctl -a. If any drives have bad or reallocated sectors, use ddrescue to copy them to a new drive.

If the drives are intact, given a recent enough kernel (4.10+) and mdadm (v4.x), you can probably assemble it like such:

mdadm --stop /dev/md0
mdadm --assemble --force /dev/md0 /dev/sdc1 /dev/sdd1 /dev/sde1

(There was a bug with assemble force in older versions, I'm not sure exactly which version though.)

...and if that doesn't work, you're left with mdadm --create but this is a path wrought with danger, see also https://unix.stackexchange.com/a/131927/30851

Improve this answer
7
  • 1
    I believe RAID is been working in degraded mode (because RAID 5 can do it), and now the last one went berserk...well you know what that means.
    – Rui F Ribeiro
    Commented Dec 23, 2018 at 22:30
  • Ah! You noticed same, but explained it better than me! +1 However, looking at OPs mgmt of the RAID, I would take a ddrescue of all the drives and have him work on a copy. And even then: he's better off restoring his backup.
    – Fabby
    Commented Dec 23, 2018 at 22:31
  • 1
    If there was a backup, they probably wouldn't have asked in the first place. That's just the way of things - people buy a RAID-in-a-box, plug it in, turn it on, then are surprised when it magically stops working a few years later. RAID needs monitoring and timely replacements of failing drives, but that's not how most people (even among sysadmins who do this for a living) actually run it...
    – frostschutz
    Commented Dec 23, 2018 at 22:38
  • 1
    The event counts aren't the same on the remaining three drives, so it's probable there will be some filesystem corruption. How much and how critical it is will be down to chance.
    – Chris Davies
    Commented Dec 23, 2018 at 22:42
  • 1
    @frostschutz I wasn't able to run the smartctl command but still tried my luck with what you suggested. It seems to have worked! Thank you very much for your help!
    – MarkoPaulo
    Commented Dec 24, 2018 at 16:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .