3

my level of Linux knowledge is very light (I'm learning) and I've got a problem that's beyond my level of Google-Fu. I'm well out of my depth and I'm hoping somebody might be able to help me...

I'm running OpenMediaVault 5.5.3-1 (Usul) in my homelab and have two software RAID arrays /dev/md0 and /dev/md1. When I logged on this morning I found that /dev/md1, a four-disk RAID5 set (aka /dev/disk/by-label/RAID5) was not available. It holds a lot of data I'd really prefer not to lose.

What I've done so far:

  1. I checked the OpenMediaVault GUI which told me that raidset was "Missing"

  2. I determined that one of the four disks that make up the Raidset wasn't showing up at BIOS level. I reseated the cables and all four drives (sdc, sdd, sde and sdh) are now visible to the OS.

  3. Upon reboot, the raidset is still showing as "missing" and there's a message coming up in dmesg

    EXT4-fs (md1): unable to read superblock
    
  4. /proc/mdstat confirms that md1 is inactive:

    root@OpenMediaTower:~# cat /proc/mdstat
    Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
    md1 : inactive sdh[3](S) sde[2](S) sdd[1](S) sdc[0](S)
          7813529952 blocks super 1.2
    
    md0 : active raid1 sdf[1] sda[0]
          3906886464 blocks super 1.2 [2/2] [UU]
          bitmap: 0/30 pages [0KB], 65536KB chunk
    
    unused devices: <none>
    
  5. Through google I found a previous post that dealt with a somewhat similar problem and from that I've had a look at mdadm, but unforunately am insufficiently knowledgable to use the detail to dignose and address the issue

Output from mdadm --detail

root@OpenMediaTower:~# mdadm --detail --scan
ARRAY /dev/md0 metadata=1.2 name=OpenMediaTower:Mirror UUID=f7b1c667:3df80c11:975d87ad:126b5401
INACTIVE-ARRAY /dev/md1 metadata=1.2 name=RAID5 UUID=c813cb15:a9d26d51:7faada85:9b76b36d

Attempt to reassemble the raidset

root@OpenMediaTower:~# mdadm --stop
/dev/md1 mdadm: stopped /dev/md1

root@OpenMediaTower:~# mdadm --assemble --scan
mdadm: /dev/md1 assembled from 2 drives - not enough to start the array.

Output from mdadm --examine

root@OpenMediaTower:~# mdadm --examine /dev/sd[dehc]
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : c813cb15:a9d26d51:7faada85:9b76b36d
           Name : RAID5
  Creation Time : Sun May 17 14:49:46 2020
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
     Array Size : 5860147200 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=176 sectors
          State : clean
    Device UUID : 42088567:13765c92:e2a5503b:c30355d0

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Jul 22 08:35:02 2020
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : dd641c67 - correct
         Events : 218937

         Layout : left-symmetric
     Chunk Size : 512K

    Device Role : Active device 0
    Array State : A.AA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : c813cb15:a9d26d51:7faada85:9b76b36d
           Name : RAID5
  Creation Time : Sun May 17 14:49:46 2020
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
     Array Size : 5860147200 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=176 sectors
          State : active
    Device UUID : 69f98cc0:b818da43:b883695d:2246b3ab

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Jul 19 03:32:33 2020
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : e55cb645 - correct
         Events : 30843

         Layout : left-symmetric
     Chunk Size : 512K

    Device Role : Active device 1
    Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sde:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : c813cb15:a9d26d51:7faada85:9b76b36d
           Name : RAID5
  Creation Time : Sun May 17 14:49:46 2020
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
     Array Size : 5860147200 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=176 sectors
          State : active
    Device UUID : 04ec1c61:a7f1bb11:ee13bfe0:7153e38a

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Jul 21 21:59:53 2020
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : b6579d04 - correct
         Events : 216993

         Layout : left-symmetric
     Chunk Size : 512K

    Device Role : Active device 2
    Array State : A.AA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdh:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : c813cb15:a9d26d51:7faada85:9b76b36d
           Name : RAID5
  Creation Time : Sun May 17 14:49:46 2020
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 3906764976 (1862.89 GiB 2000.26 GB)
     Array Size : 5860147200 (5588.67 GiB 6000.79 GB)
  Used Dev Size : 3906764800 (1862.89 GiB 2000.26 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=176 sectors
          State : clean
    Device UUID : 2af17abb:48930a97:31ce1fa5:850ac7d0

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Jul 22 08:35:02 2020
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : a0495199 - correct
         Events : 218937

         Layout : left-symmetric
     Chunk Size : 512K

    Device Role : Active device 3
    Array State : A.AA ('A' == active, '.' == missing, 'R' == replacing)
  1. I ran e2fsck which reports a problem with the superblock. At this stage I have not attempted
    root@OpenMediaTower:~# e2fsck /dev/md1
    e2fsck 1.45.5 (07-Jan-2020)
    e2fsck: Invalid argument while trying to open /dev/md1
    
    The superblock could not be read or does not describe a validext2/ext3/ext4
    filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
    filesystem (and not swap or ufs or something else), then the superblock
    is corrupt, and you might try running e2fsck with an alternate superblock:
        e2fsck -b 8193 <device>
     or
        e2fsck -b 32768 <device>
    
  2. I tried to reassemble the raidset, discretely naming each of the disks
    root@OpenMediaTower:~# mdadm --verbose --assemble /dev/md1 /dev/sdc /dev/sdd /dev/sde /dev/sdh
    mdadm: looking for devices for /dev/md1
    mdadm: /dev/sdc is identified as a member of /dev/md1, slot 0.
    mdadm: /dev/sdd is identified as a member of /dev/md1, slot 1.
    mdadm: /dev/sde is identified as a member of /dev/md1, slot 2.
    mdadm: /dev/sdh is identified as a member of /dev/md1, slot 3.
    mdadm: added /dev/sdd to /dev/md1 as 1 (possibly out of date)
    mdadm: added /dev/sde to /dev/md1 as 2 (possibly out of date)
    mdadm: added /dev/sdh to /dev/md1 as 3
    mdadm: added /dev/sdc to /dev/md1 as 0
    mdadm: /dev/md1 assembled from 2 drives - not enough to start the array.
    
3
  • 1
    Your second disk hasn't been part of the array for a very long time (look at the event count), and one of the remaining three had got out of sync with the other two. You might be able to force a build but I'm going to defer (for now) to others rather then recommend a potentially disastrous course of action – roaima Jul 22 '20 at 10:17
  • 1
    You're right. I've left that one out of the rebuild and am still finding a number of areas of corruption with the one that's much less out of date. Once I've got a full picture on what I've lost and have created the set from scratch I'll be much more careful with SMART monitoring and backup! In the interim, I'm very grateful for an active and friendly community of knowledgable people. Thank you very much :) – Phil Green Jul 23 '20 at 18:27
  • Good news... mostly, I guess. If frostschutz's answer helped please consider accepting it (with the tick mark). – roaima Jul 23 '20 at 18:52
1

To avoid further data loss, set up a copy-on-write overlay and use it for all your experiments.

Then you can try your luck with assemble force without the drive with the most outdated Update Time and Event Count, /dev/sdd.

mdadm --stop /dev/md1
mdadm --assemble /dev/md1 --force /dev/mapper/sdc /dev/mapper/sde /dev/mapper/sdh

And with any luck, that would give you access to your data (in degraded mode).

However, the big question is how it fell apart on the first place. So do check SMART values, and if any drives have reallocated / pending / uncorrectable sectors, you'll need new drives and ddrescue before proceeding. It's not a good idea to attempt data recovery on failing drives.

1
  • 1
    Many thanks for the advice. I've done as you've suggested and the array has indeed come back in a degraded state. I'm busy copying everything I can to another location. There's some corruption, but so far it seems manageable. I only had very basic disk monitoring in place which shows some bad sectors on one disk but no outright failure. There have also been some power failures recently, perhaps these contributed? Full SMART monitoring is now in place. Once I've recovered what I can, I'll tear down and rebuild the array and be much more vigilant with backups! Again, many thanks! :) – Phil Green Jul 23 '20 at 9:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.