1

today I run in a disaster...

I have a RAID 6 with 7 HDDs and yesterday one disk failed. After replacing the disk and did a rebuild over night I found out that a 2nd HDD was out of the RAID...
So today I 've started to backup my Files on external Drives but then the copying stopped and as I've checked why and saw in Webmins RAID that sdg was "down".
I shut down the server and checked the hardware and found out, that the backplate, where the HDDs are connected got lose...

After repairing it all drives are now back but my RAID 6 don't start anymore :-/

dmesg shows me:
md: kicking non-fresh sdg from array!
md: kicking non-fresh sdf from array!
md: kicking non-fresh sde from array!
md/raid:md0: not enough operational devices (3/7 failed)
...
and after many
md0: ADD_NEW_DISK not supported
I can read this:
EXT4-fs (md0): unable to read superblock

With sudo mdadm --examine I checked the sdg, sdf and sde and e and f shows "State clean" where the sdg, which was "down" before repairing shows "Active". So 6 of 7 Devices shows "Clean" except the sdg.
Here is the list of the output of all devices:

Disk sdb
/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4
           Name : N5550:0  (local to host N5550)
  Creation Time : Fri Oct 29 14:43:58 2021
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 3906770096 (1862.89 GiB 2000.27 GB)
     Array Size : 9766906880 (9314.45 GiB 10001.31 GB)
  Used Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=7344 sectors
          State : clean
    Device UUID : 9180f101:1dacdd9e:4adae9c4:fbeb2552

Internal Bitmap : 8 sectors from superblock
    Update Time : Sat Mar 26 18:13:45 2022
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 38019182 - correct
         Events : 256508

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAA.A.. ('A' == active, '.' == missing, 'R' == replacing)
Disk sdc
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4
           Name : N5550:0  (local to host N5550)
  Creation Time : Fri Oct 29 14:43:58 2021
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 3906770096 (1862.89 GiB 2000.27 GB)
     Array Size : 9766906880 (9314.45 GiB 10001.31 GB)
  Used Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=7344 sectors
          State : clean
    Device UUID : 889c6877:5ee5c647:eebd209c:d9c6abcb

Internal Bitmap : 8 sectors from superblock
    Update Time : Sat Mar 26 18:13:45 2022
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : a71ea53d - correct
         Events : 256508

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAA.A.. ('A' == active, '.' == missing, 'R' == replacing)
Disk sdd
/dev/sdd:
   MBR Magic : aa55
Partition[0] :   3907026944 sectors at         2048 (type fd)
Disk sde
/dev/sde:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4
           Name : N5550:0  (local to host N5550)
  Creation Time : Fri Oct 29 14:43:58 2021
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 3906770096 (1862.89 GiB 2000.27 GB)
     Array Size : 9766906880 (9314.45 GiB 10001.31 GB)
  Used Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=7344 sectors
          State : clean
    Device UUID : 34198042:3d4c802b:36727b02:fdf65808

Internal Bitmap : 8 sectors from superblock
    Update Time : Sat Mar 26 18:05:00 2022
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : f8fb6b18 - correct
         Events : 256494

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAAA.. ('A' == active, '.' == missing, 'R' == replacing)
Disk sdf
/dev/sdf:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4
           Name : N5550:0  (local to host N5550)
  Creation Time : Fri Oct 29 14:43:58 2021
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 3906770096 (1862.89 GiB 2000.27 GB)
     Array Size : 9766906880 (9314.45 GiB 10001.31 GB)
  Used Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=7344 sectors
          State : clean
    Device UUID : b2e8d640:1f21336f:88d823fe:66ef7be7

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Mar 23 14:46:56 2022
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 15cd05bb - correct
         Events : 238681

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 4
   Array State : AAAAAA. ('A' == active, '.' == missing, 'R' == replacing)
Disk sdg
/dev/sdg:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4
           Name : N5550:0  (local to host N5550)
  Creation Time : Fri Oct 29 14:43:58 2021
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 3906770096 (1862.89 GiB 2000.27 GB)
     Array Size : 9766906880 (9314.45 GiB 10001.31 GB)
  Used Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=7344 sectors
          State : active
    Device UUID : 2bc06e22:49aa73e2:3cf7eb79:55df1180

Internal Bitmap : 8 sectors from superblock
    Update Time : Sat Mar 26 17:57:06 2022
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 7f0ddb2a - correct
         Events : 256372

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 5
   Array State : AAAAAA. ('A' == active, '.' == missing, 'R' == replacing)
Disk sdh
/dev/sdh:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4
           Name : N5550:0  (local to host N5550)
  Creation Time : Fri Oct 29 14:43:58 2021
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 3906770096 (1862.89 GiB 2000.27 GB)
     Array Size : 9766906880 (9314.45 GiB 10001.31 GB)
  Used Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=7344 sectors
          State : clean
    Device UUID : 7af89a18:52ef08ae:dec5ad7b:75626355

Internal Bitmap : 8 sectors from superblock
    Update Time : Sat Mar 26 18:13:45 2022
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 17d7b107 - correct
         Events : 256508

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 4
   Array State : AAA.A.. ('A' == active, '.' == missing, 'R' == replacing)

I've tried to start the RAID with

mdadm --run /dev/md0

and get:

mdadm: failed to start array /dev/md0: Input/output error

But after I started it with this Webmin shows me then:

/dev/md0    active, FAILED, Not Started     RAID6 (Dual Distributed Parity)     7.27 TiB

Its 7.27 from 9TB.

Any ideas how to get my RAID back to work again without data loss?

I've read about that I could add devices back again to the RAID but I'm unsure and wanted to ask before.

Any help would be appreciated!

UPDATE: I forgot that one of the device is /dev/sdd1 and not /sdd!
Here the examine of it:

/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4
           Name : N5550:0  (local to host N5550)
  Creation Time : Fri Oct 29 14:43:58 2021
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 3906767872 (1862.89 GiB 2000.27 GB)
     Array Size : 9766906880 (9314.45 GiB 10001.31 GB)
  Used Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=5120 sectors
          State : clean
    Device UUID : d8df004e:44ee4060:ba4d2c22:e7e6bdcb

Internal Bitmap : 8 sectors from superblock
    Update Time : Sat Mar 26 18:13:45 2022
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 1c4e98a4 - correct
         Events : 256508

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAA.A.. ('A' == active, '.' == missing, 'R' == replacing)

And here an mdadm -D /dev/md0:

/dev/md0:
           Version : 1.2
        Raid Level : raid0
     Total Devices : 7
       Persistence : Superblock is persistent

             State : inactive
   Working Devices : 7

              Name : N5550:0  (local to host N5550)
              UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4
            Events : 256494

    Number   Major   Minor   RaidDevice

       -       8       64        -        /dev/sde
       -       8       32        -        /dev/sdc
       -       8      112        -        /dev/sdh
       -       8       80        -        /dev/sdf
       -       8       16        -        /dev/sdb
       -       8       49        -        /dev/sdd1
       -       8       96        -        /dev/sdg

1 Answer 1

0

Looks like somehow a lot of drives got kicked from your array? If any of these drives had read errors (check smartctl -a for reallocated/pending/uncorrectable sectors and the like), you should ddrescue them to new drives before attempting any other kind of data recovery.

It's a 7 drive RAID-6, so you need at least 5 drives to run the array. Currently, you only have 3 (sdb, sdc, sdh), so it doesn't work...

If you check Update Time and Events for each drive, you see 1 missing entirely (not included in your output at all), 1 very outdated (sdf), 2 slightly outdated (sdg, sde), and only 3 drives up to date.

/dev/sdf: Update Time : Wed Mar 23 14:46:56 2022 Events: 238681
/dev/sdg: Update Time : Sat Mar 26 17:57:06 2022 Events: 256372
/dev/sde: Update Time : Sat Mar 26 18:05:00 2022 Events: 256494
/dev/sdb: Update Time : Sat Mar 26 18:13:45 2022 Events: 256508
/dev/sdc: Update Time : Sat Mar 26 18:13:45 2022 Events: 256508
/dev/sdh: Update Time : Sat Mar 26 18:13:45 2022 Events: 256508

In this situation, you can only try your luck with mdadm --assemble --force (to ignore the "non-fresh" Events counter), using only the 5 best drives, ignoring the very outdated and missing drives. Otherwise mdadm --create with two missing drives is also an option.

So sdf should be avoided in this assembly attempt — the more outdated a drive is, the more filesystem inconsistencies and data corruption you will likely encounter. It would only be a last resort if the other drives don't have any data at all (read errors) for some sectors...

If possible, run your mdadm experiments on copy-on-write overlays.

Good luck.

4
  • ... unless perhaps you meant to --examine /dev/sdd1 instead of /dev/sdd? Commented Mar 26, 2022 at 22:06
  • First many many thanks! And yes, you're right! It's /dev/ssd1. I forgot that sorry. I'll add this to my Question above. Maybe mdadm -D /dev/md0 is for interest too? I'll add this too.
    – LeChatNoir
    Commented Mar 27, 2022 at 6:46
  • Oh and the Drives are okay. I think the disconnection of the backplate kicked them out, or?. Maybe Debian made it now in a wrong order after I've repaired it? The drives haven't changed and are on the same place as before.
    – LeChatNoir
    Commented Mar 27, 2022 at 7:01
  • 1
    I was able to access my RAID with this Overlay instructions! Many thanks for this link! I can now backup my data. What next if I'm done with the backups? Same steps but without the overlay?
    – LeChatNoir
    Commented Mar 29, 2022 at 13:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .