3

I am running a Zyxxel NAS540 with 4x4TB Hard Drives. A few days ago the NAS started beeping and I saw that the RAID is in a degraded state. Since one of the drives had really bad SMART values, I shut the NAS down, ordered a replacement disk, inserted it and started the repair process via web interface.

After a few days (was gone for the weekend) I wanted to check the progress and could not log into the web interface anymore. The NAS also showed no disk activity (neither LED flickering nor noise). I tried getting the state over ssh and mdadm, it said 3 drives are state clean, and one is in the state spare. So just as it would be before a successful repair? Because there was no access and also no noticeable disk access, I power-cycled the NAS.

Now the web interface worked again, but said "Volume down". There was no way of repairing it in the menu. Also, when I click on disks, it says that Disk 1,2 and 3 are "Hot Spare". Disk 4 has no state. The disk that I replaced was disk 3.

Since then, I put the disk in my PC and ran some commands to get clarity about the situation, since I also lost ssh access to the NAS. These are the outputs:

cat /proc/mdstat

Personalities : [raid6] [raid5] [raid4]
unused devices: <none>

sudo mdadm --examine /dev/sd[abcdef]3

/dev/sda3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 73e88019:b7cf694c:8584cbaa:47f57992
           Name : NAS540:2
  Creation Time : Tue Nov 24 23:18:19 2015
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 7805773824 (3722.08 GiB 3996.56 GB)
     Array Size : 11708660160 (11166.25 GiB 11989.67 GB)
  Used Dev Size : 7805773440 (3722.08 GiB 3996.56 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=384 sectors
          State : clean
    Device UUID : ac8c7cd6:a8f3d86e:cb210c2b:bcdfc2eb

    Update Time : Thu Nov 14 16:31:43 2019
       Checksum : 667f486f - correct
         Events : 1210

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 73e88019:b7cf694c:8584cbaa:47f57992
           Name : NAS540:2
  Creation Time : Tue Nov 24 23:18:19 2015
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 7805773824 (3722.08 GiB 3996.56 GB)
     Array Size : 11708660160 (11166.25 GiB 11989.67 GB)
  Used Dev Size : 7805773440 (3722.08 GiB 3996.56 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=384 sectors
          State : active
    Device UUID : 1bbec5f9:dec5a68a:d07cfdbe:e05d0cb4

    Update Time : Mon Nov 11 18:02:11 2019
       Checksum : 1cd3509 - correct
         Events : 74

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 2
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 73e88019:b7cf694c:8584cbaa:47f57992
           Name : NAS540:2
  Creation Time : Tue Nov 24 23:18:19 2015
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 7805773824 (3722.08 GiB 3996.56 GB)
     Array Size : 11708660736 (11166.25 GiB 11989.67 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=0 sectors
          State : clean
    Device UUID : 78f30bc0:b68074ee:9a3a223c:93decfd4

    Update Time : Sun Nov 17 23:41:48 2019
       Checksum : c9cda273 - correct
         Events : 1230

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 0
   Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 73e88019:b7cf694c:8584cbaa:47f57992
           Name : NAS540:2
  Creation Time : Tue Nov 24 23:18:19 2015
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 7805773824 (3722.08 GiB 3996.56 GB)
     Array Size : 11708660736 (11166.25 GiB 11989.67 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=0 sectors
          State : clean
    Device UUID : 85b74994:874b016e:609081d6:4cfcd0ee

    Update Time : Sun Nov 17 23:41:48 2019
       Checksum : d1f8a2d1 - correct
         Events : 1230

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 1
   Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)

sudo mdadm --examine --brief --scan --config=partitions

ARRAY /dev/md/2  metadata=1.2 UUID=73e88019:b7cf694c:8584cbaa:47f57992 name=NAS540:2
ARRAY /dev/md/0  metadata=1.2 UUID=b705c51b:2360cd8e:6b81c03f:2072f947 name=NAS540:0
ARRAY /dev/md/1  metadata=1.2 UUID=186ed461:615007c3:ab9e4576:7b5f7084 name=NAS540:1
ARRAY /dev/md/2  metadata=1.2 UUID=73e88019:b7cf694c:8584cbaa:47f57992 name=NAS540:2

sudo mdadm --assemble --scan

mdadm: Devices UUID-73e88019:b7cf694c:8584cbaa:47f57992 and UUID-73e88019:b7cf694c:8584cbaa:47f57992 have the same name: /dev/md/2
mdadm: Duplicate MD device names in conf file were found.

So now I'm wondering what the next steps should be like. I found some similar cases in which the /etc/mdadm.conf is edited manually to get rid of the duplicate entry.

In others the RAID is assembled manually (e.g mdadm--assemble /dev/mdX /dev/sda3 /dev/sdb3 ...) but here I don't know my RAID name (mdX).

Any ideas how I could proceed?

PS: Yes, I have a backup of the critical files, but it would still be nice to restore everything else as well.

I'm a few steps further now. I managed to reassemble the array using

mdadm --create --assume-clean --level=5  --raid-devices=4 --metadata=1.2 --chunk=64K  --layout=left-symmetric /dev/md2 /dev/sdd3 /dev/sde3 missing /dev/sda3

I then copied the partition layout over using

sfdisk -d

and finally joined the new disk with

sudo mdadm --manage /dev/md2 --add /dev/sdb3

The output of mdadm --details show every device as "clean" with a raid status of "AAAA". Now when i try to mount the raid I get the following error:

sudo mount /dev/md/2 /media/raid
mount: /media/raid: wrong fs type, bad option, bad superblock on /dev/md2, missing codepage or helper program, or other error.

There is a difference in the Data Offset now, which is suspected to come from a bitmap in the header.

Data Offset : 262144 sectors
Data Offset : 264192 sectors

The mdadm man page says:

When creating an array on devices which are 100G or larger, mdadm automatically adds an internal bitmap as it will usually be beneficial. This can be suppressed with --bitmap=none or by selecting a different consistency policy with --consistency-policy.

so I should have used --bitmap=none. Can I fix this by running it like it the man page says, or will that break something?

If the word none is given with --grow mode, then any bitmap that is present is removed.

I have managed to recreate the array by specifying the data-offset manually.

sudo mdadm --create --assume-clean --level=5  --raid-devices=4 --metadata=1.2 --chunk=64K  --layout=left-symmetric --data-offset=262144s /dev/md2 /dev/sdd3 /dev/sde3 /dev/sdc3 /dev/sda3

I can now mount this and access the data. It also works in the NAS.

  • /dev/sdb3 failed first, then /dev/sda3 failed too. Now it depends where the read errors are. It might be possible to recover with ddrescue, overlays, mdadm --assemble --force or re-create with correct parameters, preferably using only /dev/sd[ade]3 since /dev/sdb3 is the most outdated. – frostschutz Nov 21 '19 at 11:50
  • Also similar situation here: unix.stackexchange.com/a/490675/30851 - basically what you have is the classic double drive failure case, recovery is not entirely impossible but up to luck, depends how broken the drives are, and what filesystem inconsistency there may be after the RAID crash – frostschutz Nov 21 '19 at 11:57
0

As stated in the edits i solved my problem by running the mdadm --create command again with the correct data offset. My raid became readable again. Since i already ran the restauration on my new disk3, i was able to run the command as stated in my OP, but assuming you would be in the initial position of the failure you would run:

sudo mdadm --create --assume-clean --level=5  --raid-devices=4 --metadata=1.2 --chunk=64K  --layout=left-symmetric --data-offset=262144s /dev/md2 /dev/sdd3 /dev/sde3 missing /dev/sda3

followed by

sudo mdadm --manage /dev/md2 --add /dev/sdb3

of course use your own specifiers instead of the sdbX.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.