RAID5 repair froze on NAS, trying to restore with mdadm now

Question

I am running a Zyxxel NAS540 with 4x4TB Hard Drives. A few days ago the NAS started beeping and I saw that the RAID is in a degraded state. Since one of the drives had really bad SMART values, I shut the NAS down, ordered a replacement disk, inserted it and started the repair process via web interface.

After a few days (was gone for the weekend) I wanted to check the progress and could not log into the web interface anymore. The NAS also showed no disk activity (neither LED flickering nor noise). I tried getting the state over ssh and mdadm, it said 3 drives are state clean, and one is in the state spare. So just as it would be before a successful repair? Because there was no access and also no noticeable disk access, I power-cycled the NAS.

Now the web interface worked again, but said "Volume down". There was no way of repairing it in the menu. Also, when I click on disks, it says that Disk 1,2 and 3 are "Hot Spare". Disk 4 has no state. The disk that I replaced was disk 3.

Since then, I put the disk in my PC and ran some commands to get clarity about the situation, since I also lost ssh access to the NAS. These are the outputs:

cat /proc/mdstat

Personalities : [raid6] [raid5] [raid4]
unused devices: <none>

sudo mdadm --examine /dev/sd[abcdef]3

/dev/sda3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 73e88019:b7cf694c:8584cbaa:47f57992
           Name : NAS540:2
  Creation Time : Tue Nov 24 23:18:19 2015
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 7805773824 (3722.08 GiB 3996.56 GB)
     Array Size : 11708660160 (11166.25 GiB 11989.67 GB)
  Used Dev Size : 7805773440 (3722.08 GiB 3996.56 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=384 sectors
          State : clean
    Device UUID : ac8c7cd6:a8f3d86e:cb210c2b:bcdfc2eb

    Update Time : Thu Nov 14 16:31:43 2019
       Checksum : 667f486f - correct
         Events : 1210

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 73e88019:b7cf694c:8584cbaa:47f57992
           Name : NAS540:2
  Creation Time : Tue Nov 24 23:18:19 2015
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 7805773824 (3722.08 GiB 3996.56 GB)
     Array Size : 11708660160 (11166.25 GiB 11989.67 GB)
  Used Dev Size : 7805773440 (3722.08 GiB 3996.56 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=384 sectors
          State : active
    Device UUID : 1bbec5f9:dec5a68a:d07cfdbe:e05d0cb4

    Update Time : Mon Nov 11 18:02:11 2019
       Checksum : 1cd3509 - correct
         Events : 74

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 2
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 73e88019:b7cf694c:8584cbaa:47f57992
           Name : NAS540:2
  Creation Time : Tue Nov 24 23:18:19 2015
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 7805773824 (3722.08 GiB 3996.56 GB)
     Array Size : 11708660736 (11166.25 GiB 11989.67 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=0 sectors
          State : clean
    Device UUID : 78f30bc0:b68074ee:9a3a223c:93decfd4

    Update Time : Sun Nov 17 23:41:48 2019
       Checksum : c9cda273 - correct
         Events : 1230

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 0
   Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 73e88019:b7cf694c:8584cbaa:47f57992
           Name : NAS540:2
  Creation Time : Tue Nov 24 23:18:19 2015
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 7805773824 (3722.08 GiB 3996.56 GB)
     Array Size : 11708660736 (11166.25 GiB 11989.67 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=0 sectors
          State : clean
    Device UUID : 85b74994:874b016e:609081d6:4cfcd0ee

    Update Time : Sun Nov 17 23:41:48 2019
       Checksum : d1f8a2d1 - correct
         Events : 1230

         Layout : left-symmetric
     Chunk Size : 64K

   Device Role : Active device 1
   Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)

sudo mdadm --examine --brief --scan --config=partitions

ARRAY /dev/md/2  metadata=1.2 UUID=73e88019:b7cf694c:8584cbaa:47f57992 name=NAS540:2
ARRAY /dev/md/0  metadata=1.2 UUID=b705c51b:2360cd8e:6b81c03f:2072f947 name=NAS540:0
ARRAY /dev/md/1  metadata=1.2 UUID=186ed461:615007c3:ab9e4576:7b5f7084 name=NAS540:1
ARRAY /dev/md/2  metadata=1.2 UUID=73e88019:b7cf694c:8584cbaa:47f57992 name=NAS540:2

sudo mdadm --assemble --scan

mdadm: Devices UUID-73e88019:b7cf694c:8584cbaa:47f57992 and UUID-73e88019:b7cf694c:8584cbaa:47f57992 have the same name: /dev/md/2
mdadm: Duplicate MD device names in conf file were found.

So now I'm wondering what the next steps should be like. I found some similar cases in which the /etc/mdadm.conf is edited manually to get rid of the duplicate entry.

In others the RAID is assembled manually (e.g mdadm--assemble /dev/mdX /dev/sda3 /dev/sdb3 ...) but here I don't know my RAID name (mdX).

Any ideas how I could proceed?

PS: Yes, I have a backup of the critical files, but it would still be nice to restore everything else as well.

I'm a few steps further now. I managed to reassemble the array using

mdadm --create --assume-clean --level=5  --raid-devices=4 --metadata=1.2 --chunk=64K  --layout=left-symmetric /dev/md2 /dev/sdd3 /dev/sde3 missing /dev/sda3

I then copied the partition layout over using

sfdisk -d

and finally joined the new disk with

sudo mdadm --manage /dev/md2 --add /dev/sdb3

The output of mdadm --details show every device as "clean" with a raid status of "AAAA". Now when i try to mount the raid I get the following error:

sudo mount /dev/md/2 /media/raid
mount: /media/raid: wrong fs type, bad option, bad superblock on /dev/md2, missing codepage or helper program, or other error.

There is a difference in the Data Offset now, which is suspected to come from a bitmap in the header.

Data Offset : 262144 sectors

Data Offset : 264192 sectors

The mdadm man page says:

When creating an array on devices which are 100G or larger, mdadm automatically adds an internal bitmap as it will usually be beneficial. This can be suppressed with --bitmap=none or by selecting a different consistency policy with --consistency-policy.

so I should have used --bitmap=none. Can I fix this by running it like it the man page says, or will that break something?

If the word none is given with --grow mode, then any bitmap that is present is removed.

I have managed to recreate the array by specifying the data-offset manually.

sudo mdadm --create --assume-clean --level=5  --raid-devices=4 --metadata=1.2 --chunk=64K  --layout=left-symmetric --data-offset=262144s /dev/md2 /dev/sdd3 /dev/sde3 /dev/sdc3 /dev/sda3

I can now mount this and access the data. It also works in the NAS.

Answer 1

As stated in the edits i solved my problem by running the mdadm --create command again with the correct data offset. My raid became readable again. Since i already ran the restauration on my new disk3, i was able to run the command as stated in my OP, but assuming you would be in the initial position of the failure you would run:

sudo mdadm --create --assume-clean --level=5  --raid-devices=4 --metadata=1.2 --chunk=64K  --layout=left-symmetric --data-offset=262144s /dev/md2 /dev/sdd3 /dev/sde3 missing /dev/sda3

followed by

sudo mdadm --manage /dev/md2 --add /dev/sdb3

of course use your own specifiers instead of the sdbX.