3

Yesterday I wanted to do some maintenance on my server. I shut it down pressing the powerbutton once which works just fine every time.

After the server was still shutting down after 10 minutes I called it a day and forced it off using the powerbutton. (I tried getting into it with ssh before forcing it off but the ssh service was already stopped).

After doing the maintenance and rebooting the server I noticed my RAID5 consisting of 7x 2TB disks did not work anymore. It was split into two RAIDs that consisted of 5 disks and 2 disks all in (S) mode (spare), inactive.

I tried mdadm --assemble --scan --run -f which did not help:

mdadm: Merging with already-assembled /dev/md/128
mdadm: failed to add /dev/sdc1 to /dev/md/128: Invalid argument
mdadm: failed to add /dev/sde1 to /dev/md/128: Invalid argument
mdadm: failed to RUN_ARRAY /dev/md/128: Input/output error
mdadm: No arrays found in config file or automatically

It seemed to half assemble things:

cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md128 : inactive sda1[0] sdg1[6] sdf1[5] sdd1[7] sdb1[1]
      9766891962 blocks super 1.2

unused devices: <none>

I also tried re-assembling it manually using mdadm --assemble --run /dev/md0 /dev/sd[abcdefg]1 --verbose:

mdadm: looking for devices for /dev/md0
mdadm: /dev/sda1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 6.
mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 5.
mdadm: added /dev/sdb1 to /dev/md0 as 1
mdadm: failed to add /dev/sdc1 to /dev/md0: Invalid argument
mdadm: failed to add /dev/sde1 to /dev/md0: Invalid argument
mdadm: added /dev/sdd1 to /dev/md0 as 4
mdadm: added /dev/sdg1 to /dev/md0 as 5
mdadm: added /dev/sdf1 to /dev/md0 as 6
mdadm: added /dev/sda1 to /dev/md0 as 0
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error

Now examining all disks mdadm --examine /dev/sd[abcdefg]1 has this output see it on hastebin.com which to me looks like everything is just fine.

Here is the disks using lsblk

NAME                      MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                         8:0    0   1,8T  0 disk 
└─sda1                      8:1    0   1,8T  0 part 
sdb                         8:16   0   1,8T  0 disk 
└─sdb1                      8:17   0   1,8T  0 part 
sdc                         8:32   0   1,8T  0 disk 
└─sdc1                      8:33   0   1,8T  0 part 
sdd                         8:48   0   1,8T  0 disk 
└─sdd1                      8:49   0   1,8T  0 part 
sde                         8:64   1   1,8T  0 disk 
└─sde1                      8:65   1   1,8T  0 part 
sdf                         8:80   1   1,8T  0 disk 
└─sdf1                      8:81   1   1,8T  0 part 
sdg                         8:96   1   1,8T  0 disk 
└─sdg1                      8:97   1   1,8T  0 part 

The HDDs in use are not the best but they work. SMART output for all drives from sda to sdg can be found on hastebin.com too.

Due to the fact that two disks of my RAID5 produce errors I assume that all data is already lost. ...

EDIT 1:

dmesg -T returns:

[Sa Okt  7 15:41:08 2017] md/raid:md128: device sda1 operational as raid disk 0
[Sa Okt  7 15:41:08 2017] md/raid:md128: device sdf1 operational as raid disk 6
[Sa Okt  7 15:41:08 2017] md/raid:md128: device sdb1 operational as raid disk 1
[Sa Okt  7 15:41:08 2017] md/raid:md128: device sdd1 operational as raid disk 4
[Sa Okt  7 15:41:08 2017] md/raid:md128: device sdg1 operational as raid disk 5
[Sa Okt  7 15:41:08 2017] md/raid:md128: not enough operational devices (2/7 failed)
[Sa Okt  7 15:41:08 2017] md/raid:md128: failed to run raid set.
[Sa Okt  7 15:41:08 2017] md: pers->run() failed ...
[Sa Okt  7 15:41:12 2017] md: md127 stopped.
[Sa Okt  7 15:41:15 2017] md: md128 stopped.
[Sa Okt  7 15:41:20 2017] md: md0 stopped.
[Sa Okt  7 15:41:20 2017] md: sdc1 does not have a valid v1.2 superblock, not importing!
[Sa Okt  7 15:41:20 2017] md: md_import_device returned -22
[Sa Okt  7 15:41:20 2017] md: sde1 does not have a valid v1.2 superblock, not importing!
[Sa Okt  7 15:41:20 2017] md: md_import_device returned -22

How can I repair the superblocks?


Am I doing something wrong here?

Why do I get:

mdadm: failed to add [...] to [...]: Invalid argument?

What argument is invalid here?

How can I debug this further?

  • 1
    please show mdadm --examine /dev/sd* for all drives – frostschutz Oct 7 '17 at 14:09
  • Please see the link I also posted in my question: hastebin.com/yujebiwite.sql – Flatron Oct 7 '17 at 14:45
  • 1
    Missed that, sorry. Very odd it says Unused Space : before=262056 sectors, after=18446744073709289480 sectors. Can you check device sizes? (blockdev --getsize64 /dev/sd*) Which mdadm version? – frostschutz Oct 7 '17 at 15:05
  • 1
    for the partitions? Also parted -l, do those two devices stand out somehow, like GPT vs. msdos table scheme? – frostschutz Oct 7 '17 at 16:41
  • 1
    I can reproduce your problem - happens if the correct metadata is on a too small partition. Thus please re-check your device (partition) sizes and/or see if your kernel supports your partitioning scheme. – frostschutz Oct 7 '17 at 18:36
1

Warning: This answer is about the decisive symptom but it turned out that the real answer is different from what I suggested.

However this may have happened: The problem is probably this:

Unused Space : before=262056 sectors, after=177 sectors
Unused Space : before=262056 sectors, after=177 sectors
Unused Space : before=262056 sectors, after=18446744073709289480 sectors
Unused Space : before=262056 sectors, after=177 sectors
Unused Space : before=262056 sectors, after=18446744073709289480 sectors
Unused Space : before=262056 sectors, after=177 sectors
Unused Space : before=262056 sectors, after=177 sectors

I cannot offer a pleasant way of correcting that. You should make a backup of the MD metadata of sdc1 and then have a look at the on-disk format and use a hex editor to fix this.

Maybe you can just copy the relevant part with dd from one of the other disks. You "just" have to find out where these bytes are.

Kind of funny is this:

   Checksum : 85f67f98 - correct
   Checksum : 6a4fb921 - correct
   Checksum : 92db2c10 - correct
   Checksum : ad5c81b8 - correct
   Checksum : a657023 - correct
   Checksum : 6880d6c7 - correct
   Checksum : c0c31cf - correct

So maybe correcting the metadata destroys the checksum. I do not know if that is a real problem but at that point it probably makes sense to ask a new question.

  • Thanks for your reply. Please have a look at my last comment above. Most likely this caused the issue. – Flatron Oct 8 '17 at 9:23
  • Can you go a bit deeper into how to use what hex editor to change what exactly? I'd greatly appreciate this. – Flatron Oct 8 '17 at 13:12
  • @Flatron Currently I assume that there is nothing wrong with the MD metadata and that the problem is caused by a problem with the partitions. Information about the superblock format is here: raid.wiki.kernel.org/index.php/RAID_superblock_formats – Hauke Laging Oct 8 '17 at 13:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.