3

And so, it happened: the software RAID5 on my Linux box failed somehow and now refuses to assemble.

An overview of the setup:

  • headless server running Debian Jessie 8.6 (current stable)
  • custom built Linux kernel v4.8.8 (see below), largely based on the official Debian packages - mainly done for the updated BTRFS code
  • all packages are up to date as of right now, Nov. 18 2016
  • four HDDs in total:
    • /dev/sda is partitioned for /boot/efi, swap and / (in that order)
    • /dev/sdb is member of the RAID (no partition table)
    • /dev/sd[cd] each have a single partition spanning the entire drive, with sd[cd]1 being member of the RAID
    • sda is 500GB, sd[bcd] are 4TB disks. sd[cd]1 are partitioned for the exact same size, sdb not being partitioned at all is slightly larger (wasting some space, which would otherwise be used for the partition table).
  • after assembling into /dev/md0, the RAID is further partitioned with a single partition /dev/md0p1 spanning the entire multi-disk, which then holds the BTRFS filesystem. (This was set up by the Debian installer, and it is too late to change.)

I recently bought a fifth disk, which was dead on arrival. I had to leave the site shortly after adding the disk to the server, so I noticed that fact not until a day or two later, but as this new disk wasn't part of the RAID yet, that should not change anything.

When I added the newly bought disk, the server was running kernel 4.8.7 and everything seemed fine. I compiled kernel version 4.8.8 and installed it, shut the server down with systemctl poweroff, removed the dead disk and tried rebooting. As the server is usually headless, I could only see it not coming back up after about 5 minutes of grace time. It wouldn't react to simply pressing the power button, so I cut power, added a graphics card (it has no internal one) + monitor + keyboard and was subsequently presented with systemd's emergency shell.

I get the emergency shell with both the previous and the current kernel, so I assume this is either not the cause or the damage is done already. I am currently dding all three RAID disks into /dev/null to rule out possible physical disk failures, but as they're already over 1TB into the disks without any errors, I assume the superblock is not physically damaged on any of the disks.


Enough talk, let's be more specific. All following commands are issued from the emergency root shell, i.e. with a basic system running (not the initrd).

/proc/mdstat knows nothing about the RAID:

Personalities : [raid6] [raid5] [raid4]
unused devices: <none>

This is what I get with mdadm --assemble --verbose /dev/md0: (/dev/sde here is an external USB HDD that I used to pull out these logs)

mdadm: looking for devices for /dev/md0
mdadm: no RAID superblock on /dev/sde1
mdadm: no RAID superblock on /dev/sde
mdadm: no RAID superblock on /dev/sdd
mdadm: no RAID superblock on /dev/sdc
mdadm: no RAID superblock on /dev/sda3
mdadm: no RAID superblock on /dev/sda2
mdadm: no RAID superblock on /dev/sda1
mdadm: no RAID superblock on /dev/sda
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdb is identified as a member of /dev/md0, slot 0.
mdadm: failed to add /dev/sdc1 to /dev/md0: Invalid argument
mdadm: failed to add /dev/sdd1 to /dev/md0: Invalid argument
mdadm: added /dev/sdb to /dev/md0 as 0
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error

... with these lines in journalctl:

Nov 18 14:56:34 zoey kernel: md: sdc1 does not have a valid v1.2 superblock, not importing!
Nov 18 14:56:34 zoey kernel: md: md_import_device returned -22
Nov 18 14:56:34 zoey kernel: md: sdd1 does not have a valid v1.2 superblock, not importing!
Nov 18 14:56:34 zoey kernel: md: md_import_device returned -22
Nov 18 14:56:34 zoey kernel: md: bind<sdb>
Nov 18 14:56:34 zoey kernel: md/raid:md0: device sdb operational as raid disk 0
Nov 18 14:56:34 zoey kernel: md/raid:md0: allocated 3316kB
Nov 18 14:56:34 zoey kernel: md/raid:md0: not enough operational devices (2/3 failed)
Nov 18 14:56:34 zoey kernel: RAID conf printout:
Nov 18 14:56:34 zoey kernel:  --- level:5 rd:3 wd:1
Nov 18 14:56:34 zoey kernel:  disk 0, o:1, dev:sdb
Nov 18 14:56:34 zoey kernel: md/raid:md0: failed to run raid set.
Nov 18 14:56:34 zoey kernel: md: pers->run() failed ...
Nov 18 14:56:34 zoey kernel: md: md0 stopped.
Nov 18 14:56:34 zoey kernel: md: unbind<sdb>
Nov 18 14:56:34 zoey kernel: md: export_rdev(sdb)

Oh no, the superblock is broken, and even mdadm --assemble --force won't assemble! Let's see what mdadm --examine has to say:

/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : cdab20e2:45150011:f10d83fe:f0299188
           Name : zoey:0  (local to host zoey)
  Creation Time : Fri Sep 26 17:23:15 2014
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 7813840560 (3725.93 GiB 4000.69 GB)
     Array Size : 7813770240 (7451.79 GiB 8001.30 GB)
  Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
    Data Offset : 196608 sectors
   Super Offset : 8 sectors
   Unused Space : before=196520 sectors, after=70320 sectors
          State : clean
    Device UUID : 1751c53f:c527c4cd:10e8f426:0217bf5e

    Update Time : Fri Nov 18 04:02:16 2016
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 7fa86509 - correct
         Events : 112125

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : cdab20e2:45150011:f10d83fe:f0299188
           Name : zoey:0  (local to host zoey)
  Creation Time : Fri Sep 26 17:23:15 2014
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 7813902336 (3725.96 GiB 4000.72 GB)
     Array Size : 7813770240 (7451.79 GiB 8001.30 GB)
  Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
    Data Offset : 196608 sectors
   Super Offset : 8 sectors
   Unused Space : before=196528 sectors, after=66560 sectors
          State : clean
    Device UUID : 0c60e19b:daff087f:c1f09dbb:41bb80c3

    Update Time : Fri Nov 18 04:02:16 2016
       Checksum : 866cea0a - correct
         Events : 112125

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : cdab20e2:45150011:f10d83fe:f0299188
           Name : zoey:0  (local to host zoey)
  Creation Time : Fri Sep 26 17:23:15 2014
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 7813902336 (3725.96 GiB 4000.72 GB)
     Array Size : 7813770240 (7451.79 GiB 8001.30 GB)
  Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
    Data Offset : 196608 sectors
   Super Offset : 8 sectors
   Unused Space : before=196528 sectors, after=66560 sectors
          State : clean
    Device UUID : e80a2296:dfadb7ad:e458e491:bf1d12b0

    Update Time : Fri Nov 18 04:02:16 2016
       Checksum : 72340d8d - correct
         Events : 112125

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)

mdadm --examine --scan --verbose also seems to find all the disks just fine:

ARRAY /dev/md/0  level=raid5 metadata=1.2 num-devices=3 UUID=cdab20e2:45150011:f10d83fe:f0299188 name=zoey:0
   devices=/dev/sdd1,/dev/sdc1,/dev/sdb

This line can also be found in the mdadm.conf that the Debian installer generated for me, back when I first installed the current OS (I didn't change this file manually, as everything was working fine for so long):

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md/0 metadata=1.2 UUID=cdab20e2:45150011:f10d83fe:f0299188 name=zoey:0

# This file was auto-generated on Fri, 26 Sep 2014 18:10:27 +0200
# by mkconf 3.2.5-5

SMART also says everything is fine with all disks, with the exception of /dev/sdc having seen unusually high airflow temperature sometime In_The_Past - but the temperature problems have been resolved, and all other values (most notable reallocated / pending sectors = 0) are within acceptable margins for all drives.

I tried recreating the RAID on device mapper overlays over the physical block devices. However, that setup would not recognize the partition table inside the RAID, making it impossible to mount the filesystem. You can find a typescript here.


And that's where I am at a loss. The entire RAID looks just fine to me, so why might the md driver refuse to assemble? And is there something that I could try which might improve my chances of success?

  • Two questions: first, are there any arrays listed in cat /proc/mdstat? Sometimes you'll get partially done incremental assembly, and will need to stop those before actually assembling the array; second, did you really create an array with some devices being partitions (sd[cd]1) and some entire disks (sdb)??? If not, it might just be misdetecting the device (but I think this is pretty unlikely, and you did indeed create that weird config) – derobert Nov 18 '16 at 15:56
  • @derobert First: /proc/mdstat knows nothing, added it to the question. Second: yes, this is intended; sdb really has no partition table. I planned on unifying this once the server had more free time. – Wüstengecko Nov 18 '16 at 16:08
  • That's really odd, but it seems someone has had the problem before (alas, without a resolution). You could try explicitly telling it the device to assemble, though I doubt it'd matter... Curious what you did with device-mapper, though, as (if done right) that should work, and is a reasonable way to test out what I fear might be the next step (a create with assume clean). – derobert Nov 18 '16 at 16:15
  • @derobert I added a script of the dm stuff I tried. That wouldn't yield any positive results directly, however comparing the --examine output before and after the recreation showed several differences that shouldn't be there, for example the newly created array having a bad block log on every device and differing data offsets ... I might just be missing some switches on that one. – Wüstengecko Nov 18 '16 at 21:36
5

The stupidly simple solution: add --update=devicesize when assembling.

I have no idea how, but apparently the partition size and the size that md remembered for this drive didn't match up anymore. Cheers to the guys over at the german ubuntuusers forum who painstakingly solved their case with manual hex editing before finally finding the correct switch to mdadm.

  • I'd suggest reporting a bug—mdadm really ought to notice that and tell you. – derobert Nov 21 '16 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.