0

Description

I've got an IMSM RAID 5 array that contains 6 SSD drives. One of the drives failed a few months ago and I wasn't able to replace it yet. (Yes, I know I'm lazy sometimes. Please don't judge me.) But I already removed it from the RAID.

Yesterday however another drive seems to have failed. The array doesn't assemble. Since even the BIOS fails to build the RAID I cannot boot anything. Upon closer inspection it looks like the drive is fine though. I can access it and make backups using dd. But it seems to have an MBR record at the beginning now. Maybe some process overwrote the RAID superblock with an MBR table? If that is the case the data should still be there. I just need to be able to tell mdadm the correct metadata. When I think about it the same thing might have happened to the first drive that supposedly "failed". Since it was still readable as well. But I didn't bother to investigate back then.

Nonetheless I'm now trying to find a way to reassemble the array to access its data (if possible). I know the chunk size, the exact order of the drives and the RAID level. Shouldn't that be enough information?

Some info

The first thing I did was to create images of the remaining 5 drives using dd (named sd[a-e].backup). I also examined all drives using --examine and saved the output. You can read the output in this gist. As you can see in there mdadm reads the MBR header of sdb and proceeds to the next drive without detecting any RAID information. For all the other drives mdadm prints the correct metadata though. While we're at it, here's the output of cat /proc/mdstat

Personalities:
md127 : inactive sda[3](S) sdd[2](S) sde[1](S) sdc[0](S)
      13049 blocks super external:imsm

unused devices: <none>

What I tried

  • Obviously I tried to "turn it off and on again":
# mdadm --stop /dev/md127
mdadm: stopped /dev/md127
# mdadm --assemble /dev/md0 /dev/sdb missing /dev/sda /dev/sdc /dev/sde /dev/sdd
mdadm: Cannot assemble mbr metadata on /dev/sdb
mdadm: /dev/sdb has no superblock - assembly aborted
# mdadm --assemble --scan

After the last call to mdadm /proc/mdstat again looks identical to the output above.

I then created read-only loop devices:

# losetup --show -rf /mnt/backup/sdX.backup
[...]
# losetup -a
/dev/loop1: [...] (/mnt/backup/sda.backup)
/dev/loop2: [...] (/mnt/backup/sdb.backup)
/dev/loop3: [...] (/mnt/backup/sdc.backup)
/dev/loop4: [...] (/mnt/backup/sdd.backup)
/dev/loop5: [...] (/mnt/backup/sde.backup)
  • Next I tried to use --build since it doesn't require any superblock information and all the metadata can be supplied manually:
# mdadm --build /dev/md0 --raid-devices=6 --level=5 --chunk=32 /dev/loop2 missing /dev/loop1 /dev/loop3 /dev/loop5 /dev/loop4
mdadm: Raid level 5 not permitted with --build

But apparently I'm not allowed to use --build in the context of level 5 RAIDs.

  • Next thing I tried was using --assemble but without using the OROM information about the RAID.
# IMSM_NO_PLATFORM=1 mdadm --assemble /dev/md0 /dev/loop2 missing /dev/loop1 /dev/loop3 /dev/loop5 /dev/loop4
mdadm: Cannot assemble mbr metadata on /dev/loop2
mdadm: /dev/loop2 has no superblock - assembly aborted

That would have been too easy I guess. Can I somehow tell mdadm to just assume that loop2 is the first device in that array and use the meta data from the other drives?

  • The final thing that I would have tried is to remount the loop devices as read-write and recreate the array. However all examples I've found (like this one or this one) assume that the array was created with mdadm. But it wasn't. It was initially created by a utility in the BIOS and has the IMSM or Intel Rapid Storage format. I guess I have to have more detailed knowledge about it like layout or data offset. I'm not sure what the defaults are for IMSM or where I could find them. But more importantly I'm worrying that mdadm's metadata format uses more space and a bigger superblock than IMSM and overwrites data when it saves the metadata. Maybe it is also possible to recreate the array using IMSM? Or maybe it is possible to store the metadata externally. Long story short I have no clue how to manually recreate an IMSM array with mdadm.

Other Questions on StackExchange

  • I'm aware of this question. But I'm unsure if this can be applied to my situation since I'm using IMSM which has different superblocks (if any at all).
  • I've also read this question. However it deals with RAID 0 and the answer suggests to use --build which doesn't work with RAID 5.
  • I'm also aware of this one. But --force is not applicable in my situation since the drive in question is not just marked as failed or out of sync. And again I'm unsure how I should recreate the array specifically with IMSM.
0

Eureka - Introduction

So I found out how to get access to my data again. Unfortunately I wasn't able to recreate the array using mdadm. The problem is that with IMSM I have to create a container first. But the container does not accept missing devices. I would need all my original 6 hard drives, but I only have 5 at the moment. I also cannot use any virtual hard drives as they need to be connected to the RAID controller. Furthermore I'm not sure if or how mdadm would start to synchronize the drives as soon as I create a volume. However I found a way that involves dmsetup. I'm able to access all my files again.

While performing multiple backups of the drives to work with them I also realized that the one drive that is not part of the array anymore sometimes fails with IO errors. I was still able to make backups since these errors occurred only about every third invocation of dd. I presume that as soon as one of the IO errors occurred the drive probably got kicked out of the array by IMSM and all of its metadata got deleted.

I also realized that this drive was the first one in the array. Since I have a GPT table and since the data of the array starts at the first sector it is also logical that it start with an MBR. So the drive's supersector wasn't overwritten with an MBR. It was always there.

Reading the data

I'm trying to give a step by step solution here that explains all the commands used in the process. Hopefully this will help someone out there.

(Optional) Make a backup of all the drives

This is not strictly necessary. Especially since we're using read-only loop devices later on. However I'm particularly paranoid after a major failure of my storage solution. So I try to avoid using the actual data as much as possible. Beyond that using the backup files shows that this method does not require the original hard drives or BIOS at all. All you need is a dd image. If you are skipping this section make sure to really create the loop devices in the next section as read-only or you may risk that your data becomes even more degraded and may be lost forever.

Nevertheless here is the command to backup your hard drive. You may already be familiar with dd. But in case you aren't run this command for each hard drive that is part of your array:

# dd if=/dev/sdX of=/path/to/backups/sdX.img status=progress
# dd if=/dev/sdY of=/path/to/backups/sdY.img status=progress
# [...]

The input file if=/dev/sdX is your hard drive. Replace sdX with sda, sdb, etc. The output file of=/path/to/backups/sdX.img points to the image that shall be written. Again replace sdX appropriately. status=progress just tells the GNU version of dd to print the current progress to stderr.

Create loop devices

Next we are going to create loop devices. In case we are using backup images it ensures that they are recognized as block file. Although this may not be necessary. But in any case it ensures that the image will only be read since we are using the read-only flag -r

# losetup --show -rf /path/to/backups/sdX.img
# losetup --show -rf /path/to/backups/sdY.img
[...]
  • -r: only read from file, don't write to it
  • -f: use the next available number for the loop device so that we don't have to guess it ourselves.
  • --show: print whichever name -f actually chose. This is usually quite useful. Although we are printing these values in the next step anyway.

To get a nice overview of the loop devices we just created we can use the following command:

# losetup -a
/dev/loop1: [2129]:251265027 (/path/to/backups/sdX.img)
/dev/loop2: [2129]:251265027 (/path/to/backups/sdY.img)
[...]

Try to remember which loop device belongs to which image.

Gathering some metadata

Next we need to find out some information about the RAID. Specifically we need to find out at which sector the RAID starts (especially in the case of a matrix RAID), how many sectors it spans, what its chunk size and layout is and in what order the drives were added to the array.

If there is at least one drive that is still part of the array and has metadata attached to it you can use it to retrieve most of the needed information. Run the following command on a drive sdX that is still part of the array:

# mdadm --examine /dev/sdX
/dev/sdX:
          Magic : Intel Raid ISM Cfg Sig.
        Version : 1.3.00
    Orig Family : aa0b2c12
         Family : 48d867fb
     Generation : 0018f99c
     Attributes : All supported
           UUID : 0312fa14:fa8db3c2:2a76dc3f:299ed5b4
       Checksum : 084869b8 correct
    MPB Sectors : 6
          Disks : 6
   RAID Devices : 1

  Disk02 Serial : S21PNSBG710576N
          State : active
             Id : 00000000
    Usable Size : 488391936 (232.88 GiB 250.06 GB)

Bad Block Management Log:
       Log Size : 2040
      Signature : abadb10c
    Entry Count : 254

[NameOfYourArray]:
           UUID : 24b1e785:14f37ee5:41f6a4ab:d8b89e11
     RAID Level : 5
        Members : 6
          Slots : [__UUUU]
    Failed disk : 1
      This Slot : 2
    Sector Size : 512
     Array Size : 2441959424 (1164.42 GiB 1250.28 GB)
   Per Dev Size : 488392200 (232.88 GiB 250.06 GB)
  Sector Offset : 0
    Num Stripes : 7631124
     Chunk Size : 32 KiB
       Reserved : 0
  Migrate State : idle
      Map State : failed
    Dirty State : clean
     RWH Policy : off

The output goes on, but you can ignore the rest. The output shown above yields the following valuable information:

Sector Offset : 0       # Where the data starts
                        # (right at the first sector in my case)
Array Size : 2441959424 # Size of the volume (data) inside the array
Chunk Size : 32 KiB     # Size of a single chunk

You can even determine where in your array that particular drive is.

This Slot : 2

This means that this one is the third drive in the array. (The slot number starts at zero.) Alternatively Disk## Serial : [...] also hints at the slot number:

Disk02 Serial : S21PNSBG710576N

Run this command for all drives. For those that still yield valid results note the slot number.

There is another trick that you can use to determine the first drive in the array. Since the RAID is written in chunks and not in bytes the first 32 kiB reside on the first drive. The second 32 kiB on the second drive and so on. This means that the first drive should have enough sectors containing the start of your partition table. Which means that there should be an MBR at the start (even when you are using GPT since it starts with a protective MBR). mdadm --examine already tells you it found an MBR when there is no metadata. But you can also use fdisk -l.

In my case I was able to find out the slot numbers of four drives through their metadata. I was lucky that the fifth drive contained an MBR so I automatically knew it was the first. 5 of 6 drives is enough to start the array. If you don't know the exact slot numbers of enough drives you can try and use different permutations until this method succeeds.

Which means that the correct order of my drives and therefore of my loop devices is:

Slot Drive Loop device
MBR (0) /dev/sdb /dev/loop2
1 missing -
2 /dev/sda /dev/loop1
3 /dev/sdc /dev/loop3
4 /dev/sde /dev/loop5
5 /dev/sdd /dev/loop4

The final thing to figure out is the layout. Unfortunately mdadm gives us no information about that. However when we have a look at Intel's RAID definitions it looks like the layout for RAID 5 is always left asymmetric. I'm not sure if IMSM arrays can even be configured with a different layout, but it seems unlikely to me. If all this doesn't work for you then you might try different layouts. Have a look in the source to read more about the other layouts.

Below is an overview over all the RAID levels IMSM supports. The dmsetup keyword is used in the next chapter.

RAID level Layout dmsetup syntax
0 N/A raid0
1 N/A raid1
5 left asymmetric raid5_la
10 default (no 1E or copying) raid10

If you are not able to gather any metadata from any drive then you have to guess values and/or try different combinations. As a help these are the different modes that IMSM supports:

Info Possible values
RAID Levels 0, 1, 5, 10
Chunk Sizes 4 kiB, 8 kiB, 16 kiB, 32 kiB, 64 kiB, 128 kiB

For the start sector and the size it is best to assume zero and the size of the smallest drive in the array times the number of non-parity drives if you are unsure. You can get the size in sectors of a drive by issuing the following command:

blockdev --getsize /dev/sdX

If your data doesn't actually start at zero you can still get the correct offset later on by searching for a partition header or maybe even by searching for filesystems.

Assembling the array using dmsetup

Unfortunately there is no way to provide the metadata manually when you are using mdadm. The only exception is for the RAID levels 0 and 1 where you can use --build:

mdadm --build /dev/md0 --raid-devices=2 --level=0 --chunk=32 /dev/loop0 /dev/loop1

Since we are out of luck here we need to use a different tool. Therefore we are going to use dmsetup instead. dmsetup is a command that creates virtual hard drives that are mapped to real drives or other sources. These mappings consist of several sections and each section can map to a different drive. In our case we only need one section and we are mapping to a RAID whose metadata we will provide manually.

But first we need to talk about numbers. As we determined earlier the chunk size in my case was 32 kiB. However dmsetup requires sectors. In almost all cases one sector equals 512 bytes. If you want to be on the safe side you can check the sector size with blockdev --getss /dev/sdX. In my case this means 32 kiB / (512 bytes/sector) = 64 sectors. We already know the size of the data in the array in sectors (i.e. 2441959424). But there is a problem. I have 6 devices. With one parity chunk per stripe the number of chunks must be divisible by 5. But the number of sectors is not divisible by 5. In my case it is at least divisible the number of sectors per chunk. But I'm not even sure if that is guaranteed. It appears that the data stops halfway through the last stripe. Unfortunately dmsetup won't tolerate this. That means that we need to round up to the nearest integer that is divisible by 5 drives and by 64 sectors (adjust these numbers according to your situation). In my case this is : 2441959680. This will mean that fdisk may complain about a wrong drive size and a missing backup table. But we can fix that by truncating the dd image.

Now create a file (e.g. table.txt) which will contain one line for one section.

<start> <size> raid <raid layout> 2 <chunk size> nosync <num devices>[ - /dev/loopN|-]*num_devices

First you have to give the start and the size in sectors. The next argument says that this is a RAID. For the RAID layout see the table in the previous section. The "2" in the next argument means two special parameters for the RAID. The first is the chunk size. The second will prevent any synchronization. After that you have to describe your drives by first giving the number of devices and then giving a pair of metadata and device path for each device. Since we don't want to provide any metadata we use a dash to indicate that. If the device is missing we write two dashes indicating that neither metadata nor the device is available. It is advisable to leave out at least one device if the RAID level allows it. If you already suspect that one drive may contain faulty data chose that one.

E.g. in my case the file looks like this. Note that the second device is missing.

0 2441959680 raid raid5_la 2 64 nosync 6 - /dev/loop2 - - - /dev/loop1 - /dev/loop3 - /dev/loop5 - /dev/loop4

Now run the following command to create a new block file that maps to our array:

# dmsetup create sdr /path/to/table.txt

This may spew out a bunch of IO errors. In which case the size in sectors probably wasn't divisble by the chunk size. You can remove the block file in order to redo the last step with the following command:

# dmsetup remove sdr

Now lets have a look at this newly created device file. If you run

# fdisk -l /dev/mapper/sdr

you should be able to see your partition table. Don't worry about the two errors that will show up if you have a GPT table. The size mismatch and missing backup table is due to the fact that we chose a size for our RAID that is too big.

Mine looks like this:

Device                     Start        End    Sectors   Size Type
/dev/mapper/sdr-part1       2048     923647     921600   450M Windows recovery environment
/dev/mapper/sdr-part2     923648    1128447     204800   100M EFI System
/dev/mapper/sdr-part3    1128448    1161215      32768    16M Microsoft reserved
/dev/mapper/sdr-part4    1161216  679840003  678678788 323.6G Microsoft basic data
/dev/mapper/sdr-part5  679841792  680902655    1060864   518M Windows recovery environment
/dev/mapper/sdr-part6  680904704 2295472127 1614567424 769.9G Linux filesystem
/dev/mapper/sdr-part7 2295472128 2441957375  146485248  69.9G Linux swap

Using the start and sectors column in this table we can even mount some of these partitions. Please note that all numbers are in sectors and need to be converted to bytes by multiplying with 512.

# mount -o ro,noload,loop,offset=348623208448,sizelimit=826658521088 /dev/mapper/sdr /mnt

Which means that my Linux partition is now mounted on /mnt and I can browse all my files in ro (i.e. read-only) mode. The noload is needed to prevent ext4 from performing write operations.

And now at last we will perform a full backup using dd.

# dd if=/dev/mapper/sdr of=/path/to/backups/raid.img status=progress

Remember how we created a RAID that was slightly larger than it should be? We can use this opportunity to correct this error by truncating the image to its correct size. The number of sectors need to be converted to bytes: 2441959424*512 = 1250283225088.

# truncate -s 1250283225088 /path/to/backups/raid.img

Now fdisk -l does not complain about a size mismatch anymore.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.