Eureka - Introduction
So I found out how to get access to my data again. Unfortunately I wasn't able to recreate the array using mdadm. The problem is that with IMSM I have to create a container first. But the container does not accept missing devices. I would need all my original 6 hard drives, but I only have 5 at the moment. I also cannot use any virtual hard drives as they need to be connected to the RAID controller. Furthermore I'm not sure if or how mdadm would start to synchronize the drives as soon as I create a volume. However I found a way that involves dmsetup. I'm able to access all my files again.
While performing multiple backups of the drives to work with them I also realized that the one drive that is not part of the array anymore sometimes fails with IO errors. I was still able to make backups since these errors occurred only about every third invocation of dd. I presume that as soon as one of the IO errors occurred the drive probably got kicked out of the array by IMSM and all of its metadata got deleted.
I also realized that this drive was the first one in the array. Since I have a GPT table and since the data of the array starts at the first sector it is also logical that it start with an MBR. So the drive's supersector wasn't overwritten with an MBR. It was always there.
Reading the data
I'm trying to give a step by step solution here that explains all the commands used in the process. Hopefully this will help someone out there.
(Optional) Make a backup of all the drives
This is not strictly necessary. Especially since we're using read-only loop devices later on. However I'm particularly paranoid after a major failure of my storage solution. So I try to avoid using the actual data as much as possible. Beyond that using the backup files shows that this method does not require the original hard drives or BIOS at all. All you need is a dd image. If you are skipping this section make sure to really create the loop devices in the next section as read-only or you may risk that your data becomes even more degraded and may be lost forever.
Nevertheless here is the command to backup your hard drive. You may already be familiar with dd. But in case you aren't run this command for each hard drive that is part of your array:
# dd if=/dev/sdX of=/path/to/backups/sdX.img status=progress
# dd if=/dev/sdY of=/path/to/backups/sdY.img status=progress
# [...]
The input file if=/dev/sdX is your hard drive. Replace sdX with sda, sdb, etc. The output file of=/path/to/backups/sdX.img points to the image that shall be written. Again replace sdX appropriately. status=progress just tells the GNU version of dd to print the current progress to stderr.
Create loop devices
Next we are going to create loop devices. In case we are using backup images it ensures that they are recognized as block file. Although this may not be necessary. But in any case it ensures that the image will only be read since we are using the read-only flag -r
# losetup --show -rf /path/to/backups/sdX.img
# losetup --show -rf /path/to/backups/sdY.img
[...]
-r: only read from file, don't write to it-f: use the next available number for the loop device so that we don't have to guess it ourselves.--show: print whichever name -f actually chose. This is usually quite useful. Although we are printing these values in the next step anyway.
To get a nice overview of the loop devices we just created we can use the following command:
# losetup -a
/dev/loop1: [2129]:251265027 (/path/to/backups/sdX.img)
/dev/loop2: [2129]:251265027 (/path/to/backups/sdY.img)
[...]
Try to remember which loop device belongs to which image.
Gathering some metadata
Next we need to find out some information about the RAID. Specifically we need to find out at which sector the RAID starts (especially in the case of a matrix RAID), how many sectors it spans, what its chunk size and layout is and in what order the drives were added to the array.
If there is at least one drive that is still part of the array and has metadata attached to it you can use it to retrieve most of the needed information. Run the following command on a drive sdX that is still part of the array:
# mdadm --examine /dev/sdX
/dev/sdX:
Magic : Intel Raid ISM Cfg Sig.
Version : 1.3.00
Orig Family : aa0b2c12
Family : 48d867fb
Generation : 0018f99c
Attributes : All supported
UUID : 0312fa14:fa8db3c2:2a76dc3f:299ed5b4
Checksum : 084869b8 correct
MPB Sectors : 6
Disks : 6
RAID Devices : 1
Disk02 Serial : S21PNSBG710576N
State : active
Id : 00000000
Usable Size : 488391936 (232.88 GiB 250.06 GB)
Bad Block Management Log:
Log Size : 2040
Signature : abadb10c
Entry Count : 254
[NameOfYourArray]:
UUID : 24b1e785:14f37ee5:41f6a4ab:d8b89e11
RAID Level : 5
Members : 6
Slots : [__UUUU]
Failed disk : 1
This Slot : 2
Sector Size : 512
Array Size : 2441959424 (1164.42 GiB 1250.28 GB)
Per Dev Size : 488392200 (232.88 GiB 250.06 GB)
Sector Offset : 0
Num Stripes : 7631124
Chunk Size : 32 KiB
Reserved : 0
Migrate State : idle
Map State : failed
Dirty State : clean
RWH Policy : off
The output goes on, but you can ignore the rest. The output shown above yields the following valuable information:
Sector Offset : 0 # Where the data starts
# (right at the first sector in my case)
Array Size : 2441959424 # Size of the volume (data) inside the array
Chunk Size : 32 KiB # Size of a single chunk
You can even determine where in your array that particular drive is.
This Slot : 2
This means that this one is the third drive in the array. (The slot number starts at zero.) Alternatively Disk## Serial : [...] also hints at the slot number:
Disk02 Serial : S21PNSBG710576N
Run this command for all drives. For those that still yield valid results note the slot number.
There is another trick that you can use to determine the first drive in the array. Since the RAID is written in chunks and not in bytes the first 32 kiB reside on the first drive. The second 32 kiB on the second drive and so on. This means that the first drive should have enough sectors containing the start of your partition table. Which means that there should be an MBR at the start (even when you are using GPT since it starts with a protective MBR). mdadm --examine already tells you it found an MBR when there is no metadata. But you can also use fdisk -l.
In my case I was able to find out the slot numbers of four drives through their metadata. I was lucky that the fifth drive contained an MBR so I automatically knew it was the first. 5 of 6 drives is enough to start the array. If you don't know the exact slot numbers of enough drives you can try and use different permutations until this method succeeds.
Which means that the correct order of my drives and therefore of my loop devices is:
| Slot |
Drive |
Loop device |
| MBR (0) |
/dev/sdb |
/dev/loop2 |
| 1 |
missing |
- |
| 2 |
/dev/sda |
/dev/loop1 |
| 3 |
/dev/sdc |
/dev/loop3 |
| 4 |
/dev/sde |
/dev/loop5 |
| 5 |
/dev/sdd |
/dev/loop4 |
The final thing to figure out is the layout. Unfortunately mdadm gives us no information about that. However when we have a look at Intel's RAID definitions it looks like the layout for RAID 5 is always left asymmetric. I'm not sure if IMSM arrays can even be configured with a different layout, but it seems unlikely to me. If all this doesn't work for you then you might try different layouts. Have a look in the source to read more about the other layouts.
Below is an overview over all the RAID levels IMSM supports. The dmsetup keyword is used in the next chapter.
| RAID level |
Layout |
dmsetup syntax |
| 0 |
N/A |
raid0 |
| 1 |
N/A |
raid1 |
| 5 |
left asymmetric |
raid5_la |
| 10 |
default (no 1E or copying) |
raid10 |
If you are not able to gather any metadata from any drive then you have to guess values and/or try different combinations. As a help these are the different modes that IMSM supports:
| Info |
Possible values |
| RAID Levels |
0, 1, 5, 10 |
| Chunk Sizes |
4 kiB, 8 kiB, 16 kiB, 32 kiB, 64 kiB, 128 kiB |
For the start sector and the size it is best to assume zero and the size of the smallest drive in the array times the number of non-parity drives if you are unsure. You can get the size in sectors of a drive by issuing the following command:
blockdev --getsize /dev/sdX
If your data doesn't actually start at zero you can still get the correct offset later on by searching for a partition header or maybe even by searching for filesystems.
Assembling the array using dmsetup
Unfortunately there is no way to provide the metadata manually when you are using mdadm. The only exception is for the RAID levels 0 and 1 where you can use --build:
mdadm --build /dev/md0 --raid-devices=2 --level=0 --chunk=32 /dev/loop0 /dev/loop1
Since we are out of luck here we need to use a different tool. Therefore we are going to use dmsetup instead. dmsetup is a command that creates virtual hard drives that are mapped to real drives or other sources. These mappings consist of several sections and each section can map to a different drive. In our case we only need one section and we are mapping to a RAID whose metadata we will provide manually.
But first we need to talk about numbers. As we determined earlier the chunk size in my case was 32 kiB. However dmsetup requires sectors. In almost all cases one sector equals 512 bytes. If you want to be on the safe side you can check the sector size with blockdev --getss /dev/sdX. In my case this means 32 kiB / (512 bytes/sector) = 64 sectors. We already know the size of the data in the array in sectors (i.e. 2441959424). But there is a problem. I have 6 devices. With one parity chunk per stripe the number of chunks must be divisible by 5. But the number of sectors is not divisible by 5. In my case it is at least divisible the number of sectors per chunk. But I'm not even sure if that is guaranteed. It appears that the data stops halfway through the last stripe. Unfortunately dmsetup won't tolerate this. That means that we need to round up to the nearest integer that is divisible by 5 drives and by 64 sectors (adjust these numbers according to your situation). In my case this is : 2441959680. This will mean that fdisk may complain about a wrong drive size and a missing backup table. But we can fix that by truncating the dd image.
Now create a file (e.g. table.txt) which will contain one line for one section.
<start> <size> raid <raid layout> 2 <chunk size> nosync <num devices>[ - /dev/loopN|-]*num_devices
First you have to give the start and the size in sectors. The next argument says that this is a RAID. For the RAID layout see the table in the previous section. The "2" in the next argument means two special parameters for the RAID. The first is the chunk size. The second will prevent any synchronization. After that you have to describe your drives by first giving the number of devices and then giving a pair of metadata and device path for each device. Since we don't want to provide any metadata we use a dash to indicate that. If the device is missing we write two dashes indicating that neither metadata nor the device is available. It is advisable to leave out at least one device if the RAID level allows it. If you already suspect that one drive may contain faulty data chose that one.
E.g. in my case the file looks like this. Note that the second device is missing.
0 2441959680 raid raid5_la 2 64 nosync 6 - /dev/loop2 - - - /dev/loop1 - /dev/loop3 - /dev/loop5 - /dev/loop4
Now run the following command to create a new block file that maps to our array:
# dmsetup create sdr /path/to/table.txt
This may spew out a bunch of IO errors. In which case the size in sectors probably wasn't divisble by the chunk size. You can remove the block file in order to redo the last step with the following command:
# dmsetup remove sdr
Now lets have a look at this newly created device file. If you run
# fdisk -l /dev/mapper/sdr
you should be able to see your partition table. Don't worry about the two errors that will show up if you have a GPT table. The size mismatch and missing backup table is due to the fact that we chose a size for our RAID that is too big.
Mine looks like this:
Device Start End Sectors Size Type
/dev/mapper/sdr-part1 2048 923647 921600 450M Windows recovery environment
/dev/mapper/sdr-part2 923648 1128447 204800 100M EFI System
/dev/mapper/sdr-part3 1128448 1161215 32768 16M Microsoft reserved
/dev/mapper/sdr-part4 1161216 679840003 678678788 323.6G Microsoft basic data
/dev/mapper/sdr-part5 679841792 680902655 1060864 518M Windows recovery environment
/dev/mapper/sdr-part6 680904704 2295472127 1614567424 769.9G Linux filesystem
/dev/mapper/sdr-part7 2295472128 2441957375 146485248 69.9G Linux swap
Using the start and sectors column in this table we can even mount some of these partitions. Please note that all numbers are in sectors and need to be converted to bytes by multiplying with 512.
# mount -o ro,noload,loop,offset=348623208448,sizelimit=826658521088 /dev/mapper/sdr /mnt
Which means that my Linux partition is now mounted on /mnt and I can browse all my files in ro (i.e. read-only) mode. The noload is needed to prevent ext4 from performing write operations.
And now at last we will perform a full backup using dd.
# dd if=/dev/mapper/sdr of=/path/to/backups/raid.img status=progress
Remember how we created a RAID that was slightly larger than it should be? We can use this opportunity to correct this error by truncating the image to its correct size. The number of sectors need to be converted to bytes: 2441959424*512 = 1250283225088.
# truncate -s 1250283225088 /path/to/backups/raid.img
Now fdisk -l does not complain about a size mismatch anymore.
