I've got a server with 3 sata hard drives. Each has 2 partitions: one small is part of /dev/md0, a raid1 array (/boot), rest is part of a raid5 array (/dev/md1), which is an lvm physical volume. Inside it are 3 (IIRC) logical volumes. One of these is a reiserfs 3.6 fs holding about 100gigs of data.

Yesterday this server crashed. At power up, SMART told me that one of the drives was dead. He was indeed making very bad noises. So I removed the failed drive, and tried to restart the system on the 2 remaining disks. Which failed.

With a live cd, I started it and tried to restart the array. Unfortunately, mdadm refused to do so, because it thought one of the 2 remaining disks was failed also.

So, following advice found at How to recover a crashed Linux md RAID5 array? that looked like it could apply to my situation, I did something that was probably just plain stupid: I ran

mdadm --create /dev/md1 --assume-clean -l5 -n3 -c64 /dev/sd[ab]2 missing

Now, I can actually start this array, but the lvm tools (vgscan, vgdisplay, pvck) cannot find anything related to lvm on the array, and I'm completely unable to get to my data. Did I just wipe all the lvm metadata?

My feeling is that actual data is still there, undamaged (apart from lvm metadata). Is there a chance to get the data back? How?


Following advice from psusi (below), I tried each of the following ways of re-creating the array:

mdadm --create /dev/md1 --assume-clean -l5 -n3 -c64 /dev/sda2 /dev/sdb2 missing
mdadm --create /dev/md1 --assume-clean -l5 -n3 -c64 /dev/sdb2 /dev/sda2 missing
mdadm --create /dev/md1 --assume-clean -l5 -n3 -c64 /dev/sda2 missing /dev/sdb2
mdadm --create /dev/md1 --assume-clean -l5 -n3 -c64 /dev/sdb2 missing /dev/sda2
mdadm --create /dev/md1 --assume-clean -l5 -n3 -c64 missing /dev/sda2 /dev/sdb2
mdadm --create /dev/md1 --assume-clean -l5 -n3 -c64 missing /dev/sdb2 /dev/sda2

mdadm --create /dev/md1 --assume-clean -l5 -n3 -c512 /dev/sda2 /dev/sdb2 missing
mdadm --create /dev/md1 --assume-clean -l5 -n3 -c512 /dev/sdb2 /dev/sda2 missing
mdadm --create /dev/md1 --assume-clean -l5 -n3 -c512 /dev/sda2 missing /dev/sdb2
mdadm --create /dev/md1 --assume-clean -l5 -n3 -c512 /dev/sdb2 missing /dev/sda2
mdadm --create /dev/md1 --assume-clean -l5 -n3 -c512 missing /dev/sda2 /dev/sdb2
mdadm --create /dev/md1 --assume-clean -l5 -n3 -c512 missing /dev/sdb2 /dev/sda2

Which is basically all possible orders, both with -c64 and -c512. After each test, I ran a vgscan. None found anything. Maybe I should not use vgscan but some other tool?


I gave a try at reconnecting the failed hard drive. And miracle, it seems to somewhat work. At least, enough to examine it:

root@debian:~# mdadm --examine /dev/sda2
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 1f5462ab:6945560d:019b01a5:914dd464
  Creation Time : Fri Oct 17 12:40:40 2008
     Raid Level : raid5
  Used Dev Size : 160015360 (152.60 GiB 163.86 GB)
     Array Size : 320030720 (305.21 GiB 327.71 GB)
   Raid Devices : 3
  Total Devices : 3
Preferred Minor : 1

    Update Time : Tue Apr 12 08:15:03 2011
          State : active
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 64d514fb - correct
         Events : 137

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0       8        2        0      active sync   /dev/sda2
   0     0       8        2        0      active sync   /dev/sda2
   1     1       8       18        1      active sync   /dev/sdb2
   2     2       8       34        2      active sync   /dev/sdc2

So, is there a way to copy this superblock to the other 2 devices, so that I can start the array "properly"?

  • Most of the data is still there, but some vital metadata is missing. First, make a backup as there is a high risk of overwriting more bits and pieces in random places. Then try a Testdisk livecd (I don't know if it actually does understand md-raid, but it's the first thing I'd try). Mar 22 '12 at 20:07

I have a similar setup and I can recommend to have a complete Linux on the small partition of each drive and not mirror those small partitions, but have them separately completely boot-able.

You can sync the setup excluding a few crucial files (/etc/fstab, grub configuration). This takes more space than just /boot but saves a lot of time when troubles hit.


You probably did not assemble the drives in the same order, or using the same chunk size as before. You need to figure out what the order was before and use the same order when you recreate the array. In other words, it may not have been the third disk that died, but the first or second, our you may have mixed up sda and sdb.

  • Great suggestion! According to the man page of mdadm: "The default when creating an array is 512KB. To ensure compatibility with earlier versions, the default when Building and array with no persistent metadata is 64KB". As I created my original array with default options only, I guess that I should try with -c512. Can I damage something giving it a try? Regarding drives order, as I have only 2 drives left, what else can I do than try and swap them ([ba]2 instead of [ab]2)? Is there a way to tell mdadm that they are to be second and third in the array (and the missing one is to be first)?
    – Bill
    Mar 22 '12 at 17:51
  • @Bill, yes, just change the order you specify them on the command line.
    – psusi
    Mar 23 '12 at 2:18
  • I tried that, and posted results in an OP update above.
    – Bill
    Mar 23 '12 at 10:56
  • @Bill, at this point it is probably time to restore from backup.
    – psusi
    Mar 23 '12 at 14:23
  • 1
    @Bill, it looks like the array was using metadata format 0.90, and that is no longer the default, so you might try what you did before with the two good disks and explicitly specify that metadata format with the 64k chunk size.
    – psusi
    Mar 23 '12 at 15:52

As @psusi hinted metadata format is the kye, as it seems — now "1.2" is the default, not "0.9". It's pity to say, but this could lead to data loss, since 1.2 uses 4 KiB offset:

1, 1.0, 1.1, 1.2 default Use the new version-1 format superblock. This has fewer restrictions. It can easily be moved between hosts with different endian-ness, and a recovery operation can be checkpointed and restarted. The different sub-versions store the superblock at different locations on the device, either at the end (for 1.0), at the start (for 1.1) or 4K from the start (for 1.2).

An advice (alas, late one): never rush into re-creating an array w/o giving it a try with -B — build:

   -B, --build
          Build a legacy array without superblocks

UPD.: turned out -B refuses to build RAID-5… :-/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.