Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 4 down vote favorite
3

I have a RAID 10 array managed with mdadm which has failed . The symptom was that X has crashed and I was unable to logon from the terminal. I have restarted the computer and /dev/md1 did not start.

I've gathered the following information - some incomplete as I've manually copied it to another computer:

$ cat /proc/mdstat

....

md1 :   inactive sdb3[1](S) sdd1[3](S) sdc2[2](S) sda1[0](S)
        1250273760 blocks super 1.0

Mounting has failed

$ mount /dev/md1    
EXT4-fs (md1): unable to read superblock

Scanning did find the metadata

$ mdadm --examine --scan
...
ARRAY /dev/md/1 metadata=1.0 UUID=2c... name=linux:1

Running has failed due to lack of operational mirrors

$ mdadm --run /dev/md1
md/raid0:md1: not enough operational mirrors.
md: pers->run() failed...
mdadm:failed to run array /dev/md1: Input/output error

After the run attempt only two disks were still in the array

$ cat /proc/mdstat
....
md1  : inactive sdd1[3] sdc2[2]
       62513638 blocks super 1.0

Trying to re-add one disk has failed

$ mdadm /dev/md1 --add /dev/sda1
mdadm: /dev/sda1 reports being an active member for /dev/md1, but --re-add fails
mdadm: not performing --add as that would convert /dev/sda1 in to a spare
mdadm: To make this a spare, use "mdadm --zero-superblock /dev/sda1" first

I've recorded some metadata in order to try recreating the array

$ mdadm --examine /dev/sdb3 /dev/sdd1 /dev/sdc2 /dev/sda1 | grep -E 'dev|Update|Role|State|Chunk Size'
/dev/sdb3:
  State: active
  Update time:  Mon May 14 21:51:41 2012
  Chunk Size: 256K
  Device Role: Active device 1
  Array State : AAAA
/dev/sdd1:
  State: active
  Update time:  Mon May 14 22:11:11 2012
  Chunk Size: 256K
  Device Role: Active device 3
  Array State : ..AA
/dev/sdc2:
  State: active
  Update time:  Mon May 14 22:11:11 2012
  Chunk Size: 256K
  Device Role: Active device 2
  Array State : ..AA
/dev/sda1:
  State: active
  Update time:  Mon May 14 21:51:41 2012
  Chunk Size: 256K
  Device Role: Active device 0
  Array State : AAAA

I've tried manually recreating the array based on the previously recorded information

$ mdadm --create --metadata=1.0 --assume-clean --level=10 --raid-devices=4 --chunk=256K /dev/md1 /dev/sda1 /dev/sdb3 /dev/sdc2 /dev/sdd1
mdadm: /dev/sda1 appears to contain an ext2fs file system
mdadm: /dev/sda1 appears to be part of a raid array
mdadm: /dev/sdb3 appears to contain an ext2fs file system
mdadm: /dev/sdb3 appears to be part of a raid array
mdadmin: /dev/sdc2 appears to be part of a raid aray
mdadmin: /dev/sdd1 appears to be part of a raid aray

Mounting fails again which probably means that the raid is not properly created

$ mount /dev/md1 /mnt

EXT4-fs (md1): ext4_check_descriptors: Checksum forgroup 0 failed (54076!=0)
EXT4-fs (md1): group descriptors corrupted!

Running a fsck yielded too many errors to take into account seriously so I did not let it run.

I've tried recreating the array with 'missing' devices and just a two disks, but no combination was successful. I'm not sure if I tried them all, but I did try many.

Some details:

  • mdadm is version 2.3.2, running on OpenSUSE 12.1, kernel 3.1.10, x86_64
  • all hard disks have completed smartctl -t short without any errors

How can I rebuild the RAID array without losing the information or just extract the information and copy it somewhere else? I'm also curious to know if there is anything that I did wrong above when trying to recreate the array.

share|improve this question
Are there any disk IO errors showing up in dmesg? – psusi May 15 '12 at 15:12
@psusi - I don't see any disk errors. However, I suspect that there is an issue with the partition tables. sda and sdb are shown as having intel partition tables, while sdc and sdd are shown as having EPT partition tables. I did not partition manually - I used YAST - and I see no reason to mix partition table types. – Robert Munteanu May 15 '12 at 15:46
You mean msdos and gpt partition tables? That shouldn't be a problem. – psusi May 15 '12 at 19:26
1  
[I realize this is old, but]... When your raid array has failed, and you are tempted to run mdadm --create, then STOP. It is almost never the right answer. The right answer depends on what exactly caused the failure, the value of the data, and the budget available for recovery. – derobert Dec 12 '12 at 18:22
1  
BTW: your --examine output shows that you suffered a double-disk failure, and then possibly another. Hard to tell for sure without the fields you omitted. Most likely mdadm --stop /dev/md1; mdadm --assemble --force /dev/md1 /dev/sda1 /dev/sdb3 /dev/sdc2 /dev/sdd1 would have worked, with less data corruption. Or maybe even just --run. – derobert Dec 12 '12 at 18:31
show 1 more comment

1 Answer

up vote 0 down vote accepted

With a little help, I managed to rebuild a degraded array in the following manner:

  1. Use the testdisk utility and notice that the /dev/sdc and /dev/sdd disks were marked with a GPT partition table
  2. Fix the partition table on /dev/sdd to have the partition type listed as fd instead of 83
  3. Create a RAID array with only /dev/sda1 and /dev/sdd1
  4. Run e2fsck -y /dev/md1 ( expect lots of fixes )

A useful hint when rebuilding raid arrays is to peek at the first 100 MB of the raw device to see if it looks like a valid ext2 which should be passed to e2fsck:

dd if=/dev/md1 of=/tmp/md1.img bs=1k count=1024
strings md1.img | lesss
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.