Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have /dev/sdb1 and /dev/sdc2 that were previously setup into a RAID-1 with mdadm, but then I reinstalled and lost the old configuration. Out of idiocy, I ran

sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1

in an attempt to reconfigure the RAID. After I let the drives sync (oops?), now none of /dev/md0, /dev/sdb1, or /dev/sdc2 will mount. For /dev/md0, it complains about a bad magic number in the super block. For the /dev/sd{b,c}1, it complains about missing inodes.

In short, the question is this: Did I just bork all my data, or is it possible to recover the array still?

The following is the output of dumpe2fs for those partitions:

brent@codpiece:~$ sudo dumpe2fs /dev/md0 
dumpe2fs 1.42 (29-Nov-2011)
dumpe2fs: Bad magic number in super-block while trying to open /dev/md0
Couldn't find valid filesystem superblock.
brent@codpiece:~$ sudo dumpe2fs /dev/sdb1 
dumpe2fs 1.42 (29-Nov-2011)
Filesystem volume name:   <none>
Last mounted on:          /var/media
Filesystem UUID:          1462d79f-8a10-4590-8d63-3fcc105b601d
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              61054976
Block count:              244189984
Reserved block count:     12209499
Free blocks:              59069396
Free inodes:              60960671
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      965
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Wed Feb 10 21:04:42 2010
Last mount time:          Fri May 10 20:25:34 2013
Last write time:          Sun May 12 14:41:02 2013
Mount count:              189
Maximum mount count:      38
Last checked:             Wed Feb 10 21:04:42 2010
Check interval:           15552000 (6 months)
Next check after:         Mon Aug  9 22:04:42 2010
Lifetime writes:          250 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:           256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      7cd5ce46-b823-4453-aa66-00ddaff69952
Journal backup:           inode blocks
dumpe2fs: A block group is missing an inode table while reading journal inode

Edit:

It seems like @hauke-laging was correct that I created a 1.2 metadata version RAID-1 over what used to be a 1.0 metadata raid. I've reran mdadm --create for the right version, but now my filesystem is corrupt. Do I need to mess with the partition table, or can I simply run fsck /dev/md0?

The following is the new output of fsck and dumpe2fs:

brent@codpiece:~$ sudo fsck /dev/md0 
fsck from util-linux 2.20.1
e2fsck 1.42 (29-Nov-2011)
The filesystem size (according to the superblock) is 244189984 blocks
The physical size of the device is 244189952 blocks
Either the superblock or the partition table is likely to be corrupt!

brent@codpiece:~$ sudo dumpe2fs /dev/md0
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          1462d79f-8a10-4590-8d63-3fcc105b601d
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         not clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              61054976
Block count:              244189984
Reserved block count:     12209499
Free blocks:              240306893
Free inodes:              61054965
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      965
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
Flex block group size:    16
Filesystem created:       Wed Feb 10 21:04:42 2010
Last mount time:          n/a
Last write time:          Mon May 13 10:38:58 2013
Mount count:              0
Maximum mount count:      38
Last checked:             Wed Feb 10 21:04:42 2010
Check interval:           15552000 (6 months)
Next check after:         Mon Aug  9 22:04:42 2010
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:           256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      7cd5ce46-b823-4453-aa66-00ddaff69952
Journal backup:           inode blocks
Journal features:         journal_incompat_revoke
Journal size:             128M
Journal length:           32768
Journal sequence:         0x00215ad3
Journal start:            0


Group 0: (Blocks 0-32767) [ITABLE_ZEROED]
  Checksum 0x4453, unused inodes 0
  Primary superblock at 0, Group descriptors at 1-59
  Reserved GDT blocks at 60-1024
  Block bitmap at 1025 (+1025), Inode bitmap at 1041 (+1041)
  Inode table at 1057-1568 (+1057)
  23513 free blocks, 8181 free inodes, 2 directories
  Free blocks: 12576-12591, 12864-12879, <...>
  Free inodes: 
Group 1: (Blocks 32768-65535) [ITABLE_ZEROED]
  Checksum 0x348a, unused inodes 0
  Backup superblock at 32768, Group descriptors at 32769-32827
  Reserved GDT blocks at 32828-33792
  Block bitmap at 1026 (bg #0 + 1026), Inode bitmap at 1042 (bg #0 + 1042)
  Inode table at 1569-2080 (bg #0 + 1569)
  31743 free blocks, 8192 free inodes, 0 directories
  Free blocks: 43232-43239, 43264-43271, <...>
  Free inodes: 
Group 2: (Blocks 65536-98303) [ITABLE_ZEROED]
  Checksum 0x2056, unused inodes 0
  Block bitmap at 1027 (bg #0 + 1027), Inode bitmap at 1043 (bg #0 + 1043)
  Inode table at 2081-2592 (bg #0 + 2081)
  32768 free blocks, 8192 free inodes, 0 directories
  Free blocks: 66417-66432, 66445-66456, 66891, <...>
  Free inodes: 23921-24576
Group 3: (Blocks 98304-131071) [ITABLE_ZEROED]
  Checksum 0x4254, unused inodes 0
  Backup superblock at 98304, Group descriptors at 98305-98363
  Reserved GDT blocks at 98364-99328
  Block bitmap at 1028 (bg #0 + 1028), Inode bitmap at 1044 (bg #0 + 1044)
  Inode table at 2593-3104 (bg #0 + 2593)
  31743 free blocks, 8192 free inodes, 0 directories
  Free blocks: 99334-99339, 99438-99443, 99456-99459, <...>
  Free inodes: 24585-32768
Group 4: (Blocks 131072-163839) [ITABLE_ZEROED]
  Checksum 0x6a00, unused inodes 0
  Block bitmap at 1029 (bg #0 + 1029), Inode bitmap at 1045 (bg #0 + 1045)
  Inode table at 3105-3616 (bg #0 + 3105)
  32768 free blocks, 8192 free inodes, 0 directories
  Free blocks: 131074-131075, 131124-131129, <...>
  Free inodes: 32769-40960
Group 5: (Blocks 163840-196607) [ITABLE_ZEROED]
  Checksum 0x37e0, unused inodes 0
  Backup superblock at 163840, Group descriptors at 163841-163899
  Reserved GDT blocks at 163900-164864
  Block bitmap at 1030 (bg #0 + 1030), Inode bitmap at 1046 (bg #0 + 1046)
  Inode table at 3617-4128 (bg #0 + 3617)
  31743 free blocks, 8192 free inodes, 0 directories
  Free blocks: 164968-164970, 164979, <...>
  Free inodes: 40961-49152
Group 6: (Blocks 196608-229375) [ITABLE_ZEROED]
  <...>
share|improve this question
    
Can you mount either /dev/sdb1 or /dev/sdc1? I'd mount them readonly! My md0 raid doesn't show anything for a dumpe2fs either. Also did you setup a LVM on top of the raid or is it just a ext3/4 sitting on top? –  slm May 13 '13 at 4:23
    
@slm If your md0 contains ext3/ext4 but dumpe2fs -h /dev/md0 doesn't show you anything that's something to be really worried about. I don't see how such a volume could be mounted. –  Hauke Laging May 13 '13 at 15:03
    
My md0 doesn't contain ext3/ext4 directly. I always create my raids as follows: make 1 large partition on disks (/dev/sd{b,c}1), add these partitions to a raid1 device (/dev/md0), add (/dev/md0) as a physical volume to a LVM volume group (lvm-raid1), and create a logical volume within the group (/dev/lvm-raid1/lvm0). OP didn't say how he created the file-system on the raid. –  slm May 13 '13 at 15:33
    
@slm OK but you have to admit that without this information the remark "My md0 raid doesn't show anything for a dumpe2fs either" is quite a bit away from being useful. –  Hauke Laging May 13 '13 at 16:14
    
The size difference is small and probably has not destroyed data but of course a file system hates such situations. Maybe you did not have 1.0 before but 0.9. Or there are other reasons why the MD metadata needs more space now. Try with 0.9. If that does not work you should ask a new question how one can influence the size of this metadata. dmsetup tricks are possible, too: Make the block device a bit larger, let fsck do its job and reduce the FS afterwards so that it fits in the MD device. –  Hauke Laging May 13 '13 at 16:26
show 1 more comment

1 Answer

Have a look at this question. I assume that is familiar to your problem.

Recreating and even syncing a RAID-1 should not destroy data. Obviously the MD device starts at another offset now. Thus where mount looks for a superblock there is data. This can have happened in at least two ways:

  1. You (or rather: the default setting) have created the new array with a different superblock format (see --metadata in man mdadm). Thus the superblock is in another position (or has a different size) now. Do you happen to know what the old metadata format was?
  2. The offset has changed even with the same format due to a different default offset. See mdadm --examine /dev/sdb1 (add the output to your question).

You should look for a superblock in the first area of the disks (/dev/sdb1). Maybe this can be done with parted or similar tools. You may have to delete the respective partitions for that though (no problem as you can easily backup and restore the partition table).

Or you create loop devices / DM devices with increasing offsets (not necissarily over the whole disk, a few MiB are enough) and try dumpe2fs -h on them. If you want to do this but don't know how then I can provide some shell code for that.

The worst case would be that the new MD superblock has overwritten the file system superblock. In that case you may search for superblock copies (see the output of mke2fs). A mke2fs run on a dummy device of the same size may tell you the positions of the superblock copies.

Edit 1:

Now I have read and understood your dumpe2fs output. Your old RAID-1 had its superblock at the end (0.9 or 1.0). Now you probably have 1.2 so that a part of your file system has been overwritten. I cannot assess how big the damage may be. This is a case for e2fsck. But first you should reset the RAID to its old type. Would help to know the old version.

You can reduce the risk by putting DM devices over the complete /dev/sdb1 and /dev/sdc1, create snapshots for them (with dmsetup directly and create the new array over the snapshots. That way the relevant parts of your disks are not written. From the dumpe2fs output we know that the MD device must be 1000202174464 bytes in size. This should be checked at once after a test creation

share|improve this answer
    
Thanks for the quick help! I'm halfway through solving my problem, but I don't know partition tables well, so I'm a bit scared to progress any further at the moment. I've reran mdadm --create --metadata=1.0, and dumpe2fs is now able to read /dev/md0. The question is this: Given that fsck complains that either the superblock or partition table is wrong (see the above edit), is it safe to run fsck, or do I need to recreate the partition table? –  bestephe May 13 '13 at 16:04
    
It is hard to tell when fsck is "safe". What does "safe" mean? It is the job of fsck to get your file system usable again. Rescuing data is not its job. It is possible that an fsck run makes the situation worse for a professional data rescuer. If you are not going to spend several hundred USD for getting your data back then just run fsck. As I said you may do that on a snapshot instead for being safe in a broad sense. Forget the partition table. That is just a general hint by dumpe2fs. In this case we know better. –  Hauke Laging May 13 '13 at 16:20
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.