5

I have tried to remove 1 HDD from a RAID-5 but something went wrong, but I still hope I can recover my data (in fact, I have all the backups so it is just a question on mdadm possibilities)

I had a 4 x 1 Tb RAID-5, and one of the disks started to show lots of Reallocated_Sector_Ct, so I decided to remove it.

What I did:

  1. mdadm --manage /dev/md0 --fail /dev/sdc
  2. mdadm --manage /dev/md0 --remove /dev/sdc

  3. tried to run:

    root@darkstar:/home/anton# mdadm --grow /dev/md0 --raid-devices=3
    mdadm: this change will reduce the size of the array.
    use --grow --array-size first to truncate array.
    e.g. mdadm --grow /dev/md0 --array-size 1953262592
    
  4. mdadm --grow /dev/md0 --array-size 1953262592

and finally:

  1. mdadm --grow /dev/md0 --raid-devices=3 --backup-file=/root/grow_md1.bak

Now that the reshape and recovery is done, I cannot access my /dev/md0 (it does not mount), resize2fs /dev/md0 tells to run e2fsck first, and e2fsck tells:

The filesystem size (according to the superblock) is 732473472 blocks
The physical size of the device is 488315648 blocks
Either the superblock or the partition table is likely to be corrupt!

on the other hand, mdadm -D /dev/md0 tells:

Array Size : 1953262592 (1862.78 GiB 2000.14 GB)
Used Dev Size : 976631296 (931.39 GiB 1000.07 GB)

That leaves some hope that not all my data is lost. Does anybody have some ideas on what I should do to have a valid RAID-5 array of 3 x 1 Tb disks?

8

All you should have done was your step one

mdadm --manage /dev/md0 --fail /dev/sdc

At this point your RAID 5 array is running in degraded mode and you can replace the disk with a new one.

Unfortunately it looks like you have truncated your array's effective size from 2TB to 1TB, destroying the second half of your filesystem. Fortunately you say you have backups.

I'm slightly puzzled. With four disks in a RAID5 configuration you should have had 3TB available to you. But without seeing the results of mdadm --examine I'm not sure there's much else I can offer you.

| improve this answer | |
  • i didn't want to replace the disk, i wanted to remove it. – Anton S. Sep 8 '17 at 14:27
  • root@darkstar:/home/anton# mdadm --examine /dev/md0 mdadm: No md superblock detected on /dev/md0. – Anton S. Sep 8 '17 at 14:28
  • @AntonS. without the mdadm details for the remaining physical devices I can't make any other suggestions. – roaima Sep 8 '17 at 14:29
  • alas, this site does not allow long comments. otherwise i'd copy everything that mdadm --examine returns. – Anton S. Sep 8 '17 at 14:34
  • 1
    @AntonS. comments are for commenting. Updates like this I'm asking you for, go into your question for everyone to find easily. (Don't forget the formatting.) Then I can delete my comments because they're no longer required. – roaima Sep 8 '17 at 14:38
7

You got the order backwards.

To shrink, you first shrink the filesystem (resize2fs) then second shrink the block device (mdadm). The order you did is correct for enlarging a filesystem, but backwards for shrinking one.

You've destroyed your data. To recover from this, you first confirm your backups are good. Then mkfs the array and restore from backup. If your backups aren't good, you can probably recover files that happened to be on the 1st 2TB of the filesystem. (See below)

PS: The normal way an array is managed is if a disk fails, you replace that disk with one of the same capacity or larger. mdadm --grow is not a normal part of dealing with disk failures.

Recovery

What used to be the 3rd terabyte of your filesystem has been overwritten; essentially that space is now used for the parity. (The actual sectors contain a mix of parity and data which was moved from other disks, where those sectors now contain parity.) That part of the data is gone for good; absent (possibly theoretical) high-cost approaches capable of reading the previous contents of sectors it's not recoverable.

In addition, ext4 does not keep all the metadata at the start of the filesystem; it's spread out throughout the filesystem. So you've lost a bunch of metadata as well. Importantly, if any part of the file data or metadata was in that lost third, the file will be inaccessible. Limited recovery of snippets may be possible from the 4th disk (which was probably untouched by the grow, since it was failed at the time.)

The first, and most important step, is to purchase a 4TB disk and use that to make a complete copy (image) of the filesystem. Then, set the 4 original disks aside. If there is any question of the reliability of the original disks, make a second copy and only work on one of the copies. You will also need additional disks to copy recovered files on to, including possibly multiple copies of partially damaged files.

Now you can try recovery steps on a copy. Note that most these will need to be done on a fresh copy—the steps are destructive, this is one of the many reasons to only work on a copy. Do not destroy your originals:

  1. Let e2fsck -y /path/to/copy do it's thing. Probably it'll produce something you can mount. Go ahead and do so, copy files off.

  2. Extend your copy back to the original size (should be OK for it to be sparse; truncate -s can do this). Then it'll likely mount (do so read-only). Copy what you can. Unmount it, and again let e2fsck -y do its thing again. Again mount and copy what you can.

  3. Run fsck without -y and actually go through all those messages. E.g., I'd expect it actually to give you a choice of what to do when part of a file's data is in the lost area (replace with 0s, delete file). Possibly it gives choices about lost metadata too. I'd do -y first, because it will have a lot of questions for you...

  4. If you have an old filesystem image backup, combine the 2TB you have + the missing 1TB from the backup. fsck the result, see if you can get any additional files off it. Risk that the recovered files are corrupt is quite high, though.

  5. Use programs that scan through the filesystem image for data patterns (e.g., photorec to look for JPEGs). This is the only one that doesn't strictly need a fresh copy.

  6. In theory, ¾ of the final ⅓ of the "failed" disk #4 contain some of your missing data. If you can figure out the sector/chunk mapping (I sure don't know it!) you could copy ~250GB from that disk back in to your image, and repeat all the above recovery steps to recover additional files.

Note that all these recovered files may have corruption in them (e.g,. blocks full of 0s instead of data). Verifying them is easy if you have checksums somewhere, but otherwise a tedious manual process.

We have a bunch of questions about recovering data from damaged filesystems; as long as you only work on copies you can experiment without putting your data at further risk.

| improve this answer | |
  • are you 100% sure i have destroyed my data? if i really have destroyed it, can you tell me how i can "probably recover files that happened to be on the 1st TB of the filesystem" – Anton S. Sep 8 '17 at 15:10
  • 1
    @AntonS. You truncated the array, meaning "just cut it off". So anything that was stored on the 2nd half of that filesystem (data or metadata) is now gone (that second --grow that changed the number of devices overwrote it all). I'll edit in some of the data recovery steps... – derobert Sep 8 '17 at 15:13
  • ok i trunkated it but can i access what was on the first part (in fact, two thirds) of the disk (md0 array)? – Anton S. Sep 8 '17 at 15:17
  • @AntonS. recovery notes added. – derobert Sep 8 '17 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.