1

I ended up with my 3-disk mdadm array not being able to assemble because each of the disks had corrupted superblocks.

I tried to force reassembling the array using

mdadm --create

This resulted in the ability to assemble the raid, but now there is no (ext4) filesystem. Other indicators seem to show that the data is still there, but I need to fix the filesystem.

I tried to fix the filesystem by seeing if I can restore the superblock:

mkfs.ext4 -n /dev/md0
fsck.ext4 -b <tried_all_of_the_blocks> /dev/md0

But I get

fsck.ext4: Filesystem has unexpected block size while trying to open /dev/md0
The superblock could not be read or does not describe a valid ext2/ext4/ext4 filesystem...

So I take it that there is no filesystem and therefore, no valid superblocks from which to restore the filesystem.

I have two questions:

1. Can I safely use mkfs.ext4 on the raid to regenerate the filesystem, without losing the data that appears to still be in the array?

2. Can I fix the superblock of the array using a backup from one of the individual disks?

Obviously, I do not understand many things, so I appreciate your kind response.

Here is the mdadm --examine output:

    root@server:~# mdadm --examine /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/md0
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : aa70a8ec:192f1719:23bc5df4:1ddac384
           Name : server:0  (local to host server)
  Creation Time : Sat Oct 28 00:21:46 2017
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 4294703103 (2047.87 GiB 2198.89 GB)
     Array Size : 4294702080 (4095.75 GiB 4397.77 GB)
  Used Dev Size : 4294702080 (2047.87 GiB 2198.89 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=1023 sectors
          State : clean
    Device UUID : bf5a4ff5:e4e3659e:99caca7c:333475f3

Internal Bitmap : 8 sectors from superblock
    Update Time : Sat Oct 28 05:48:33 2017
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 71521ea5 - correct
         Events : 3358

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : aa70a8ec:192f1719:23bc5df4:1ddac384
           Name : server:0  (local to host server)
  Creation Time : Sat Oct 28 00:21:46 2017
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 4294703103 (2047.87 GiB 2198.89 GB)
     Array Size : 4294702080 (4095.75 GiB 4397.77 GB)
  Used Dev Size : 4294702080 (2047.87 GiB 2198.89 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=1023 sectors
          State : clean
    Device UUID : 91528c6d:77861852:a1a4f630:9d8eb8ab

Internal Bitmap : 8 sectors from superblock
    Update Time : Sat Oct 28 05:48:33 2017
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 9b0ed7c - correct
         Events : 3358

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : aa70a8ec:192f1719:23bc5df4:1ddac384
           Name : server:0  (local to host server)
  Creation Time : Sat Oct 28 00:21:46 2017
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 4294703103 (2047.87 GiB 2198.89 GB)
     Array Size : 4294702080 (4095.75 GiB 4397.77 GB)
  Used Dev Size : 4294702080 (2047.87 GiB 2198.89 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=1023 sectors
          State : clean
    Device UUID : bdc61c9f:321a7ca6:2ed914d0:d10b96a4

Internal Bitmap : 8 sectors from superblock
    Update Time : Sat Oct 28 05:48:33 2017
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 27a0a727 - correct
         Events : 3358

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
mdadm: No md superblock detected on /dev/md0.

This is all on Ubuntu Server 16.04, with mdadm version 1.2

  • mdraid doesn't protect against silent/filesystem corruption, only against disk failures. This is why you need backups. You might be able to salvage something from your disk using testdisk or some similar utility. For anyone to be able to provide more specific advice, you need to provide more details and a specific enough problem to answer. – sebasth Oct 28 '17 at 22:28
  • Would you all please check my completely revised question and consider reopening it? – fectus Nov 3 '17 at 16:26
  • "Can I safely use mkfs.ext4 on the raid to regenerate the filesystem...?" NO! If you run mkfs.ext4 you will overwrite whatever is present and create an empty filesystem. DO NOT DO THIS – roaima Nov 3 '17 at 17:52
  • Ubuntu Server 16.04, mdadm 1.2 – fectus Nov 3 '17 at 17:55
  • First sentence, "my 3-disk mdadm array not being able to assemble because each of the disks had corrupted superblocks". Are you talking about the RAID metadata or the filesystem superblock? The RAID metadata looks ok, and on that basis your RAID array should have assembled just fine. – roaima Nov 3 '17 at 18:23
0

Between the original failure and your recovery efforts, it sounds like your array is pretty well wrecked. If you're very lucky, mdadm --create put the disks back in their original order with the original layout. In that case, you can point some data-recovery software such as foremost at the array and pull some of the files out, or you can send the disks off to a data recovery company and hope they'll do a better job at it than you can.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.