3

I recently moved my Kubuntu 12.04 workstation to a new location. It was shutdown normally but when I rebooted it in its new home the RAID-1 array /dev/md0 was GONE! The Elements of the array are listed as spares ?(!)

Contained on the RAID-1 Array /dev/md0 were only the critical files and the OS is on its own HDD.

Both elements of the array appear to be healthy and listed as: Linux raid autodetect.

fdisk -l output:

# fdisk -l

Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000669b6

Device Boot      Start         End      Blocks   Id  System
/dev/sdc1              63  1953520064   976760001   fd  Linux raid autodetect

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes                                         
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors               
Units = sectors of 1 * 512 = 512 bytes                                                
Sector size (logical/physical): 512 bytes / 512 bytes                                 
I/O size (minimum/optimal): 512 bytes / 512 bytes                                     
Disk identifier: 0x0000f142                                                           

Device Boot      Start         End      Blocks   Id  System                        
/dev/sdb1              63  1953520064   976760001   fd  Linux raid autodetect   

mdadm output:

# mdadm --examine /dev/sdb1 /dev/sdc1
 /dev/sdb1:
         Magic : a92b4efc
       Version : 1.2
   Feature Map : 0x0
    Array UUID : f3d0cc70:52dfd786:d81c7e2d:1c12b06d
          Name : forsaken:0
 Creation Time : Tue Sep  3 04:52:19 2013
    Raid Level : -unknown-
  Raid Devices : 0

Avail Dev Size : 1953517954 (931.51 GiB 1000.20 GB)
   Data Offset : 2048 sectors
  Super Offset : 8 sectors
         State : active
   Device UUID : 75c525b2:cdfbc3c4:918ac90a:a1bedfd0

   Update Time : Thu Nov 20 16:50:46 2014
      Checksum : ff0eb2ba - correct
        Events : 1


  Device Role : spare
  Array State :  ('A' == active, '.' == missing)

/dev/sdc1:
         Magic : a92b4efc
       Version : 1.2
   Feature Map : 0x0
    Array UUID : f3d0cc70:52dfd786:d81c7e2d:1c12b06d
          Name : forsaken:0
 Creation Time : Tue Sep  3 04:52:19 2013
    Raid Level : -unknown-
  Raid Devices : 0 

Avail Dev Size : 1953517954 (931.51 GiB 1000.20 GB)
   Data Offset : 2048 sectors
  Super Offset : 8 sectors
         State : active
   Device UUID : e62e960a:3adf4b5e:f1fb773f:a7a80cfa  

   Update Time : Thu Nov 20 16:50:46 2014
      Checksum : 4ee25b00 - correct
        Events : 1


  Device Role : spare
  Array State :  ('A' == active, '.' == missing)

Stopping the array and attempting a self-assembly:

# mdadm --stop /dev/md0
  mdadm: stopped /dev/md0
# mdadm --assemble -v --scan --uuid=f3d0cc70:52dfd786:d81c7e2d:1c12b06d
  mdadm: looking for devices for /dev/md/0
  mdadm: cannot open device /dev/sr0: No medium found
  mdadm: no RAID superblock on /dev/sdb
  mdadm: no RAID superblock on /dev/sde1
  mdadm: no RAID superblock on /dev/sde
  mdadm: no RAID superblock on /dev/sdd1
  mdadm: no RAID superblock on /dev/sdd
  mdadm: no RAID superblock on /dev/sdc
  mdadm: no RAID superblock on /dev/sda9
  mdadm: no RAID superblock on /dev/sda8
  mdadm: no RAID superblock on /dev/sda7
  mdadm: no RAID superblock on /dev/sda6
  mdadm: no RAID superblock on /dev/sda5                                                                                                 
  mdadm: no RAID superblock on /dev/sda4                                                                                                 
  mdadm: no RAID superblock on /dev/sda3                                                                                                 
  mdadm: no RAID superblock on /dev/sda2                                                                                                 
  mdadm: no RAID superblock on /dev/sda1                                                                                                 
  mdadm: no RAID superblock on /dev/sda                                                                                                  
  mdadm: /dev/sdb1 is identified as a member of /dev/md/0, slot -1.                                                                      
  mdadm: /dev/sdc1 is identified as a member of /dev/md/0, slot -1.                                                                      
  mdadm: added /dev/sdc1 to /dev/md/0 as -1                                                                                              
  mdadm: added /dev/sdb1 to /dev/md/0 as -1                                                                                              
  mdadm: /dev/md/0 assembled from 0 drives and 2 spares - not enough to start the array.

Attempting to assemble the array and force it to run:

# mdadm --assemble -v --scan --force --run --uuid=f3d0cc70:52dfd786:d81c7e2d:1c12b06d
mdadm: looking for devices for /dev/md/0
mdadm: cannot open device /dev/sr0: No medium found
mdadm: no RAID superblock on /dev/sdb
mdadm: no RAID superblock on /dev/sde1
mdadm: no RAID superblock on /dev/sde
mdadm: no RAID superblock on /dev/sdd1
mdadm: no RAID superblock on /dev/sdd
mdadm: no RAID superblock on /dev/sdc
mdadm: no RAID superblock on /dev/sda9
mdadm: no RAID superblock on /dev/sda8
mdadm: no RAID superblock on /dev/sda7
mdadm: no RAID superblock on /dev/sda6
mdadm: no RAID superblock on /dev/sda5
mdadm: no RAID superblock on /dev/sda4
mdadm: no RAID superblock on /dev/sda3
mdadm: no RAID superblock on /dev/sda2
mdadm: no RAID superblock on /dev/sda1
mdadm: no RAID superblock on /dev/sda
mdadm: /dev/sdb1 is identified as a member of /dev/md/0, slot -1.
mdadm: /dev/sdc1 is identified as a member of /dev/md/0, slot -1.
mdadm: added /dev/sdc1 to /dev/md/0 as -1
mdadm: added /dev/sdb1 to /dev/md/0 as -1
mdadm: failed to RUN_ARRAY /dev/md/0: Invalid argument
mdadm: Not enough devices to start the array.

This is still not working. How can I re-assemble my RAID-1 array and regain access to my data?

1

It seems the RAID metadata got damaged somehow. How did that happen? Once you've fixed any misconfigurations, errant scripts, hardware problems, etc., try to mount read-only:

mkdir /mnt/{sdb1,sdc1}
mount -o ro,loop,offset=$((2048*512)) /dev/sdb1 /mnt/sdb1
mount -o ro,loop,offset=$((2048*512)) /dev/sdc1 /mnt/sdc1

See if either one mounts, verify files of known types, compare to see if either side has damaged files. This would also be a good time to make a backup.

Once you have decided which side to keep, use it to create a new RAID.

Umount first:

umount /mnt/{sdb1,sdc1}

If there is any /dev/md* using either device according to /proc/mdstat, stop it.

mdadm --stop /dev/md0

Create a new RAID-1, in this example using /dev/sdb1. Note you must use the correct metadata version, and the correct offset. So do this only if the mount above actually worked, otherwise you'll have to determine the correct offset first.

mdadm --create /dev/md0 --metadata=1.2 --data-offset=2048 \
      --level=1 --raid-devices=2 /dev/sdb1 missing

Again, mount it read-only to verify that it actually worked as expected:

mount -o ro,loop /dev/md0 /mnt/sdb1

If all is in order, finally add the missing device to your RAID.

mdadm /dev/md0 --add /dev/sdc1

That will overwrite /dev/sdc1 with the data on /dev/sdb1, hopefully putting your RAID back in sync.


On a sidenote, your partitions start at sector 63; if your disks still have 512 byte sectors, that is fine. However most newer disks use 4k sectors, so if you have to replace a disk in your RAID, you may also have to pay attention to partition alignment for those new disks.

  • Worked GREAT! Thank YOU sooo Much !!!! – exitpoint Nov 24 '14 at 16:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.