Drive was unable to assemble into the array

Question

I ran

mdadm -A /dev/md0 /dev/sda1 /dev/sdb1

Only /dev/sdb1 was loaded into the array though. I have a few more arrays on the same two drives too. Each time the partition on sda failed. dmesg told me that sda was out of sync... Since this was from a rescue cd. I've disconnected sda (hardware wise) for the time being, since it was preventing me from booting.

How should I proceed? Is this likely the cause of a borked drive? I had some weird fs issues the other day I couldn't track down (maybe a precursor): missing files that later magically re-appeared. Maybe a missing cable?

The main question is how do I try to re-sync the drive?

cat /proc/mdstat
Personalities : [raid10] 
md3 : active raid10 sda4[1]
      955683840 blocks super 1.2 512K chunks 2 far-copies [2/1] [_U]

md2 : active raid10 sda3[1]
      10483712 blocks super 1.2 512K chunks 2 far-copies [2/1] [_U]

md1 : active raid10 sda2[1]
      10484736 blocks 512K chunks 2 far-copies [2/1] [_U]

md0 : active raid10 sda1[1]
      101376 blocks 512K chunks 2 far-copies [2/1] [_U]

unused devices: <none>

update

I ran badblocks on the whole other drive, and a long smartctl test, it found no problems.

a request output of mdadm -D /dev/md0 (I have md0-3 if others are needed)

/dev/md0:
        Version : 0.90
  Creation Time : Mon May 31 20:24:14 2010
     Raid Level : raid10
     Array Size : 101376 (99.02 MiB 103.81 MB)
  Used Dev Size : 101376 (99.02 MiB 103.81 MB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Mon Oct 25 07:58:25 2010
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

         Layout : far=2
     Chunk Size : 512K

           UUID : 30ffe1d2:f5759995:820bb796:b5530bd2 (local to host slave-iv)
         Events : 0.212

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8        1        1      active sync   /dev/sda1

Since I've found no actual issues with the drive, but obviously something went wrong, I'm wondering what I should do next? as of today a full backup of important data is in place

update 2 Whenever I try to add what was sda back in (at least without wiping it) it screw's up my boot process with a kernel magic number error. I'm guessing because the kernel version got out of sync. currently this drive is in an external enclosure as sdd. Should I re-add (re-sync) this drive while it's connected via usb? will that 'cause problems?

df
Filesystem            Size  Used Avail Use% Mounted on
udev                   10M  284K  9.8M   3% /dev
/dev/md1              9.9G  7.0G  2.4G  75% /
shm                   3.0G  1.5M  3.0G   1% /dev/shm
/dev/md0               96M   15M   77M  16% /boot
/dev/md2              9.9G  6.5G  3.0G  69% /var
/dev/md3              898G  451G  402G  53% /home
none                  1.0G   45M  980M   5% /tmp
/dev/sdb1             992M   36M  956M   4% /media/D4A4-B7C1

each md drive has an sda/sdb corresponding. it was the sda drive (or 0 drive) in the array that I had to pull.

Answer 1

Okay, clearly one of your disks is not active in the array right now. Let's say, under the current enumeration, that /dev/md0 is missing /dev/sdb1.

To add /dev/sdb1 back in its former state:

mdadm -a --re-add /dev/md0 /dev/sdb1

You may want to re-check the array to make sure the data is all consistent.

echo check >>/sys/block/md0/md/sync_action

To re-mirror from scratch:

Make sure that /dev/sdb1 is completely removed from the array.

mdadm -f /dev/md0 /dev/sdb1
mdadm -r /dev/md0 /dev/sdb1

Wipe out the configuration on /dev/sdb1 so that mdadm will add it to back to the array as if it were a completely new disk.

mdadm --zero-superblock /dev/sdb1

Add /dev/sdb1 back to the array as a member.

mdadm -a /dev/md0 /dev/sdb1

---

Syncing a disk on USB works, slowly.

However, I would guess that your problem is that your bootloader isn't installed properly on your second drive, which is preventing you from booting when it's inserted.

You could swap the order of those two drives, so that your system boots off of the drive with known good configuration.

Once you are up and running and can sync both drives, you should re-install the bootloader on both drives. This way you can always boot no matter which drive is missing or enumerates first.

How to re-install your bootloader varies:

• If you are using lilo ≥ 22.0, specify boot = /dev/md0 and raid-extra-boot = mbr in /etc/lilo.conf and re-run lilo. If you are using an older lilo, create two lilo configurations, one for each disk in the the array, and install lilo to each disk individually.
• If you are using Grub, try to run grub-install /dev/sda and grub-install /dev/sdb. If that fails, then try using the Grub shell to setup (hd0) and setup (hd1) (cat /boot/grub/device.map to see what Grub's hard disk numberings are).