1

I recently had a RAID1 volume fail. It had three primary partitions in it, all participating in separate, but similarly configured VGs. This configuration comes from the fact that the machine is easily a decade old, having upgraded from multiple single disks to multiple VGs with arrays.

I replaced the disk, created the partitions and added them back to the RAID arrays using mdadm. This worked for md0 and md1, but for md2, it's failed twice. This is the largest volume, it fails without error and is left in this state:

# mdadm --examine /dev/sdc3
/dev/sdc3:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : <omitted>
  Creation Time : Wed Nov  3 10:56:13 2010
     Raid Level : raid1
  Used Dev Size : 1914417728 (1825.73 GiB 1960.36 GB)
     Array Size : 1914417728 (1825.73 GiB 1960.36 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 2

    Update Time : Tue Oct 21 12:25:59 2014
          State : clean
 Active Devices : 1
Working Devices : 2
 Failed Devices : 1
  Spare Devices : 1
       Checksum : 5d247e27 - correct
         Events : 188268


      Number   Major   Minor   RaidDevice State
this     2       8       35        2      spare   /dev/sdc3

   0     0       8        3        0      active sync   /dev/sda3
   1     1       0        0        1      faulty removed
   2     2       8       35        2      spare   /dev/sdc3

How can I have a spare disk if the RAID 1 array has only one working disk?

There are some real gotchas on this one though:

  • The old disk has 512byte physical sectors, the new one 4096byte physical
  • The old disk has a conventional MBR partition, the new one a GPT partition
  • The old disk is 2TB, the new one 3TB.

The plan for the 3TB is to eventually replace the other 2TB disk and expand the array, but for now, it's unused space.


Note:

Old configuration:  RAID1, 1x2TB disk, 512 byte physical sectors, MBR from 2010
                           1x2TB disk, 512 byte physical sectors, MBR from 2010 (failed)

New configuration:  RAID1, 1x2TB disk, 512 byte physical sectors, MBR from 2010
                           1x3TB disk, 4096 byte physical sectors, GPT brand new

The 2TB disks were purchased in 2010, they've had a long run. One failed, so I replaced it with a 3TB disk. The GPT and physical partition thing might be a red herring. Just mentioning it in case it strikes anyone as important.


Some messages are appearing in syslog,

Oct 21 18:02:58 blacktower kernel: [169469.308398] mdadm: sending ioctl 800c0910 to a partition!
Oct 21 18:02:58 blacktower kernel: [169469.308402] mdadm: sending ioctl 800c0910 to a partition!
Oct 21 18:03:54 blacktower kernel: [169525.933975] mdadm: sending ioctl 1261 to a partition!
Oct 21 18:03:54 blacktower kernel: [169525.933978] mdadm: sending ioctl 1261 to a partition!
Oct 21 18:03:54 blacktower kernel: [169525.934250] mdadm: sending ioctl 1261 to a partition!
Oct 21 18:03:54 blacktower kernel: [169525.934253] mdadm: sending ioctl 1261 to a partition!

This thread suggests that it's harmless, http://www.spinics.net/lists/raid/msg44887.html

May not be related, but it's possible that there's an issue in the kernel verison. I'm on Squeeze LTS.

# uname -r
2.6.32-5-686

Hmm..

  • Either disk may be bad, preventing successful syncing. Did the resync ever complete? Follow its progress in /proc/mdstat and check dmesg. Also check smartctl -a for /dev/sda and /dev/sdc. – frostschutz Oct 21 '14 at 22:37
  • I removed the failed disk and replaced it, this is the result afterwards. Two of the three partitions on the disk properly resynced. – mgjk Oct 21 '14 at 23:29
  • 1
    Did you check SMART? Run a long self test on both disks. – frostschutz Oct 22 '14 at 14:04
  • hmm, SMART was telling me for a while that my disk was running at 192 degrees Celcius, looks like a few other errors got lost in the wash of dumb messages over the years. It doesn't look too severe, I'm duplicating the LVs with ddrescue now, creating a new RAID array... it looks like if the data moves smoothly, I'll need another replacement disk. – mgjk Oct 23 '14 at 2:31
1

You don't have a extra spare, the failing disk has been categorized as a "potential" spare once the array failed. The output is a little confusing, but you have 2 devices in the array:

   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 2

And 1 has failed, and 1 is still active:

 Active Devices : 1
Working Devices : 2
 Failed Devices : 1
  Spare Devices : 1

And the failed device is being counted twice, basically. When HDD is in this state I typically fail it and then remove it.

$ sudo mdadm /dev/md0 -f /dev/sdc3
$ sudo mdadm /dev/md0 -r /dev/sdc3

Usually when the devices start to fail you can sometimes get away and do the above and then re-add them back in:

$ sudo mdadm /dev/md0 -a /dev/sdc3

And perhaps get a little bit more life out of them, but this is typically the time where you don't want to risk your data and you should plan on removing the HDD and replacing it immediately.

Often times the HDDs can be removed and then you could run a tool such as HDAT2 or Spinrite on them to see if you can't repair them, at which point you can use them for something less critical or recycle them.

References

  • I replaced the disk already, this is the outcome after trying to rebuild. I can remove and re-add, it takes about 6 hours, then quietly returns to this state. I've done that twice. – mgjk Oct 21 '14 at 23:27
  • 1
    /dev/sdc3 is the new HDD? Perhaps it's faulty as well? I'd take it out and make sure that something like HDAT2 or Spinrite say it's OK. Also I'd swap the cable out, they tend to become faulty too over time. – slm Oct 21 '14 at 23:31
  • @mgjk - also what does dmesg show when this device fails? – slm Oct 21 '14 at 23:32
  • dmesg is overrun with some iptables messages, I've got it logging some firewall stuff. Syslog has a pair of repeating messages. I'm including them in the question, along with a link which discusses them... I might need a kernel upgrade. I do have some odd hardware for an older kernel. – mgjk Oct 22 '14 at 1:07
  • It looks like the kernel route would need a custom kernel for Debian Squeeze LTS, so... trying a hunch. The partitions are identically sized to the byte, but maybe between GPT and 4096 byte physical sectors, there could be a slightly different usage of data... I recreated the partition and added 200M, I also turned down the logging on iptables and it's running again. Will know if it worked tonight. – mgjk Oct 22 '14 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.