Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a software RAID5 array (Linux md) on 4 disks.

I would like to replace one of the disks with a new one, without putting the array in a degraded state, and if possible, online. How would that be possible?

It's important because I don't wan't to:

  • take the risk of stressing the other disks so one may crash during rebuild,
  • take the risk of being in a "no-parity state" so I don't have a safety net for some time.

I suppose doing so online is too much asking and I should just raw copy (dd) the data of the old disk to the new one offline and then replace it, but I think it is theoretically possible...

Some context: Those disks have all been spinning almost continuously for more than 5.5 years. They still work perfectly for the moment and they all pass the (long) SMART self-test. However, I have reasons to think that one of those 4 disks will not last much longer (supposed predictive failure).

share|improve this question
add comment

3 Answers 3

If you don't mind running RAID-6 (2 parity disks rather than 1), and if you're running mdadmin 3.1.x or higher, you could convert your RAID-5 array to RAID-6 to add an additional parity disk. This will will place the array under stress during the rebuild, however. And it has some performance implications since there are more parity disks to update during writes.

But if it completes successfully, then you can keep your failing disk in place and when it ultimately fails, you've still got parity protection for the array. I think you can conver the array from RAID6 back to RAID5 if you don't wait to keep it as RAID6.

I don't know of an online way to keep the array as RAID-5 and replace the disk without putting the array in degraded mode, as I think you have to mark it as failed to replace it. Your dd copy idea might be the way to do that.

share|improve this answer
    
@haukelaging FYI it is now possible since kernel 3.2, see my answer. –  Totor Dec 10 '13 at 23:20
add comment

This may be possible meeting the requirements

  1. online
  2. don't stress any disk except for the one which is to be replaced

But even if the following may work you will probably not find any recommendation of that kind "in the books"...

Idea:

  1. Take disk OLD out of the array (for a short moment): mdadm --manage /dev/raid5 --fail /dev/OLD
  2. Create a new md device (RAID-1) from disks OLD and NEW: mdadm --build /dev/md42 --level=mirror --raid-devices=2 /dev/OLD /dev/NEW
  3. Put the RAID-1 back in the array (instead of /dev/OLD): mdadm --manage /dev/raid5 --re-add /dev/md42

What should :-) happen:

  1. The RAID-5 gets /dev/md42 in sync. This should not take long.
  2. The RAID-5 is normally operational again (but slower).
  3. /dev/NEW is synced with /dev/OLD.

Watch the sync progress (cat /proc/mdstat or mdadm --monitor). If the sync is finished take the RAID-1 out of the RAID-5, stop the RAID-1, re-add /dev/NEW to the RAID-5. If everything is fine, overwrite the mdraid superblocks on /dev/OLD in order to avoid problems: mdadm --zero-superblock

Warning: The fast RAID-5 sync may work only if you use a bitmap. If you don't have one then better make a test with a dummy RAID-5 (without a bitmap) first. Or add one. At least adding an external one should be possible. Otherwise it may be necessary to stop the RAID-5 before changing the devices. If you boot from the RAID-5 this would become a bit complicated, though.

share|improve this answer
add comment
up vote 1 down vote accepted

Using mdadm 3.3

Since mdadm 3.3 (released 2013, Sep 3), if you have a 3.2+ kernel, you can proceed as follows:

# mdadm /dev/md0 --replace /dev/sdd1 --with /dev/sdc1

sdd1 is the device you want to replace, sdc1 is the preferred device to do so and must be declared as a spare on your array.

The --with option is optional, if not specified, any available spare will be used.

Older mdadm version

Note: You still need a 3.2+ kernel.

First, add a new drive as a spare (replace md0 and sdc1 with your RAID and disk device, respectively):

# mdadm /dev/md0 --add /dev/sdc1

Then, initiate a copy-replace operation like this (sdd1 being the failing device):

# echo want_replacement > /sys/block/md0/md/dev-sdd1/state 

Result

The system will copy all readable blocks from sdd1 to sdc1. If it comes to an unreadable block, it will reconstruct it from parity. Once the operation is complete, the former spare (here: sdc1) will become active, and the failing drive will be marked as failed (F) so you can remove it.

Note: credit goes to frostschutz and Ansgar Esztermann who found the original solution (see the duplicate question).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.