5

I'm thinking that the smart thing to do with my raid setup is to replace the drives before they start failing and as they start to get old... I can't really afford a lot of cloud backup space, and I want to get a jump on the guaranteed eventual fail of my drives due to wear.

I have 3 2TB drives with GPT, grub, a small system raid1 partition, and a large raid5 home partition. I'm using Arch Linux.

I was going to replace the drives one at a time. I wanted to post my plan of action and see if anyone could think of a reason why it wouldn't work or if there was a better way to do it.

step one:

figure out which device (ie /dev/sda) I am replacing by unplugging it physically and checking /proc/mdstat to find out the /dev/sdx that fails.

step two:

Plug it back in and use sfdisk to copy the partition table

sfdisk -d /dev/sdx > partition.layout

step three:

Put in a new physical drive of the same size

step four:

sfdisk /dev/sdx < partition.layout

step five:

Use mdadm to add the new drive to the array based on the instructions on the arch wiki.

mdadm --add /dev/md0 /dev/sdx1
mdadm --add /dev/md1 /dev/sdx2

step six:

Reinstall grub? wait for the resync to complete, then repeat the whole process with the other 2 drives?

I guess my question is mostly like, will this work out? is there anything I'm missing? I don't want to miss something obvious and lose all my data.

Thank you very much for any assistance/insight.

Edit:

Just to get the results of the discussion down in the same place, I wanted to say that I figured out how to have mdadm and smartmontools (smartd) montior and notify me via email if things start going bad with my hard drives. I set up ssmtp with a gmail account that I have synced to my phone.

Since I already bought the new drives, I'm going to keep them around, and swap them in as things go bad. It is my understanding that eventually all hard drives fail. Thanks for the suggestions and protips on how to do that (without degrading the array). Once I can afford an upgrade I'm going to use ZFS with an ECC motherboard/memory/etc. and thanks for the tips in that direction. Thanks a lot you guys really helped :D

  • Consider also whether you might benefit from a "hot spare" drive. A hot spare is an inactive drive that can be automatically added to the array to replace an active drive that fails. This reduces the window during which you are vulnerable to double drive failures to the time it takes for the array to sync data to the hot spare that is being made active. – Wayne Conrad Aug 8 '16 at 2:10
  • 3
    Remember that RAID is not a backup: you need to have some other copy of your data somewhere, even if it's just a single large hard drive in a drawer. – pjc50 Aug 8 '16 at 9:19
  • your step one is likely to result in one or more disk failures due to re-syncing the raid after the disk is re-installed. Instead, get a manual label writer label printer, and print out the serial numbers for each disk (these are usually printed on the disk or you can get them from /dev/disk/by-id/ to print in a batch). label each disk (and/or it's caddy if it's in a hot-swap bay) so that the label is clearly visible without removing the disk. – cas Aug 8 '16 at 11:27
  • in case it's not obvious, labelling the disks should be done while the system is shutdown if you need to pull them out of the system to label them correctly. – cas Aug 8 '16 at 11:36
11

That's a bad idea because you're deliberately degrading your RAID and Resyncs might fail unexpectedly. It's better to hook the new disk up to the system (so you have n+1 disks) and then use mdadm --replace to sync it in. That way the RAID never degrades in between.

You don't have to fail / remove drives to find out which is which. You can see a device's role number in mdadm --examine, in mdstat output [UUU] in role numbers is [012]; and you can check the drive's serial number with hdparm or smartctl and compare to the sticker on the drive itself.

For partitions, it might be better to use GPT nowadays instead of MSDOS. If you are not only replacing disks but also upgrading them in size, you might have no other choice anyhow, since MSDOS partitions pretty much stop at 2TB.

Personally I don't do this at all. So what if the disks are 3 years old? Disks live a lot longer than that, and new disks die all the same.

It's much more important to test your disks on a regular (automated) basis, and replace disks once they have their first pending/uncorrectable/reallocated sector, read error in selftest, or other issues.

Even more important is having backups of any data you don't want to lose.

You could also switch to RAID6 for more redundancy, but the case of two disks dying at the same time is highly unlikely as long as you actively check for errors. Don't let your rebuild be your first read test in years.

  • 2
    "MBR" would be a much better term to use (the way it has been used in this answer) than "MSDOS". – TOOGAM Aug 8 '16 at 4:52
  • Thanks a lot, this is really helpful. I will add the new disk without taking out the old one and check out mdadm --replace. What do you mean when you say check the disks on a regular basis? I'm not really sure how to do that. I know about scrubbing, and I know I can set up mdadm monitoring, is that what you mean? I haven't done that, mostly because setting up system mail seems like a hassle, and because I don't actually log on to the computer that often (to get the mail), I use it mostly as a file server to my laptops. – Eveready Aug 8 '16 at 6:51
  • I guess what I am asking is, what do you do to "test your disks on a regular automated basis?" I like getting things like this right! Thank you again. – Eveready Aug 8 '16 at 6:54
  • @Eveready Modern hard drives have a series of self-test functions you can access via SMART, which also reports on various aspects of the drive's health whether you run the self-tests or not. I upvoted this for making the point that old drives don't necessarily fail; in my desktop I've got a 1TB drive I picked up in 2009 with 63,000 hours on the clock that has yet to report any sort of error or even the slightest indication that it might fail in my lifetime. When it does give the slightest indication, or possibly when SATA becomes obsolete, only then will I get rid of it. – Michael Hampton Aug 8 '16 at 8:40
5

You first step is unnecessary and still gives you no guarantee that the /dev/sdX stays stable all the time (i.e. that it points to the same device). In the worst case, plugging it in again triggers some long-running re-sync operation.

The better and reliable approach is to work with serial numbers (S/N, SN) of the drives. They are printed on the drives and they can be queried from the system.

For example, looking at a random system:

# cat /proc/mdstat 
Personalities : [raid1] 
md126 : active raid1 sda2[0] sdb2[2]
      976245464 blocks super 1.2 [2/2] [UU]
      bitmap: 1/8 pages [4KB], 65536KB chunk

Then I can get the serial number of - say - sda via:

# hdparm -i /dev/sda

/dev/sda:

 Model=SAMSUNG HD103UJ, FwRev=1AA01113, SerialNo=S13PJ0123456789

(I redacted the actual serial number)

I can also get the serial number via looking at the symbolic links under /dev/disk/by-id:

# ls -l /dev/disk/by-id | grep 'sda$'
lrwxrwxrwx. 1 root root  9 2016-08-05 09:49 ata-SAMSUNG_HD103UJ_S13PJ0123456789 \
  -> ../../sda

Thus, I can write down the serial number of the drive I want to replace, power-down the system, and replace the drive with the matching serial number (which is printed on the drive label).

ProTip #1: When adding new drive, write the serial number on an extra label and put that label on that side that is directly visible (e.g. the connector side).

The /dev/disk/by-id/... links also come handy when adding the new device to the RAID.

ProTip #2: Chose drives heterogeneously, i.e. use models from independent manufacturers, drives that already have different power-on-hours etc. Thus you lower the probability of multiple failures due to systematic issues.

ProTip #3: Use a checksumming filesystem like BTRFS or ZFS. Besides the benefits of checksumming, they also support replacing a drive without having to remove it first.

  • Right now I thnk recommendations of BTRFS should be accompanied by a warning that (phoronix.com/… build in RAID implementations is broken] (using BTRFS on top of md - or hardware raid - as the OP uses currently is not affected). – Henrik - stop hurting Monica Aug 7 '16 at 21:57
  • 1
    @Henrik The link you posted is broken. – Michael Hampton Aug 7 '16 at 22:47
  • 1
    This is phoronix.com/… cc @MichaelHampton – Braiam Aug 8 '16 at 2:32
  • @Henrik, the article only talks about issues with RAID-5/RAID-6. Since BTRFS gained RAID-5/6 support, the BTRFS site contains some prominent warnings regarding its experimental status. Thus, yes, currently, you should only use RAID-1 or RAID-10 for redundant BTRFS configurations. – maxschlepzig Aug 8 '16 at 6:54
  • 1
    Thanks for this, I will use your protips. I am going to try ZFS next time I upgrade, that seems like the best choice. – Eveready Aug 8 '16 at 7:04
1

No, I would not pre-emptively replace the HDDs.

In your case, I would suggest switching to a dedicated RAID file system; my personal darling is ZFS, but I guess BTRFS would work as well.

With ZFS, you would install the new HDD, add it to your pool as hot spare, let ZFS "scrub" (auto-check for errors) your HDDs regularly, and activate the hot spare when you see errors on the scrub. Then ZFS will automatically initialize it, and when that's done you can rip out the defective one. As long as the HDDs in your pool are not the same make/run, you can be reasonably sure that they don't fail right at the same time (depending on your level of paranoia, of course).

Next time, it goes without saying that you should label the HDDs when you install them so they can be found without removing them... ;)

If you do all this in parallel to your existing solution (provided your case has enough space for two additional HDDs), then you're set for the future. You can migrate all your data over to new disks, and then use your old disks as hot spare for the future.

(N.B.: if you go to ZFS, which was just used as an example here, please google "zfs ecc" and chose wisely).

  • If ZFS requires ECC RAM, then so does Btrfs, for exactly the same reasons. – Michael Hampton Aug 8 '16 at 8:47
  • As I said, I have no knowledge about BTRFS internals. ZFS requires ECC because in effect checksums the whole disk and refuses to use the whole disk when 1 bit flips - so the problem is not so much that a single sector is corrupted, but that it takes everything else with it (that's by design, not an error). That said, ECC is certainly advisable for any RAID, but especially so with ZFS (not an option anymore ;) ). – AnoE Aug 8 '16 at 8:51
  • I've been using zfsonlinux for over 5 years on 2 home servers (1 file server with "export" and "backup" pools, 1 mythtv backend), both without ECC RAM. ECC is recommended but not required. With properly tested and burned-in (i.e. leave it running memtest for 24+ hours) RAM, non-ECC is safe enough to use for a home server. ECC isn't even available on most consumer Intel CPUs and motherboards (all server and most consumer AMD CPUs and m/bs are ECC-capable). – cas Aug 8 '16 at 11:34
  • Since the point of this question is not primarily to discuss the use of ECC, I have reworded it slightly to make it less discussion-prone. – AnoE Aug 8 '16 at 11:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.