Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

This is a hypothetical question, I'm just evaluating in which way I should configure my hard drives for my home-server and for that I need to know this:

Let's say I have a Raid5 configured with mdadm and put the resulting devices md0 and md1 into LVM to get one big, failsafe drive:

enter image description here

Now the grey 1TB drive crashes and since it as been a few years when 1TB drive were the most recent technology, I want to buy a 2TB drive instead.

What would be the best way to bring this new 2TB drive into my Raid system? Of course I could replace the broken 1TB drive with a 2TB one, but then I would waste 1TB of this drive that can't be used. Would it be possible to shrink the md1-array and place the new drive into the md0-array instead?

share|improve this question
1  
    
Do you know an alternative? –  Simon D. Seim Sep 21 '13 at 21:39
    
RAID10. –  Martin von Wittich Sep 21 '13 at 21:40
    
I think it is a bit too much to use 50% of all drives for redundancy. Moreover.. Raid10 doesn't work with different sized disk, either. –  Simon D. Seim Sep 21 '13 at 21:45
    
A completely different question: why exactly does your home server need a RAID anyway? Do you have planned for backups, ie. do you have remaining spare hard disks that you will use for backup purposes? –  Martin von Wittich Sep 21 '13 at 21:51

3 Answers 3

With Software RAID, you don't have to use whole disks.

If you have 3x2TB and 3x1TB, and planning to replace the 1TB with 2TB in the future, you could use 1TB members. So that's RAID5 (or if you prefer RAID6) over 6x1TB, and RAID5 over 3x1TB. So the 2TB will be shared by both RAIDs.

When you kick out an 1TB and add a 2TB instead, then one RAID will see a replacement, and the other will have the remaining 1TB added as new member.

share|improve this answer
    
So you say create partitions of 1TB size on the 2TB disks and run a Raid5 over the partitions instead of the whole disk? Sounds ok, what do you do with those two parallel running Raid-arrays when you swapped the last 1TB disk for a 2TB one? Is it possible to merge those two arrays into one? –  Simon D. Seim Sep 21 '13 at 21:50
    
LVM merges them into one. It doesn't really matter if it's one or two RAIDs. I actually use 250G members instead of 1TB - that way you can distribute some operations better (RAID grow/check/resync takes an awful lot of time with huge RAIDs). –  frostschutz Sep 21 '13 at 21:52
2  
be very careful how you set this up - if your raid5 includes two partitions on the same disk then if/when that disk dies, your raid5 array dies with it. it could work if you had 3 RAID-5 arrays (each made up of 3x1TB partitions) striped together. performance would be pretty bad, though, due to disk head contention between the R5 arrays. overall, not recommended. –  cas Sep 22 '13 at 2:40
1  
Yes, be careful. It's not what I'm suggesting at all, if you read that in there you are reading it wrong. Of each mdadm raid, each member must be on a different drive, naturally. As for disk head contention, you have the same issue in every other setup that uses partitions (it's not uncommon to have multiple md for various reasons), and I've found it to not make much of a difference at all, to be honest. ymmv. –  frostschutz Sep 22 '13 at 3:01
    
i didn't read it in there. that's exactly why i felt it was necessary to point out the risk. it's a big enough risk that it deserves a warning. –  cas Sep 22 '13 at 3:07

If you are still in the design stage (i.e. you're not already committed to mdadm and/or lvm) then I recommened that you seriously consider using a modern filesystem like Btrfs or ZFS.

btrfs is built-in to the mainline linux kernel, and zfs is available from the zfsonlinux web site as easily installable kernel modules or dkms packages for most linux distributions. There is also an Ubuntu PPA for zfsonlinux.

Both ZFS and btrfs have excellent, easy-to-use tools for managing disks and groups of disks. They also implement error-detection and correction (pretty much essential with large modern drives as errors are statistically almost guaranteed), sub-volumes, snapshotting (incl. access to and/or rollback of previous versions), and they both have a very useful snapshot-based method of sending a full or incremental backup to another machine (zfs send/receive and btrfs send/receive). Both have many other features and benefits, too long to list here.

One other big advantage that ZFS and btrfs have over lvm is that lvm logical volumes are of fixed size (kind of like a virtual disk partition) and you have to take special steps if you need to grow or shrink the size of an lv (e.g. if you make a mistake and allocate far too much space to /usr and not enough to /var, you'll have a lot of work ahead of you to fix that). With ZFS & btrfs, a sub-volume is more like a soft-quota allocation (with optional reservation) from the total space and changing the allocation is a trivial process that does not require any file-system 'juggling' (backup/restore/moving files around). In fact, you don't even need to set a quota per sub-volume if you don't want to - all sub-volumes will happily share the total pool of available space.

ZFS has the concept of pools, which are made up of virtual devices (vdevs) which are, in turn, made up of physical devices (i.e. disks). Using ZFS, your drives could be configured as a single pool containing two vdevs, one with a RAID-5 array made up of 3x2TB drives, and one made up of a RAID-5 array of 3x1TB drives. The vdevs are striped to make up the pool. (BTW, ZFS's raid5 and raid6 is not actually raid5 or raid6, it just works very similarly. it's called raid-z)

One of the nice things about ZFS is that you can easily replace drives in a vdev and when you replace all of the drives in a vdev with larger drives, then the extra space is automatically made available for storage. e.g. if you replace one or two of the 3x1TB drives with 2TB drives you would get no extra space, but as soon as you replaced the third drive you would instantly get the extra space.

You can also add another vdev (i.e. made up of one or more drives) at any time. e.g. you could add a pair of 3TB or 4TB drives. You can replace the drives in a vdev but you can not remove a vdev from a pool - if you need to do that, the only way is to backup, destroy the pool and create a new pool from scratch.

Btrfs recently got experimental raid-5/6 support, and has supported raid-0/1/10 for several years. It is conceptually quite different to zfs, and there are some things it does better than zfs (e.g. "rebalancing" data across drives when you add or change the drives in the btrfs filesystem) but overall it has less features than zfs. The big advantage is that it is in the mainline kernel and thus standard - guaranteed to be available on any modern linux system.

btrfs has specific support for using drives of different sizes, enabled by using the -d single option of mkfs.btrfs. See Using Btrfs with Multiple Devices for details.

share|improve this answer
    
I was expecting a magical solution here, but since you say ZFS requires all three drives to be replaced, how does that help? –  frostschutz Sep 22 '13 at 2:46
    
there is no magic solution, there are just tradeoffs you can choose between. flexibility, error correction, snapshotting etc in zfs or btrfs. or enhanced risk or reduced performance if using partitions as you suggested (which is not a bad solution, it just has some potential issues you need to be aware of) –  cas Sep 22 '13 at 2:50
    
Sure, but the question was specifically about how to not waste the remaining 1TB when replacing only one disk, or at last that's what I understood. I don't have experience with ZFS myself (I prefer the old fashioned mdadm-lvm-filesystem structure) but being able to stripe across all disks (the same way I'd do it with my partitioned mdadm setup) is something I'd have expected there at the least... oh well. –  frostschutz Sep 22 '13 at 2:56
    
the question was also about ...I'm just evaluating in which way I should configure my hard drives for my home-server... - knowing about other options is useful when deciding what to do. btw, zfs can stripe across multiple disks, it would be pretty useless at its primary job if it couldn't. –  cas Sep 22 '13 at 3:00
    
ZFS is not well supported on Linux and btrfs is not production ready yet. –  psusi Sep 22 '13 at 3:55

When you replace the 1 TB drive, add the new 2TB drive to md0, which will increase its capacity by 2 TB. Convert md1 from raid5 to raid10, which will reduce its capacity by 1 TB, thus giving you the full benefit of the additional 1 TB on the replacement disk.

As an alternative, you could convert md1 into a raid0, then add md1 to md0. This will give you 1 TB more usable space, with slightly higher risk of failure.

share|improve this answer
    
That would work (if you take care to pvmove before reducing anything) but when you replace the 2nd drive the remaining 1TB disk has nothing left to pair with, so you have to remove or replace the two remaining disks together when the time comes... –  frostschutz Sep 22 '13 at 2:51
    
@frostschutz, yes, you have to pvmove all logical volumes off of md1, then destroy it, and then recreate it as a raid10. –  psusi Sep 22 '13 at 3:53

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.