Hi I am going to build a 6 drive consumer level RAID 5 (10TB) with Ubuntu10.10 and using EXT4 as filesystem, and OS on another drive.
Question: Should you use LVM on top of the RAID5 or should just directly use EXT4 on top?
5 Answers
It would be a good idea to use LVM on top of RAID. Then you can grow the RAID array and also grow the LV.
LVM on top of anything is probably a good idea because it gives you quite a bit of flexibility at pretty marginal cost (the extra abstraction layer is really cheap compared to disk I/O).
That said, I'd use RAID6, as RAID5 leaves you with no redundancy during a rebuild, which is precisely the time of high stress where drives are most likely to fail.
If you only need one filesystem in your RAID then there are not real advantages of using LVM. Contrary, without LVM on top you get following advantages:
- reduced overall complexity
- better performance
Btw, you can resize ext4 filesystems without LVM as well ( resize2fs(8) ).
Regarding the performance impact of lvm, some people report decreases by 5 % other a 20 fold degradation when snapshotting is involved, i.e. it depends on the lvm features/layouts you use and your usage pattern.
answered Mar 8, 2011 at 8:56
-
1Complexity goes both ways: LVM adds a layer but makes many common tasks easier and safer. As for performance, I've never seen a benchmark showing a substantial slowdown from LVM, do you know one? Mar 8, 2011 at 18:30
-
I don't think the added layer slow things down too much. Most of the delay comes from actually reading/writing stuff to the disk. A few simple calculations and lookups should not cause any measurable delay. LVM can increase fragmentation but with 16MB extent sizes and multi-gigabyte LVs I don't think that's significant either. I never measured it but I'm using LVM everywhere I can and I never noticed any slowdown.– stribikaMar 8, 2011 at 18:53
-
@Gilles, @stribika: I updated it and included links to benchmarks - if you google for 'lvm performance' you find more stuff. Gilles, sure if you regularly want to use lvm features then lvm makes your life easier - but the original poster did not include details about that. For example I have some raid devices with exactly one filesystem where there is really no need for any lvm related stuff. Mar 9, 2011 at 10:20
-
Thank you. Even if I don't regularly use LVM features, I find it valuable for administration tasks such as moving a filesystem to a different disk (with LVM, you can do it without unmounting and with no risk of overwriting the wrong partition by mistake). Mar 9, 2011 at 19:09
LVM and RAID have some similar functionality (both can do mirroring and stripping) but they serve different purposes.
RAID is designed to make the storage more reliable, faster and bigger. The differend RAID levels each avchieve one or more of these 3 goals. For example RAID0 gives you speed and more space while RAID1 provides reliablility and fast read. RAID5 gives you some reliability at the cost of some write speed RAID6 does this even more. With 10TB I would consider creating partitions on the disks and adding the partitions to different RAID arrays. For example you can have swap on RAID0, system files on RAID5, boot partition on RAID1 (so grub can use it), /home on RAID1+0.
LVM is designed to hide what storage you use. It doesn't matter how many disks or where all you see is a logical volume. You can easily add/remove physical volumes without the filesystems on the logical volume knowing about it. Most importantly it gives you snapshots. Snapshots save lives. Make one before every upgrade or make a daily snapshot of /home.
Having too many snapshots can greatly reduce write performance on the original LV. Snapshots are implemented with copy-on-write which causes an extra read and write operation/snapshot. Even for very small writes a complete block is copied. See the links in maxschlepzig's answer.
An other advantage is not having to know in advance how large the filesystems will be. You can create small LVs and grow them as needed. Use the extra space for snapshots don't just create a 9.9TB /home immediately.
So yes it makes sense to use both.
-
cool, 1 question: Is there any redundancy can be used for LVM? (i.e. use LVM instead of software raid?)– c2h2Mar 9, 2011 at 4:49
-
@c2h2: Technically, yes, LVM includes features that resemble RAID-0 and RAID-1 (logical volumes spanning over multiple physical volumes, and mirroring). For more advanced RAID modes, you need something else. Even for RAID-0 and RAID-1, mdraid is easier and (I think) performs better. Mar 9, 2011 at 19:12
Unless you really need the whole 10 TB I'd rather recommend building 3 RAID 1 arrays and adding those to a LV.
The reason: The next time you are going to increase your storage capacity, you don't need to replace all 6 disks. Remember that RAID 5 needs identical storage capacity of all drives. You can instead replace just one of your RAID 1 arrays.
-
This is software raid not HW raid. You can easily add a partition of the disk to the raid, therefore capacity difference of the disks is not an issue. I have 2 Raid 1 over 3 disks (0,5T 1T and 1,5T) on one of my machines. Mar 8, 2011 at 14:45
-
@Let_Me_Be: It could be an issue. I have a RAID5 setup at home. Originally it was 3x1TB but later I added a 1.5TB disk because 1T was only marginally cheaper. What can I do with the remaining 500GB? Swap? /tmp? RAID1+0 is easier to extend because I only need pairs of same sized disks.– stribikaMar 8, 2011 at 15:40
-
That's correct: you can add a larger disk to the array, but the kernel will only use 2GB of that disk, since it needs partitions of same size in each array. The rest is wasted storage space Mar 9, 2011 at 15:08
-
1@stribika: Use it for whatever you like, just as if you had extra room on a single disk of a RAID1+0 array. If you have two disks that are larger, you can use RAID1 on the rest of the space. Mar 9, 2011 at 19:10