Hi I am going to build a 6 drive consumer level RAID 5 (10TB) with Ubuntu10.10 and using EXT4 as filesystem, and OS on another drive.
Question: Should you use LVM on top of the RAID5 or should just directly use EXT4 on top?
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.Sign up to join this community
LVM on top of anything is probably a good idea because it gives you quite a bit of flexibility at pretty marginal cost (the extra abstraction layer is really cheap compared to disk I/O).
That said, I'd use RAID6, as RAID5 leaves you with no redundancy during a rebuild, which is precisely the time of high stress where drives are most likely to fail.
If you only need one filesystem in your RAID then there are not real advantages of using LVM. Contrary, without LVM on top you get following advantages:
Btw, you can resize ext4 filesystems without LVM as well ( resize2fs(8) ).
Regarding the performance impact of lvm, some people report decreases by 5 % other a 20 fold degradation when snapshotting is involved, i.e. it depends on the lvm features/layouts you use and your usage pattern.
LVM and RAID have some similar functionality (both can do mirroring and stripping) but they serve different purposes.
RAID is designed to make the storage more reliable, faster and bigger. The differend RAID levels each avchieve one or more of these 3 goals. For example RAID0 gives you speed and more space while RAID1 provides reliablility and fast read. RAID5 gives you some reliability at the cost of some write speed RAID6 does this even more. With 10TB I would consider creating partitions on the disks and adding the partitions to different RAID arrays. For example you can have swap on RAID0, system files on RAID5, boot partition on RAID1 (so grub can use it), /home on RAID1+0.
LVM is designed to hide what storage you use. It doesn't matter how many disks or where all you see is a logical volume. You can easily add/remove physical volumes without the filesystems on the logical volume knowing about it. Most importantly it gives you snapshots. Snapshots save lives. Make one before every upgrade or make a daily snapshot of /home.
Having too many snapshots can greatly reduce write performance on the original LV. Snapshots are implemented with copy-on-write which causes an extra read and write operation/snapshot. Even for very small writes a complete block is copied. See the links in maxschlepzig's answer.
An other advantage is not having to know in advance how large the filesystems will be. You can create small LVs and grow them as needed. Use the extra space for snapshots don't just create a 9.9TB /home immediately.
So yes it makes sense to use both.
Unless you really need the whole 10 TB I'd rather recommend building 3 RAID 1 arrays and adding those to a LV.
The reason: The next time you are going to increase your storage capacity, you don't need to replace all 6 disks. Remember that RAID 5 needs identical storage capacity of all drives. You can instead replace just one of your RAID 1 arrays.