4

So I've got two fresh (4TB) drives destined to serve as main storage for a desktop system. The primary requirement is redundancy and reliability. Secondary considerations are data read speed and maintainability especially in case of drive failure.

The file system of choice will be BTRFS for reasons outside the scope of this question.

What I can not decide on is, whether to create an MD raid 10 array in 'far 2' configuration with non-raided BTRFS on top or to use the raid 1 functionality included in BTRFS for both metadata and file content.

Either option will fulfil the redundancy requirement and the fact that BTRFS is included in the Linux kernel and not flagged as experimental reassures me that it will probably be reliable enough, escpecially in respect to potential data corruption (I am writing this as kernel 4.10.3 has just been released).

I have not been able to find any performance data so far that covers this setup specifically. While the administrative details are different for each setup I dont' think there is much difference in the complexity of maintaining the two - but I am saying this without experience with btrfs so far (but with md).

I currently favour md10 over btrfs raid 1 because it is raid 10 that can run on only two disks and has there a decided performance advantage over md1. However the underlying architecture of btrfs raid 1 is different to md1 as it only guarantees two copies of each dataset, even on setups with more than two disks, rather than a complete mirror of each disk.

The question is now if this freedom of data allocation has been implemented in a way that there are speed improvements in btrfs raid 1 over md1 that make it perform on par with md10? Are there configuration options I should consider?

A second question is, whether the integration of the raid architecture into the file system itself gives any benefits at all?

And then of course I would like to hear if any of my assumptions regarding redundancy, reliability and maintainability are incorrect.

4

As a general rule, MD will outperform BTRFS for the foreseeable future. BTRFS currently serializes writes to multiple devices, and only reads from one for a given read() call.

As far as data safety though, you're better off using BTRFS for replication than MD or LVM2. Because of the internal block checksumming done by BTRFS, it can tell when you make a read call whether or not the data is good and automatically fall back to the other copy if it isn't, and it will correctly select the good copy of a block on a per-block basis when scrubbing.

Now, all of that said, the best option I've found so far is actually to use BTRFS in raid1 mode on top of two MD or LVM2 RAID0 volumes. While this does not perform quite as well as ext4 or XFS on top of MD RAID10, it still performs far better than the BTRFS raid10 implementation or running a single BTRFS device on top of MD RAID10 on the same devices, and provides the same data safety guarantees. Unfortunately, that does require four devices.

4
  • 1
    WRT BTRFS RAID-10, please remember that it's not the RAID-10 we're used to. ANY second disk failure will destroy the file-system, whereas a 4-drive RAID-10 array will have a 66% chance of survival. – Peter Jan 11 '18 at 18:03
  • @Peter, I don't think that's true. Btrfs RAID 10 also requires 4 devices and guarantees that 2 volume members can fail without data loss, so long as it's not a mirrored pair that fail in a 4 device configuration. – Adrian Günter May 21 '18 at 19:05
  • 2
    @AdrianGünter Actually, no, Peter is correct given the current implementation. It does not guarantee surviving 2 failed devices because of how it handles the striping and replication (an exact explanation is too long for a comment). – Austin Hemmelgarn May 21 '18 at 19:45
  • 1
    @AustinHemmelgarn Seems I was very misinformed. After digging further I found the Sysadmin Guide which states: "Note that chunks within a RAID grouping are not necessarily always allocated to the same devices. This allows Btrfs to do data duplication on block devices with varying sizes, and still use as much of the raw space as possible. With RAID-1 and RAID-10, only two copies of each byte of data are written, regardless of how many block devices are actually in use on the filesystem." – Adrian Günter May 22 '18 at 1:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.