BTRFS data integrity (CRC32c) and HDD data integrity (sector ECC)

Question

I'm new to BTRFS and I'm trying to understand why BTRFS use CRC32c while HDD already have sector level data integrity ECC. Is it because BTRFS don't relay on media to have data integrity protection? Thank you.

Answer 1

Disks can and do silently corrupt data. See http://storagemojo.com/2007/09/19/cerns-data-corruption-research/ for just one example of research on this.

Answer 2

I just don't buy such arguments that disks regularly have unreported errors and chalk it up to FUD. Yes, if you throw enough random data at the error detecting code, it will sometimes report the data is correct when it isn't. Here's the thing though: the drive isn't trying to read random data. It is reading data that has mostly been written and read back correctly. That then passes through an error correcting code that can fix a number of errant bits. To get an unreported error you have to get a much higher than usual number of raw errors to overwhelm the ECC, and then they have to be arranged just right so that the output of the ECC is itself arranged just right that it fools the EDC into thinking it is good. The odds are much higher that at least the EDC will notice the error and report it as an uncorrectable error. How often does that happen? Basically never unless a drive is approaching failure or had a sudden power loss during a write. So if an uncorrectable error almost never happens, and an unreported error is a million times less likely, what does that tell you?

On the other hand, if you are storing a duplicate copy of your data anyhow, it is probably nice to have some way of telling which one is correct in the highly unlikely event that one copy does become silently corrupt. Also the crc is useful for detecting blocks that happen to contain duplicate copies of the same data, so they can be deduplicated, which is another design feature of btrfs.

Answer 3

Yes, it doesn't trust the device to report errors or to store the correct data in the first place. Whether this is actually necessary is another question entirely. It's not something anyone worries about, usually, and things just work.

If you have a disk that does not report errors, you have a huge problem anyway; it's not only filesystems that rely on such error reporting, but also other components such as RAID controllers etc.; unreliable storage puts your entire data at risk, not just a few bits.

Regardless whether your filesystem does checksumming, you should always run your own tests on storage; such as SMART selftests, or in case of RAID, check for mismatches in the parity data (/sys/block/mdX/md/mismatch_cnt=0 after running a check sync_action).

Answer 4

btrfs is a next-gen filesystem - it encompasses many of the same purposes as past layering models handled between them. btrfs is a formiddably extensive stack as well - the faq recommends it be written out to an unpartitioned disk*[s]* and that all partitioning, quotas, compression, imaging, striping, copy-on-write, deduplication, and probably 10 other things I'm forgetting be handled as qualities of thefilesystem alone. It can do all of these things and many more.

btrfs disk arrays are dynamic - they can be added to and deleted from on a live system without issue. This works because btrfs chunks out storage block groups only when it wants them - and they might be on any particular device in its current array when it does. The FAQ has some stuff to say about this - particularly where it talks about the unreliability of free-space estimates:

For example, if you have one subvolume as "single", and one as RAID-1, then the first subvolume will consume raw storage at the rate of one byte for each byte of data written. The second subvolume will take two bytes of raw data for each byte of data written. So, if we have 30GiB of raw space available, we could store 30GiB of data on the first subvolume, or 15GiB of data on the second, and there is no way of knowing which it will be until the user writes that data.

So, in general, it is impossible to give an accurate estimate of the amount of free space on any btrfs filesystem. Yes, this sucks. If you have a really good idea for how to make it simple for users to understand how much space they've got left, please do let us know, but also please be aware that the finest minds in btrfs development have been thinking about this problem for at least a couple of years, and we haven't found a simple solution yet.

Reading the relevant section will offer you more specific examples, but it makes it very clear that btrfs devices can be variable in number, epehemeral in persistence, blocked and striped separately or together and... well, it goes on. Another quote from the FAQ:

Device management is a complex subject, and there are many different opinions about the best way to do it. Internally, the Btrfs code separates out components that deal with device management and maintains its own layers for them. The vast majority of filesystem metadata has no idea there are multiple devices involved.

It says this about RAID:

btrfs supports RAID-0, RAID-1, and RAID-10. As of Linux 3.9, btrfs also supports RAID-5 and RAID-6 although that code is still experimental.

btrfs combines all the devices into a storage pool first, and then duplicates the chunks as file data is created. RAID-1 is defined currently as "2 copies of all the data on different devices". This differs from MD-RAID and dmraid, in that those make exactly n copies for n devices. In a btrfs RAID-1 on three 1 TB devices we get 1.5 TB of usable data. Because each block is only copied to 2 devices, writing a given block only requires exactly 2 devices to be written to; reading can be made from only one.

Data recovery:

The advantage in btrfs-raid 5/6 is that unlike MD-RAID, btrfs knows what blocks are actually used by data/metadata, and can use that information in a rebuild/recovery situation to only sync/rebuild the actually used blocks on a re-added or replacement device, skipping blocks that were entirely unused/empty in the first place.

MD-RAID can't do that, because it tries to be a filesystem agnostic layer that doesn't know nor care what blocks on the layers above it were actually used or empty. For it to try to track that would be a layering violation and would seriously complicate the code and/or limit usage to only those filesystems or other layers above that it supported/understood/could-properly-track.

Of course, btrfs is designed from the ground up to transcend layers. In order to this it must maintain a checksummed, rebuildable, and hopefully at least somewhat redundant tree which comprises all of its currently incorporated devices. btrfs is, in many ways, a file database as well as a file-system. It does not rely on underlying devices for ecc because, in large part, it does not consider that there are underlying devices. You might think of it like a disk kudzu, maybe.

In any case, it is precisely the constant checksumming and metadata management that enable btrfs to do so many of the interesting things that it does, and to do so without much regard at all for its underlying hardware.