17

I'm about to re-organise all my HDDs in my home linux box nas and would like to use mdadm raid for data protection and its flexibility for reshaping the arrays. However, before I use mdadm for this I'd like to know how it handles bit rot. Specifically the kinds of bit rot that do not result in unrecoverable read error messages being sent from the HDD.

Given that I'll likely be using at least 21TB of HDDs in 8 disks in the nas and the various quotes on probabilities of failures on HDDs, I'm thinking that during a rebuild from a single disk failure I'm reasonably likely to encounter some form of bit rot on the remaining disks. If it is an unrecoverable read error on 1 of the drives, that the drive actually reports it as an error, I believe that should be fine with raid6(is it?). However if the data read from the disk is bad but not reported as such by the disk, then I can't see how this can be automatically corrected even with raid6. Is this something we need to be concerned about? Given the article It is 2010 and RAID5 still works, and my own successful experiences at home and work, things are not necessarily as doom and gloom as the buzz words and marketing would have us believe, but I hate having to restore from backups just because a HDD failed.

Given that the usage patterns will be, write at most a few times, and read occasionally, I'll need to perform data scrubbing. I see on the archlinux wiki the mdadm commands for data scrubbing an array as

echo check > /sys/block/md0/md/sync_action

then to monitor the progress

cat /proc/mdstat

This seems to me that it will read all sectors of all disks and check that the data matches the parity and vice-versa. Though I notice there is heavy emphasis in the docs to say that there are significant circumstances that the "check" operation will not be able to auto correct, only detect, and it will leave it up to the user to fix.

What mdadm RAID level(s) should I choose to maximise my protection from bit rot and what maintenance and other protective steps should I be doing? And what will this not protect me from?

Edit: I'm not looking to start a RAID vs ZFS or any other technology QA. I want to know specifically about mdadm raid. That is also why I'm asking on Unix & Linux and not on SuperUser.

Edit: is the answer: mdadm can only correct URE's that are reported by the disk systems during a data scrub and detect silent bit rot during a scrub but cannot/will not fix it?

  • As far as data protection goes, the main benefit I see in zfs is it scrubs the disk locations of files whenever you read a file. This is why I currently have it setup with zfs. But I still need to perform regular full scrubs anyway. I have 2 zfs pools each with 3 disks, and I want to upgrade to an 8 disk system where any drive can fail and there will still be 1 more redundant drive and zfs is not flexible to allow a reshape like that. Since I'm rebuilding anyway I'm re-visiting mdadm. – BeowulfNode42 Dec 16 '13 at 11:48
  • You've been lucky with RAID5/6 so far. The fact is, it's 2013 and RAID still suffers from a write hole. If you lose power after data is written but before parity is written then you've just corrupted your good data and it's possible that with the inconsistency that your array is toast too. Thanks RAID5. – bahamat Dec 17 '13 at 7:58
  • The thing is, what you want to do is best done at the file system layer. Otherwise, you'd need some way to detect and preferably correct bit rot, possibly in a reduced- or no-redundancy situation, and RAID just isn't suited for that. Not only is there no guarantee that you won't end up with bit rot anyway (what if one drive fails and another reads the bit wrong off the platter?), but plain RAID also has no concept of what is important data and what is just noise. Since ZFS only scrubs referenced data, bit rot on an unused portion of the disk becomes a non-issue. – a CVn Dec 17 '13 at 8:15
  • Really, you can't expect layering a random file system on top of multiple disks (even with redundancy) to suddenly protect you against storage faults. I'm not on a holy crusade to bring ZFS to the masses (though I do think it is a great invention, and use it myself on Linux for basically everything but the root partition, which is ext4 on mdraid1 for software compatibility), but I also recognize that yours is one of the kind of problem ZFS was designed from the ground up to solve: guaranteed detection and if possible repair of data corruption regardless of cause. – a CVn Dec 17 '13 at 8:20
  • I think you should revise your requirements. Do you really need bitrot protection even for the case when error correction is applied? Do you know how unlikely is it for a bitrot to exist GIVEN that it was also corrected by disk's ECC? – caveman Mar 10 '16 at 2:57
5

Frankly, I find it rather surprising that you'd reject RAIDZ2 ZFS. It seems to suit your needs almost perfectly, except for the fact that it isn't Linux MD. I'm not on a crusade to bring ZFS to the masses, but the simple fact is that yours is one of the kinds of problems that ZFS was designed from the ground up to solve. Relying on RAID (any "regular" RAID) to provide error detection and correction possibly in a reduced- or no-redundancy situation seems risky. Even in situations where ZFS cannot correct a data error properly, it can at least detect the error and let you know that there is a problem, allowing you to take corrective action.

You don't have to do regular full scrubs with ZFS, although it is recommended practice. ZFS will verify that the data read from disk matches what was written as the data is being read, and in the case of a mismatch either (a) use redundancy to reconstruct the original data, or (b) report an I/O error to the application. Also, scrubbing is a low-priority, online operation, quite different from a file system check in most file systems which can be both high-priority and offline. If you're running a scrub and something other than the scrub wants to do I/O, the scrub will take the back seat for the duration. A ZFS scrub takes the place of both a RAID scrub and a file system metadata and data integrity check, so is a lot more thorough than just scrubbing the RAID array to detect any bit rot (which doesn't tell you if the data makes any sense whatsoever, only that it's been written correctly by the RAID controller).

ZFS redundancy (RAIDZ, mirroring, ...) has the advantage that unused disk locations don't need to be checked for consistency during scrubs; only actual data is checked during scrubs, as the tools walk the allocation block chain. This is the same as with a non-redundant pool. For "regular" RAID, all data (including any unused locations on disk) must be checked because the RAID controller (whether hardware or software) has no idea what data is actually relevant.

By using RAIDZ2 vdevs, any two constituent drives can fail before you are at risk of actual data loss from another drive failure, as you have two drives' worth of redundancy. This is essentially the same as RAID6.

In ZFS all data, both user data and metadata, is checksummed (except if you choose not to, but that is recommended against), and these checksums are used to confirm that the data hasn't changed for any reason. Again, if a checksum does not match the expected value, the data will either be transparently reconstructed or an I/O error will be reported. If an I/O error is reported, or a scrub identifies a file with corruption, you will know for a fact that the data in that file is potentially corrupted and can restore that specific file from backup; no need for a full array restore.

Plain, even double-parity, RAID doesn't protect you against situations like for example when one drive fails and one more reads the data incorrectly off the disk. Suppose one drive has failed and there's a single bit flip anywhere from any one of the other drives: suddenly, you've got undetected corruption, and unless you're happy with that you'll need a way to at least detect it. The way to mitigate that risk is to checksum each block on disk and make sure the checksum cannot be corrupted along with the data (protecting against errors like high-fly writes, orphan writes, writes to incorrect locations on disk, etc.), which is exactly what ZFS does as long as checksumming is enabled.

The only real downside is that you cannot easily grow a RAIDZ vdev by adding devices to it. There are workarounds for that, usually involving things like sparse files as devices in a vdev, and very often termed "I wouldn't do this if it was my data". Hence, if you go a RAIDZ route (regardless of whether you go with RAIDZ, RAIDZ2 or RAIDZ3), you need to decide up front how many drives you want in each vdev. Although the number of drives in a vdev is fixed, you can grow a vdev by gradually (making sure to stay within the redundancy threshold of the vdev) replacing the drives with larger-capacity ones and allowing a complete resilver.

  • 5
    In my original question I was trying to avoid the zfs vs raid argument as there is lots of info on that. I want specific info about mdadm. Also since I will not be reading all of the data often enough to ensure that data is scrubbed regularly, I will need to force a full array scrub regularly regardless of zfs or raid. – BeowulfNode42 Dec 16 '13 at 23:34
  • @BeowulfNode42 personally I suggest using application layer checksums for exceptionally important data (e.g. use sha256 to checksum your important data). ZFS can do this per block which I think is really an overkill. I think this explains why not much file systems checksum their blocks like ZFS does because IMO this is more of an application layer problem in my view. – caveman Mar 10 '16 at 20:36
  • 1
    @caveman I don't know about you; I really like the fact that I don't have to constantly checksum files just to be certain that they haven't been corrupted. Sure, the huge majority of the time there is no corruption, in which case no harm's done (with ZFS, you get your pick of checksum algorithm among a handful, so you can pick your preferred point along the security/performance continuum), but automated file system level checksums guarantees that there is no uncorrected corruption because if there is, you will know about it, in ZFS' case by receiving an I/O error instead of corrupted data. – a CVn Mar 10 '16 at 21:12
  • @MichaelKjörling nope it doesn't "guarantee" (only reduces the probability of undetected errors relative to disk-only checks, by an amount that no one has quantified yet! therefore no one really knows how useful ZFS's checksuming is :)), plus you can use a simple "read" and "write" wrappers that transparently do the checksumming for you. One doesn't need to put this fancy thing into the kernel space. – caveman Mar 10 '16 at 23:01
  • 3
    @caveman no, zfs is not on topic. Neither are possible implementations of RAID that are not mdadm. I want to know about mdadm. I have already down voted this answer as much as I can and your comments on an off topic answer filling in more information about the off topic answer is not helping with the original question. – BeowulfNode42 Jul 6 '16 at 5:04
3

This answer is the product of reasoning based on the various bits of evidence I've found. I don't know how the kernel Linux implementation works, as I am not a kernel dev and there seems to be a fair amount of nonsensical misinformation out there. I presume that the kernel Linux makes sane choices. My answer should apply unless I am mistaken.

Many drives use ECCs (error-correcting codes) to detect read errors. If data is corrupt, the kernel should receive a URE (unrecoverable read error) for that block from an ECC supporting drive. Under these circumstances (and there is an exception below), copying corrupt, or empty, data over good data would amount to insanity. In this situation the kernel should know which is good data and which is bad data. According to the It is 2010 and RAID5 still works … article:

Consider this alternative, that I know to be used by at least a couple of array vendors. When a drive in a RAID volume reports a URE, the array controller increments a count and satisfies the I/O by rebuilding the block from parity. It then performs a rewrite on the disk that reported the URE (potentially with verify) and if the sector is bad, the microcode will remap and all will be well.

However, now for the exception: if a drive does not support ECC, a drive lies about data corruption, or the firmware is particularly disfunctional, then a URE may not be reported, and corrupted data would be given to the kernel. In the case of mismatching data: it seems that if you are using a 2 disk RAID1, or a RAID5, then the kernel can't know which data is correct, even when in a non-degraded state, because there is only one parity block and there was no reported URE. In a 3 disk RAID1 or a RAID6, a single corrupted non-URE-flagged block would not match the redundant parity (in combination with the other associated blocks), so proper automatic recovery should be possible.

The moral of the story is: use drives with ECC. Unfortunately not all drives that support ECC advertise this feature. On the other hand, be careful: I know someone who used cheap SSDs in a 2 disk RAID1 (or a 2 copy RAID10). One of the drives returned random corrupted data on each read of a particular sector. The corrupted data was automatically copied over the correct data. If the SSD used ECCs, and was properly functioning, then the kernel should have taken proper corrective action.

  • 1
    I thought all modern HDD have some form of internal ECC. Whether or not it is effective, correct, or malfunctioning is another matter. ECC has to be used internally in the drive to be able to report an URE. Silent bit rot, that I'm most interested in, does not report a URE even on drives that support it, as they think they have the correct data, when they don't. – BeowulfNode42 Oct 5 '16 at 0:07
  • By bit rot, I assume you mean bits randomly flipping. In any case the ECC is designed to detect flipped bits. According to Wikipedia, Reed–Solomon error correction is a common ECC format invented in 1960 and is still used in Blu-Ray disks + HDDs. If you discover that that algorithm is extremely reliable, then your question should be pretty much answered, as decent modern hardware, by definition, is just as good, if not better, even if you don't know a piece of hardware's decency just by looking at it. – sudoman Oct 5 '16 at 1:07
  • 1
    Bit rot can also occur due to other problems such as when some issue causes the drive heads to not be aligned properly to where it thinks it is writing and it spilling over to nearby sectors. It may fix the sector it was intending to work on, but the nearby sector will be damaged. If it happens to have written over the data+ecc in such a way that the ECC for the nearby sector reports as being fine then the drive will never know it has a problem. Much more likely, some rogue software instructs the drive to write bad data, the hdd will faithfully store that bad data. eg a bad dd command – BeowulfNode42 Oct 5 '16 at 1:26
2

For the protection you want, I'd go with RAID6 + the normal offsite backup in 2 locations.

I personally scrub once a week anyway, and backup nightly, weekly and monthly depending on the data importance and change speed.

  • 1
    but what bit rot detection / correction capabilities does that offer? – BeowulfNode42 Mar 24 '15 at 22:10
  • 1
    RAID6 with frequent scrubbing offers some bit-rot protection, as the double parity effectively creates three versions of the same block, so a "voting" can be held on which version is right. AFAIK, RAID6 scrubbing in linux dm-raid does just that, please correct me if I'm wrong. – P.Péter Oct 7 '15 at 13:34
  • 1
    @P.Péter I realise that the math involved COULD use a voting system, but does mdadm? Do you know of any documentation about this or have had personal experience that has led you to this conclusion. Particularly in light of Ethan's answer. – BeowulfNode42 Jul 6 '16 at 5:07
  • This was some time ago, but I vaguely remember reading up on mdadm RAID6 mechanisms before commenting. Sorry, not very specific. :( I guess we could use a real expert on mdadm... – P.Péter Jul 7 '16 at 10:32
2

I don't have enough rep to comment, but I want to point out that the mdadm system in Linux DOES NOT correct any errors. If you tell it to "fix" errors during a scrub of, say, RAID6, if there is an inconsistency, it will "fix" it by assuming the data portions are correct and recalculating the parity.

  • 1
    This seems rather unlikely, unless I misunderstand you. Do you mean that data from corrupted blocks often gets copied over correct blocks? This would require that the bad block doesn't come from a drive that supports ECC (and thus would not report a URE), and that you are using RAID5 or 2 copy RAID1 (instead of RAID6 as you suggested.) – sudoman Oct 4 '16 at 18:57
  • @sudoman, during a scrub, if the Linux MD subsystem detects a mismatch between the data and the parity, it blindly assumes that the parity is wrong and re-writes it based on the data. It is possible to use the double-parity of RAID 6 to figure out which is wrong, but the Linux MD subsystem does not do this. – Mark Oct 17 '17 at 22:52
  • 1
    Ethan, I don't suppose you have any references for this information? or examples of personal experience you're willing to share what you remember? Given the tumbleweeds that this Q has generated, even anecdotal info would be helpful. Since this Q was posted I've had some problems with mdadm RAID1 for the boot drive, on (cheap) usb sticks when 1 of them went bad. Some investigation later points to that failing usb stick not having enough or any error checking, or it was just failing to write data to some blocks and not producing a write error. I had to reinstall the OS. – BeowulfNode42 Jun 26 at 4:55
-2

bit rot fud.? sure...

I guess you need to talk to SEAGATE. (forget to? is that the excuse)? the drives now all have 100bit ECC correction you need to prove the rot first.
I bet you can't. (it's FUD thing to worry right?) like fear of ghosts or the #13? and not done here. zero proof happened. and worse no proof of cause.

First define what bit rot means.? ouch... HDD: ECC checks the data (even 1 bit) against the ECC 100 bit storage. if it's wrong, it corrects it, if it keeps failing the SMART engine, for sure on SAS drives, it logically replaces the cluster or sector with one that is good. using spare clusters. this repairs the damage. Yes all drives grow bad bits from day one to end, from IBM first drives to NOW. but now we do self repair, Read the full Seagate white papers. endless there, and learn how a drive works. ok?

this keeps going until you run out of spares,(hdd brain, smart) and then SMART screams END OF LIFE. (or even more early, like HP does) on say a HP P420 controller, it watches this all the time. Mine even emails me, showing NEAR OUT OF SPARE clusters. Sometime the spares go way faster, a sure sign of doom soon, (10 years old sas sure,less in junky sata.

I call BOGUS, and FUD on bit rot.

My guess is someones toy PC wrote the data wrong , for what ever reasons. not running ECC memory?? oops , real servers have ECC RAM. virus infected.? or lost power during the write (no UPS>?)? or has bad memory.? or ESD damaged. Or PSU making tons of noise (bad)

I call FUD here. sorry,

  • 1
    I've just clarified I was talking about my home system, so ECC and server grade hardware is out of my budget price range. My home lab is much more prone to unexpected power loss even with its mini ups, or other random events, like the tower falling over or something. There are plenty of other ways for a HDD to be told to store the wrong data and have the HDD store the ECC bits for that wrong data. I don't care how errors occurred, I want them easily fixed. – BeowulfNode42 May 9 '18 at 12:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.