8

I was recently told about some problems concerning Parity-RAIDs without a non-volatile cache. More exspensive HW-controllers do have battery-powered caches to finish write-operations in case of power failure. Now, some people say that such a failure, perhaps in combination with a degraded array, may kill your whole filesystem. Others claim that those issues are outdated and/or misconceptions.

Unfortunately, nobody gives hard references and neither a search for md RAID and non-volatile cache, nor for bitmap caching gives reliable answers about if md-RAID5 is advisable or not.

Any information about that?

Improve this question
4
  • 1
    Linux's mdraid does have a journal, which at least is safe in case of sudden halt (i.e. system crash, or all disks power down at once), or in the case of RAID-1. I don't know if its RAID-[56] copes with all forms of staged power down.
    – Gilles 'SO- stop being evil'
    Jul 11, 2011 at 13:49
  • 2
    @Gilles, it does not have a journal, but it does have an optional write-intent bitmap. This just speeds up the process of resynchronization when the system comes back up though, because it identifies what areas need resynchronized, and what areas can be skipped.
    – psusi
    Jul 11, 2011 at 18:26
  • @psusi I went back and checked what I'd researched a few months back. As far as I understand, the write-intent bitmap (not a journal, my mistake) does more than speed up resynchronization, it indicates which of the components have a dirty block, so that's enough to ensure block consistency on RAID-1 but not on RAID-5 (where you might not have enough disks in both the old and the new state to restore either state).
    – Gilles 'SO- stop being evil'
    Jul 11, 2011 at 22:19
  • 1
    @Gilles I'm not sure what you mean by "bock consistency". You seem to be hung up on the entire stripe being in either the state before the write started, or the state after. This is never guaranteed and is entirely likely not to be the case if the power fails in the middle of the write. It is up to the filesystem to handle this just like it does on a single disk; using the journal.
    – psusi
    Jul 12, 2011 at 13:59

6 Answers 6

Reset to default
4

I assume Linux's software RAID is as reliable as a hardware RAID card without a BBU and with write-back caching enabled. After all, uncommitted data in a software RAID system resides in the kernel's buffer cache, which is a form of write-back caching without battery backup.

Since every hardware RAID-5 card I have ever used allows you to enable write-back caching without having a BBU, I expect software RAID-5 can work okay for people with a certain level of risk tolerance.

ObWarStory:

That having been said, I have personally experienced serious data loss due to having no BBU installed on a RAID-5 card though write-back caching was enabled. (No UPS, either. Don't yell at me, not my call.)

My boss called me in a panic while I was on vacation because one of our production systems wouldn't come back up after a power outage. He'd run out of things to try. I had to pull off to the side of the road, pull out the laptop, turn on WiFi tethering on my phone, ssh into the stricken system, and fix it, while my family sat there with me on the side of the road until I finished restoring a roached database table from backup. (We were about a mile away from losing cell reception at the time.)

So tell me: how much would you pay for a RAID card + BBU now?

Improve this answer
2
  • Right - I deleted my comment. But a raid without BBU should write through, shouldn't it? This is at least what the PERC-controllers do when the battery learns and falls below threshold.
    – Nils
    Oct 3, 2011 at 21:29
  • Yes, without a BBU or with a dead BBU, the RAID card still writes data. What it doesn't do is remember what was in the write buffer when power fails to the server. Since RAID depends on consistency among the redundant bits, it gets confused when it becomes inconsistent. Therefore, a power failure during RAID writes risks corrupting something on the RAID, because the controller is forced to pick one of the two-or-more copies of the data, not knowing which is correct.
    – Warren Young
    Oct 3, 2011 at 22:13
3

Just a warning notice : RAID-5/6 write operations take a significant CPU time while your array is degraded. If your server is already fully loaded when a disk comes to fail, it may drop into an abyss of unresponsiveness. Such problem won't happen with a hardware RAID controller. So I'd strongly advise against using software RAID-5/6 on a production server. For a workstation or lightly loaded server, it's OK though.

Improve this answer
3

SW RAID does have a failure mode - if the server goes down halfway through a write you can get a corrupted stripe. A HW RAID controller with a BBU isn't all that expensive, and it will retain dirty blocks until you can restart the disks.

The BBU on the cache does not guarantee writes in the event of power failure (i.e. it does not power the disks). It powers the cache for a few days until you can re-start the disks. Then the controller will flush any dirty buffers to disk.

Some notes about SW vs. HW RAID-5

  1. Writes on a SW RAID-5 volume can be slow if write-through caching is used with blocking I/O, as the call doesn't return until all the I/O has completed. A HW RAID controller with a BBWC can optimise this considerably, so you can see substantially better performance.

  2. The last time I looked you couldn't do direct I/O (i.e. zero-copy DMA) on a SW RAID volume. This may have changed and is really only relevant to applications like database managers using raw partitions.

  3. A modern SAS RAID controller can pull or push 1GB/sec or more of data off a disk array, particularly if formatted with a large (say 256kb) stripe size. I've even benchmarked an older Adaptec ASR-2200s at speeds that indicated it was pretty much saturating both its scsi channels at 600MB/sec+ in aggregate (10x 15k disks) with very little CPU load on the host machine. I'm not sure you could get that out of software RAID-5 without a lot of CPU load if at all, even on a modern machine. Maybe you could read that quickly.

  4. Configuration for booting off a HW RAID volume is simple - the RAID volume is transparent to the O/S.

A low-end RAID controller from a tier-1 vendor such as adaptec is not that expensive at retail street prices and can be purchased for peanuts off ebay. But remember, if you buy secondhand, stick to tier-1 vendors and make sure you know the model and verify the avialability of drivers from their web site.

Edit: From @psusi's comment, make sure you don't get a fakeraid (transparent SW RAID hidden in the driver) controller, but most of the offerings from the bigger names (Adaptec, 3Ware or LSI) aren't fakeraid units. Anything that can take a BBU won't be fakeraid.

Improve this answer
4
  • If power fails in the middle of a write, then you get a stripe that is out of sync not corrupted. An out of sync stripe just means that the parity is not up to date, so when the array is mounted, the parity must be updated. Also those "raid" controllers that can be had for peanuts are often fakeraid; they have bios rom extensions and windows drivers that do the raid in software.
    – psusi
    Oct 14, 2011 at 19:54
  • @psusi - Most of the ASR-2200s controllers I bought a few years ago were under 100 USD and they're pukka HW RAID controllers. I don't think Adaptec actually make fakeraid controllers at all. You can quite readily get 4 or 8 port Adaptec, 3Ware or LSI SAS RAID controllers off ebay for a few hundred dollars.
    – ConcernedOfTunbridgeWells
    Oct 15, 2011 at 19:12
  • I wouldn't call a few hundred bucks for a used product from an unknown source "pennies"; that indicates more along the line of $50-$100 for a new product. Devices in that class are usually fakeraid.
    – psusi
    Oct 17, 2011 at 13:25
  • @psusi - You are trying to rebuff an argument that I never made; I never used the word 'pennies' at all. Please do not resort to straw man arguments - the examples I used are not fakeraid controllers.
    – ConcernedOfTunbridgeWells
    Oct 17, 2011 at 21:52
1

If you got data in cache but not on the disk yet, and power fails, then the data is going to disappear, and your disk is most likely going to be in an inconsistent state. The probability of that isn't very high unless you got a system that's constantly writing, but I still wouldn't want to bet my data on probability games.

An interesting twist would be to make a main filesystem on RAID5/6 but put a journal on a regular drive, so the data is first dumped on the regular drive. The performance would probably go to the crapper as you'd be limited to the write speed of a single drive, but the reliability would go up. So I guess in a situation where your write performance isn't important, but your read is, that might work just fine.

Or you could just spend another 100$ and get the card with BBU, or a small UPS, and avoid all these complications altogether ;)

Improve this answer
5
  • What do you think about a journal on a fast SSD?
    – user773568
    Jul 12, 2011 at 10:31
  • That'd do it, but at that moment you're paying more than a decent controller in the first place ;) Also, speed goes up, but the reliability goes down, because most SSD's die very very quickly.
    – Marcin
    Jul 12, 2011 at 11:55
  • 1
    @Marcin what do you base that on? They don't seem to have shorter specified design lifetime, and I have had one for over a year and have only used 5% of its write cycles.
    – psusi
    Jul 12, 2011 at 13:51
  • @MarcinWell When you say reliability, is that savety or availability? I did not expect that loosing my journal threatens the overall data. Does it? Anyway I plan to put OS and Swap on a smaller SSD, due to noise and power-saving reasons. The RAID can go to sleep that way.
    – user773568
    Jul 12, 2011 at 15:37
  • MLC SSDs have a track record of reliability issues. SLC units are much more reliable, but also much more expensive. A white paper about SSD reliability can be found here
    – ConcernedOfTunbridgeWells
    Oct 17, 2011 at 22:00
1

Linux mdadm software raid is designed to be just as reliable as a hardware raid with battery backed cache. There are no problems with sudden loss of power, beyond those that also apply to sudden power loss on a single disk.

When the system comes back up after power fail, the array will be resynchronized, which basically means that the parity is recomputed to match the data that was written before the power failure. It takes some time, but really, no big deal. The resynchronize time can be greatly reduced by enabling the write-intent bitmap.

Improve this answer
18
  • 1
    That sounds a bit optimistic. How can a pure software solution be as reliable as a battery backed cache?
    – user773568
    Jul 12, 2011 at 10:32
  • 1
    There are bad things that can happen to a RAID array that can't happen to a single disk. With a single disk, every sector is in either the old or the new state. With e.g. RAID-5 over 4+1 disks, what if sector 42 of disks 1 and 2 are in the old state and sector 42 of disks 3, 4 and 5 are in the new state? Neither the old state nor the new state are recoverable. I don't know if Linux takes measures to avoid this, and this is what the question is about.
    – Gilles 'SO- stop being evil'
    Jul 12, 2011 at 11:23
  • @user773568 umm... I just explained how?
    – psusi
    Jul 12, 2011 at 13:46
  • @Gilles you have just restated the same case as the single disk. Some sectors are in the old state, and some are in the new state. It doesn't matter which disk they are on. Filesystems deal with incomplete writes during a crash with the journal.
    – psusi
    Jul 12, 2011 at 13:48
  • @psusi No: with a single disk, each sector is in either the new state or the old state. With multiple disks, if the driver uses the naive approach of overwriting the sector on each disk without storing information elsewhere, a sector that was in a transitory state (old state on some disks, new state on others) cannot be recovered at all. The error can possibly be detected (if you're lucky: the parity could match by accident), but it cannot be corrected.
    – Gilles 'SO- stop being evil'
    Jul 12, 2011 at 14:05
1

Here is a blog explaining the issue with RAID5 and how ZFS RAIDZ is resolving it.

Its key points are :

RAID-5 (and other data/parity schemes such as RAID-4, RAID-6, even-odd, and Row Diagonal Parity) never quite delivered on the RAID promise -- and can't -- due to a fatal flaw known as the RAID-5 write hole. Whenever you update the data in a RAID stripe you must also update the parity, so that all disks XOR to zero -- it's that equation that allows you to reconstruct data when a disk fails. The problem is that there's no way to update two or more disks atomically, so RAID stripes can become damaged during a crash or power outage.

and

RAID-Z is a data/parity scheme like RAID-5, but it uses dynamic stripe width. Every block is its own RAID-Z stripe, regardless of blocksize. This means that every RAID-Z write is a full-stripe write. This, when combined with the copy-on-write transactional semantics of ZFS, completely eliminates the RAID write hole.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .