bcache on md or md on bcache

Question

bcache allows one or more fast disk drives such as flash-based solid state drives (SSDs) to act as a cache for one or more slower hard disk drives.

If I understand correctly,

• an SSD^* could be assigned to cache multiple backing HDDs, and then the resulting cached devices could be RAIDed with mdadm
  or
• multiple HDDs could be RAIDed into a single backing md device and the SSD assigned to cache that

I'm wondering which is the saner approach. It occurs to me that growing a RAID5/6 may be simpler with one or other technique, but I'm not sure which!

Are there good reasons (eg growing the backing storage or anything else) for choosing one approach over the other (for a large non-root filesystem containing VM backing files)?

---

_{^* by "an SSD" I mean some sort of redundant SSD device, eg a RAID1 of two physical SSDs}

Answer 1

I think caching the whole md device make most sense.

Putting bcache to cache the whole md device sacrifices the whole idea of having raid, because it introduces another single point of failure.

• OTH failurs of SSD disks are relatively rare, and bcache can be put into the writethrough/writearound mode (in contrast to the writeback mode), where there is no data stored only to the cache device, and failure of the cache doesn't kill the information in the raid makes it a relatively safe option.

• Other fact is that there is significant computational overhead of soft RAID-5; when caching each spinning raid member separately, computer still has to re-calculate all the parities, even on cache hits

• Obviously, you'd sacrifice some expensive ssd space, if you cache each spinning drive separately. - Unless you plan to use raided ssd cache.

• Both options relatively don't affect the time of growing process - although the option with spinning drives being cached separately has potential to be slower due to more bus traffic.

It is fast and relatively simple process to configure bcache to remove the ssd drive, when you need to replace it. Thanks to the blocks it should be possible to migrate the raid setup both ways on-place.

You should also remember, that at the moment most (all?) live-CD distributions don't support bcache, so you can't simply access your data with such tools regardless of the bcache-mdraid layout option you chose.

Answer 2

I'd think the sane approach is to cache the resulting MD device.

bcache is designed to pass-trough sequential reads and writes.

If you bcache each device separately, logically, several devices striping into a raided or stripped MD, will, from the bcache perspective, constantly be writing random blocks.

While a bcached MD volume will look as normal, writing files to the volume, rather then random blocks to several devices.

The entire point of hard and software raid is to do the striping of data in the backend so that the resulting filesystem looks like a normal volume.

This might not be correct (as bcache devs might be clever and account for that kind of situation), but the logical optimal thing to do is to cache volumes, rather then block devices.