2

I am trying to get my personal filesystem in order

I want to tar everything and get it organized into proper directories

I have many HDDs and a few SSDs

I want to use Btrfs with it's checksum and redundancy features

The question I bring to you now is: How do I configure a data scrubbing daemon that automatically detects when I randomly lose a HDD sector causing an archive to not pass it's checksum verification, then to copy a backup onto another sector.

I'm under the impression that this backup can actually be on the same HDD. It's unlikely I'm going to lose an entire drive all at once with normal aging and wear. In the future I plan to expand my backup filesystem to multiple HDDs. But for now I'm interested in having duplicates scattered across multiple sectors of the same drive.

My end result is to never again have to deal with losing a random JPEG or text file

I do have multiple HDDs I can dedicate towards a single large redundant autocorrecting Btrfs filesystem. But I don't have multiple of the same sized drives

Please link me to some good reading material covering what I'm interested in here

  • The drives wouldn't need to be the same size to build a btrfs array. Just go to the btrfs wiki. – mikeserv Jul 5 '15 at 8:02
  • Yea, I am actually aware that Btrfs doesn't need same sized drives like a classic RAID1. However, wouldn't that limit my duplication benefits to the size of the smaller drive? What about "internal mirroring" where it duplicates the data on a single drive. – nope Jul 5 '15 at 8:09
  • @nope No, using different-sized drives doesn't necessarily limit the effective capacity to the size of the smaller drive unless you have exactly 2 drives. If you have more than 2 drives, some data can be on 1+2 while other data can be on 3+4 and still other data can be on 1+4, and so on. It can potentially balance out quite well. – Celada Jul 5 '15 at 8:18
  • What about having 2 btrfs partitions on the same physical drive. Then balancing the btrfs filesystem on the two partitions? Or is there a way to do this internal mirroring with a single partition? – nope Jul 5 '15 at 8:19
  • When you mkfs.btrfs you can tell it what kind of metadata/actual data duplication to apply, and, to some extent, where. do man mkfs.btrfs. And yes, the performance benefits are limited there, but not the duplication benefits. Besides, you can partition a disk to hold more than one member. You can also change this at will w/ a live fs with btrfs balance later on - but that can take awhile. – mikeserv Jul 5 '15 at 8:19
2

How do I configure a data scrubbing daemon that automatically detects when I randomly lose a HDD sector causing an archive to not pass it's checksum verification, then to copy a backup onto another sector.

It's built in to Btrfs. Try:

btrfs scrub start

It has never yet happened to me that an error was found, but I expect that if one is found then:

  • It would be reported in the kernel log output
  • It would be corrected if possible. That is, it would be corrected as long as the storage policy is not single.

Since you have multiple devices, you can use raid1 or raid5 or raid6 as your storage policy for both metadata and data. Be aware that raid5 and raid6 were introduced much more recently and may not be considered as stable (trustworthy) as the rest of Btrfs.

But I don't have multiple of the same sized drives

With Btrfs, that's perfectly OK, unlike with block-level RAID.

Using different-sized drives doesn't necessarily limit the effective capacity to the size of the smaller drive unless you have exactly 2 drives. If you have more than 2 drives, some data can be on 1+2 while other data can be on 3+4 and still other data can be on 1+4, and so on. It can potentially balance out quite well. If it gets out of balance over time (perhaps due to uneven data churn) you can just run btrfs balance later on – but that can take awhile.

Related: ideal btrfs storage scheme to incorporate external USB HDD as backup media

  • great answer! thank you - however, what about having data duplicated with a single drive? would i have to use 2 (or more) partitions and raid1 between them? – nope Jul 5 '15 at 10:25
  • Yes, you would have to do that since you would need the DUP policy for that (raid1 won't do it) and Btrfs only offers DUP for metadata, not data. – Celada Jul 5 '15 at 10:56
  • 1
    Note that running btrfs scrub start manually will not act like a daemon that detects errors in the future. You have to make sure to run a scrub periodically and check the results. Here's an example cronjob that would email you the results. – basic6 Nov 14 '15 at 13:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.