2

I have a server with about 6TB of media files on a single 8TB WD hdd.

Before I ask a question I probably should provide some background.

These files were on BTRFS for a few weeks but after an unrelated hardware issue and subsequent OS rebuild I accidentally trashed the disk & had to restore the files from a backup, so I decided to use that as an opportunity to try out ZFS instead.

The main reason that I want to use zfs is for its ability to maintain data integrity. Before I moved to btrfs (and now zfs) I had these files on ext4 and after a drive developed a bitrot issue I had a bunch of files get silently corrupted.

So, after reloading the data onto zfs things were OK for a few weeks until I noticed in the morning today that the disk was being flogged relentlessly.

After a bit of poking around I found that it was being "scrubbed" by zfs at the blindingly fast rate of 586K/s. At that rate it will never complete!

Now part of this process is me getting more familiar with ZFS so if I am misunderstanding something here please let me know but I believe that the scrub is needed for data integrity purposes because the whole dataset is stored on a single physical disk?

If this is correct will the flogging problem be solved if I were to buy more disks and use some form of raidz?

If so what would be the best way to solve this problem?

1) Buy 2 x 4TB disks and use non-redundant striping? (cheapest)

2) Buy 3 x 4TB disks and use redundant stripe? (more expensive)

3) Buy a second 8TB disk and mirror it? (most expensive)

Bear in mind I don't really require the redundancy of options 2 & 3 (availability) & I am more interested in maintaining the data (integrity) without having the disk constantly flogging doing scrubs.

System details: intel i3 6100T 16Gb RAM 8TB WD Red Ubuntu 16.04 (on a separate SSD)

zfs compression and dedup are turned off (they were turned on at first but I have since turned them off)

Thanks for reading

1

1 Answer 1

3

I believe that the scrub is needed for data integrity purposes because the whole dataset is stored on a single physical disk?

A ZFS scrub is only needed for data sets where you do not access all of the important files in that data set often enough for ZFS's normal file checking mechanisms to notice problems fast enough. If you were choosing to scrub once a week but your normal pattern of usage read or modified all the important data blocks once a day, the scrub is redundant.

For cases like yours, where some files may not be accessed for years but then need to be there when they finally do get accessed, periodic scrubs are indeed a good idea.

Now, that having been said, all a scrub will do with a single-disk pool is mark files with corrupted blocks as inaccessible. That's right: if ZFS detects even a single bit error in such a file, it will refuse to let you read that entire file. ZFS only delivers data it knows is not corrupt.

This is why you want to use ZFS with some amount of redundant storage, if you possibly can. If you give ZFS a mirror to work with, then the same block in both mirrors has to be corrupt to get the same outcome.

ZFS will still let you recover manually from such a situation on a single-disk pool: restore that file from backup. (Which hopefully is also similarly checksummed, so you know you're restoring an uncorrupted version of the file.) Because ZFS is a copy-on-write filesystem, the restored file is stored in unused data blocks and the old version is removed, including its corrupted data block(s).

will the flogging problem be solved if I were to buy more disks and use some form of raidz?

A scrub will take approximately the same time as reading the entire pool, if the pool is idle. For such a large disk, I'd expect it to take something like 8 to 24 hours.

If it's taking much longer than that, the pool must not be idle. If you thought the pool was idle, then you probably have some background task that's frequently interrupting the scrub, preventing ZFS from getting its low-priority background scrub job done quickly.

I get the sense that this is some kind of home media server, so try this experiment: start the scrub just before you go to bed, shut the machine down to single user mode, and then run this command:

# watch -n 120 zpool status

Take note of how long it says the scrub will take, then go to bed. When you get up in the morning after N hours of sleep, see if more or less than N hours have been taken off that estimate. I suspect it will have dropped by quite a bit more than N, because now the system has little else to do but scrub that disk.

In that case, you'd want to find out which background process that runs in multi-user mode keeps jiggling ZFS's elbow, preventing it from getting useful amounts of work done.

As to your actual question, no, buying more disks won't make the scrub go faster. It will just give ZFS more chances to automatically recover from data corruption events. No matter how many disks you give it, ZFS must still read all of the data blocks on disk in order to do the scrub, and that takes many hours with today's multiterabyte disks.

Beware: adding disks also lowers the MTBF for the whole pool, but that's the tradeoff: more money spent both up front and over the long haul to recover from errors without having to resort to backups. ZFS is not a backup, but with redundant storage, it may reduce the number of times you need to go back to that backup for a file.

(Backups protect from more than just storage medium failure: accidental deletions, malware, off-site disaster recovery, etc. You still need backups with ZFS.)

Buy 2 x 4TB disks and use non-redundant striping? (cheapest)

That's half the MTBF for no advantage except speed. Do this only on pools where you can afford to lose the whole pool.

Buy 3 x 4TB disks and use redundant stripe? (more expensive)

Given that replacing a drive in a modern multi-terabyte pool can take many hours, I recommend using at least dual redundancy, which means you'd want four disks and raidz2. With a simple 2-way mirror, if one disk goes down, you've got no redundancy if you repair it by directly replacing the bad disk. You've also got no redundancy while you wait for the replacement to arrive. (If you had the replacement on hand already, you'd have made it a spare in the pool, right? Right?)

There's only one limited exception to that, which is when you have a free slot in your storage system, ZFS mirrors let you add the replacement to the N-way mirror, temporarily making it an N+1-way mirror. The ZFS rebuild process then has the ability to the good blocks on the bad disk to check against any bad blocks on the disk you think is still good. This only works with disks giving soft errors, though. If a disk goes down hard, you're back to N-1 redundancy, which is no redundancy if N was 2.

Buy a second 8TB disk and mirror it? (most expensive)

Of your three options, that's the one I'd choose, since expanding ZFS pools by sets of mirrors is the easiest option. It gives you the option of moving to 3-way mirrors easily, whereas it is not possible to move from raidz1 to raidz2 without rebuilding the whole pool.

I don't really require the redundancy of options 2 & 3 (availability) & I am more interested in maintaining the data (integrity)

That seems like a distinction without a difference unless you have 100% confidence in your backups. If there is a chance that your backups have bitrotted since you took them, or you didn't back up the one file you need back, your unavailable data has zero remaining integrity.

Frankly, I recommend backing up redundant ZFS pools with another redundant ZFS pool, else how can you trust the backup?

Yeah, it gets expensive. Now you get to decide how badly you want to have 8TB of stuff attached to your computer.

1
  • Thanks very much for that Warren. I wasn't aware that the scrubbing is required for infrequently access files so I will file that away in my knowledge bank. You are right that this is just a huge repository of media files for various streaming devices in my house. Like most of us I use my home network to trial and learn things that I can't do at my day job. If I lost some of the data in this scenario then it would be a pain but not the end of the world. I am on a bit of a budget and don't want to spend a thousand bucks on hd's just for my home stuff.
    – brettg
    Dec 12, 2016 at 20:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .