I have a server that I am running for myself and my friends. We host games using Ubuntu 18.04 LTS Server on the primary boot drive, and use a RAIDZ2 pool for storing backups of those games, our music, movies, etc.
Every week to two weeks I get a faulted pool and a lot of read/write errors.
me@server:/$ zpool status NAS pool: NAS state: ONLINE status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://zfsonlinux.org/msg/ZFS-8000-HC scan: scrub repaired 0B in 3h19m with 0 errors on Sun Aug 11 07:14:28 2019 config: NAME STATE READ WRITE CKSUM NAS ONLINE 0 511 0 raidz2-0 ONLINE 0 200 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 3 224 0 sdf ONLINE 12 225 0 sdg ONLINE 3 226 0 sdh ONLINE 3 227 0 spares sdb AVAIL
These errors never cause any data loss, and scrubbing never causes the pool to have to repair any bytes. I always have to restart the machine to get the pool to mount to the file system again. I've had this same pattern for months, now. Looking at this, this suggests to me that either one disk is bad (sdf), or in fact e-h are all on their way to failure and showing pre failure signs. Running disk self tests using S.M.A.R.T. always come back fine, showing no issues on the drives after I reset the machine and run tests. I assigned a hot-spare, hoping that it might be of use in case of a failure. At this point I am thinking I should replace drive sdf with sdb and see if this resolves the issue.
So, my question essentially is, when I see errors in a pool on multiple drives like this, is it always the case that all of the drives are pre-failure, or can the redundancy algorithm cause one bad disk to "spread" errors across other drives?
EDIT: Added in a comment but here as well for visibility. I bought all of these drives used. All of these are plugged directly into the board. I cannot recall the exact setup, but I think there are two chips on the mobo that handle 2/3 of the ports, and the intel southbridge handles the rest - I don't have a hardware raid controller. I never get errors on sd[cd] only on those other four, and always in that pattern, [f] has the most, [egh] have less and all around the same.