0

I have a server that I am running for myself and my friends. We host games using Ubuntu 18.04 LTS Server on the primary boot drive, and use a RAIDZ2 pool for storing backups of those games, our music, movies, etc.

Every week to two weeks I get a faulted pool and a lot of read/write errors.

me@server:/$ zpool status NAS
pool: NAS
state: ONLINE
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool 
clear'.
see: http://zfsonlinux.org/msg/ZFS-8000-HC
scan: scrub repaired 0B in 3h19m with 0 errors on Sun Aug 11 07:14:28 2019
config:

    NAME        STATE     READ WRITE CKSUM
    NAS         ONLINE       0   511     0
      raidz2-0  ONLINE       0   200     0
        sdc     ONLINE       0     0     0
        sdd     ONLINE       0     0     0
        sde     ONLINE       3   224     0
        sdf     ONLINE      12   225     0
        sdg     ONLINE       3   226     0
        sdh     ONLINE       3   227     0
    spares
      sdb       AVAIL

These errors never cause any data loss, and scrubbing never causes the pool to have to repair any bytes. I always have to restart the machine to get the pool to mount to the file system again. I've had this same pattern for months, now. Looking at this, this suggests to me that either one disk is bad (sdf), or in fact e-h are all on their way to failure and showing pre failure signs. Running disk self tests using S.M.A.R.T. always come back fine, showing no issues on the drives after I reset the machine and run tests. I assigned a hot-spare, hoping that it might be of use in case of a failure. At this point I am thinking I should replace drive sdf with sdb and see if this resolves the issue.

So, my question essentially is, when I see errors in a pool on multiple drives like this, is it always the case that all of the drives are pre-failure, or can the redundancy algorithm cause one bad disk to "spread" errors across other drives?

EDIT: Added in a comment but here as well for visibility. I bought all of these drives used. All of these are plugged directly into the board. I cannot recall the exact setup, but I think there are two chips on the mobo that handle 2/3 of the ports, and the intel southbridge handles the rest - I don't have a hardware raid controller. I never get errors on sd[cd] only on those other four, and always in that pattern, [f] has the most, [egh] have less and all around the same.

  • either you have a bad batch of drives or (IMO, more likely) your disk controller is faulty. what are the drives plugged into? motherboard sata ports? an SAS controller? USB (no!!!)? – cas Aug 19 at 1:42
  • from the errors shown, i'd guess that maybe sdc and sdd are plugged into motherboard sata ports, and sd[e-f] are on some kind of 4-port card? do you ever get errors on sd[cd] or only on sd[e-f]? – cas Aug 19 at 1:43
  • I bought all of these drives used, so that's a good change. All of these are plugged directly into the board. I cannot recall the exact setup, but I think there are two chips on the mobo that handle 2/3 of the ports, and the intel southbridge handles the rest - I don't have a hardware raid controller. I never get errors on sd[cd] only on those other four, and always in that pattern, f has the most, egh have less and all around the same. – Matt Bucklew Aug 19 at 3:30
  • IME extra sata ports on motherboards provided by 3rd party chips tend to be cheap and nasty, not as reliable as those provided by the CPU's main chipset. not always, but often enough that I tend to avoid using them. What brand/model of motherboard is it? BTW, the fact that you don't have a hardware raid controller is a good thing - you don't want to use hardware raid with ZFS, give it the raw drives so ZFS can manage the redundancy. – cas Aug 19 at 3:41
  • try replacing sdf with sdb (and physically remove sdf from the system). I still suspect a controller error because of the results of your smart tests and the fact that scrubbing the pool doesn't result in any errors or repairs. – cas Aug 19 at 3:43
0

Any time you're getting errors simultaneously across multiple drives, especially if they are on the same controller, either the controller or cabling should be your prime suspect. Even quality SATA cables are fairly cheap, so an easy thing to try is to just replace the cabling for one or two of the drives and see if that makes any difference. Make sure they are plugged in firmly at both ends.

Assuming that your disks are reporting SMART data correctly and honestly (far from all consumer disks do), you can probably confirm a data cabling problem by looking at SMART attribute 199, UDMA_CRC_Error_Count. If that is growing, especially if it's growing in tandem with the OS reporting storage errors, then it is very likely that you have a data cabling issue. Attribute 184 End-to-End_Error can also be informative.

Generally, as long as Reallocated_Sector_Ct, Reallocated_Event_Count, Current_Pending_Sector and Offline_Uncorrectable are staying put, it's probably not the disk itself that is having difficulties, which would leave either the cabling or the controller. About the only situation I can imagine where those would stay put but the disk would be having issues is something like a head alignment issue, but if that were the case, you wouldn't be getting just a handful of errors.

If you aren't already doing this, it might not be a bad idea to set up your system to somehow allow you to track SMART data trends over time. I tend to find that trends are more informative than just one snapshot in time and a feeling of "wasn't that value higher/lower before?".

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.