ZFS - "Add. Sense: Unrecovered read error - auto reallocate failed" in syslog, but SMART data looks OK

Question

I have a zpool (3x 3TB Western Digital Red) that I scrub weekly for errors that comes up OK, but I have a recurring error in my syslog:

Jul 23 14:00:41 server kernel: [1199443.374677] ata2.00: exception Emask 0x0 SAct 0xe000000 SErr 0x0 action 0x0
Jul 23 14:00:41 server kernel: [1199443.374738] ata2.00: irq_stat 0x40000008
Jul 23 14:00:41 server kernel: [1199443.374773] ata2.00: failed command: READ FPDMA QUEUED
Jul 23 14:00:41 server kernel: [1199443.374820] ata2.00: cmd 60/02:c8:26:fc:43/00:00:f9:00:00/40 tag 25 ncq 1024 in
Jul 23 14:00:41 server kernel: [1199443.374820] res 41/40:00:26:fc:43/00:00:f9:00:00/40 Emask 0x409 (media error) <F>
Jul 23 14:00:41 server kernel: [1199443.374946] ata2.00: status: { DRDY ERR }
Jul 23 14:00:41 server kernel: [1199443.374979] ata2.00: error: { UNC }
Jul 23 14:00:41 server kernel: [1199443.376100] ata2.00: configured for UDMA/133
Jul 23 14:00:41 server kernel: [1199443.376112] sd 1:0:0:0: [sda] tag#25 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 23 14:00:41 server kernel: [1199443.376115] sd 1:0:0:0: [sda] tag#25 Sense Key : Medium Error [current] [descriptor]
Jul 23 14:00:41 server kernel: [1199443.376118] sd 1:0:0:0: [sda] tag#25 Add. Sense: Unrecovered read error - auto reallocate failed
Jul 23 14:00:41 server kernel: [1199443.376121] sd 1:0:0:0: [sda] tag#25 CDB: Read(16) 88 00 00 00 00 00 f9 43 fc 26 00 00 00 02 00 00
Jul 23 14:00:41 server kernel: [1199443.376123] blk_update_request: I/O error, dev sda, sector 4181982246
Jul 23 14:00:41 server kernel: [1199443.376194] ata2: EH complete

A while back I had a faulty SATA cable that caused some read/write errors (that were later corrected by zpool scrubs and restoring from snapshots) and originally thought this error was a result of this. However it keeps randomly recurring, this time while I was in the middle of a scrub.

So far ZFS says that there are no errors, but it also says it's "repairing" that disk:

 pool: sdb
 state: ONLINE
  scan: scrub in progress since Sun Jul 23 00:00:01 2017
    5.41T scanned out of 7.02T at 98.9M/s, 4h44m to go
    16.5K repaired, 77.06% done
config:

    NAME                                        STATE     READ WRITE CKSUM
    sdb                                         ONLINE       0     0     0
      ata-WDC_WD30EFRX-68EUZN0_WD-WMC4N1366685  ONLINE       0     0     0  (repairing)
      ata-WDC_WD30EFRX-68EUZN0_WD-WMC4N0K3PFPS  ONLINE       0     0     0
      ata-WDC_WD30EFRX-68EUZN0_WD-WMC4N0M94AKN  ONLINE       0     0     0
    cache
      sde                                       ONLINE       0     0     0

errors: No known data errors

SMART data seems to tell me that everything is OK after running a short test, I'm in the middle of running the long self-test now to see if that comes up with anything. The only thing that jumps out is the UDMA_CRC_Error_Count, but after I fixed that SATA cable it hasn't increased at all.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   195   175   021    Pre-fail  Always       -       5233
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       625
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   069   069   000    Old_age   Always       -       22931
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       625
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       581
193 Load_Cycle_Count        0x0032   106   106   000    Old_age   Always       -       283773
194 Temperature_Celsius     0x0022   118   109   000    Old_age   Always       -       32
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   133   000    Old_age   Always       -       1801
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     22931         -

In addition to that, I'm also getting notifications about ZFS I/O errors, even though according to this it's just a bug related to drive idling/spin up time.

  eid: 71
 class: io
  host: server
  time: 2017-07-23 15:57:49-0500
 vtype: disk
 vpath: /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-WMC4N1366685-part1
 vguid: 0x979A2C1464C41735
 cksum: 0
  read: 0
 write: 0
  pool: sdb

My main question is how concerned should I be about that drive? I'm inclined to go replace it to be safe, but waned to know how soon I need to.

Here are the possibilities that I'm thinking might explain discrepancy between SMART data and ZFS/kernel:

1. ZFS io error bug makes the kernel think that there's bad sectors, but according to SMART there aren't any.
2. ZFS keeps repairing that drive (related to previous errors with faulty cable), which also might point to drive failure, despite SMART data.
3. The error is a false alarm and related this unfixed bug in Ubuntu

EDIT: Now I just realized that the good drives are on firmware version 82.00A82, while the one that's getting the errors is 80.00A80. According to the Western Digital forum, there's no way to update this particular model's firmware. I'm sure that's not helping either.

EDIT 2: Forgot to update this a long time ago but this did end up being a hardware issue. After swapping multiple SATA cables, I finally realized that the issue the whole time was a failing power cable. The power flakiness was killing the drive, I but managed to get better drives and save the pool.

Answer 1

In the end, it's your data, so you would be the one to say whether the drive should be replaced or not. In the end, it's just spinning rust.

Though, I should point out that it appears you've created a cat/RAID0 pool, so if a drive fails, you'll lose everything. And without a mirror, ZFS is unable to repair any failed files -- only report them.

If you're seeing the error messages sent to syslog while the scrub is running, perhaps its from the drives being taxed while the check the ZFS checksums. And since not all data is accessed, the scrub could be hitting a block the drive deems needs to be reallocated. Or noise on the line. And I'm not referring to Brendan Gregg yelling at disks. ;o) You did note a cable issue, perhaps a controller or port issue is also in the mix?

You also noted a Western Digital forum. I've seen many "complaints" on consumer drives not playing well with software or hardware RAID. If your data is important, you may want to consider using a mirror, and possibly even a 3-way mirror since disks aren't that much and something else could fail during a rebuild/resilver.

As far as "smart data," the verdict is out on how "smart" or useful it is. I've seen drives pass the vendors tests, yet be useless.