I have a zpool (3x 3TB Western Digital Red) that I scrub weekly for errors that comes up OK, but I have a recurring error in my syslog:
Jul 23 14:00:41 server kernel: [1199443.374677] ata2.00: exception Emask 0x0 SAct 0xe000000 SErr 0x0 action 0x0
Jul 23 14:00:41 server kernel: [1199443.374738] ata2.00: irq_stat 0x40000008
Jul 23 14:00:41 server kernel: [1199443.374773] ata2.00: failed command: READ FPDMA QUEUED
Jul 23 14:00:41 server kernel: [1199443.374820] ata2.00: cmd 60/02:c8:26:fc:43/00:00:f9:00:00/40 tag 25 ncq 1024 in
Jul 23 14:00:41 server kernel: [1199443.374820] res 41/40:00:26:fc:43/00:00:f9:00:00/40 Emask 0x409 (media error) <F>
Jul 23 14:00:41 server kernel: [1199443.374946] ata2.00: status: { DRDY ERR }
Jul 23 14:00:41 server kernel: [1199443.374979] ata2.00: error: { UNC }
Jul 23 14:00:41 server kernel: [1199443.376100] ata2.00: configured for UDMA/133
Jul 23 14:00:41 server kernel: [1199443.376112] sd 1:0:0:0: [sda] tag#25 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul 23 14:00:41 server kernel: [1199443.376115] sd 1:0:0:0: [sda] tag#25 Sense Key : Medium Error [current] [descriptor]
Jul 23 14:00:41 server kernel: [1199443.376118] sd 1:0:0:0: [sda] tag#25 Add. Sense: Unrecovered read error - auto reallocate failed
Jul 23 14:00:41 server kernel: [1199443.376121] sd 1:0:0:0: [sda] tag#25 CDB: Read(16) 88 00 00 00 00 00 f9 43 fc 26 00 00 00 02 00 00
Jul 23 14:00:41 server kernel: [1199443.376123] blk_update_request: I/O error, dev sda, sector 4181982246
Jul 23 14:00:41 server kernel: [1199443.376194] ata2: EH complete
A while back I had a faulty SATA cable that caused some read/write errors (that were later corrected by zpool scrubs and restoring from snapshots) and originally thought this error was a result of this. However it keeps randomly recurring, this time while I was in the middle of a scrub.
So far ZFS says that there are no errors, but it also says it's "repairing" that disk:
pool: sdb
state: ONLINE
scan: scrub in progress since Sun Jul 23 00:00:01 2017
5.41T scanned out of 7.02T at 98.9M/s, 4h44m to go
16.5K repaired, 77.06% done
config:
NAME STATE READ WRITE CKSUM
sdb ONLINE 0 0 0
ata-WDC_WD30EFRX-68EUZN0_WD-WMC4N1366685 ONLINE 0 0 0 (repairing)
ata-WDC_WD30EFRX-68EUZN0_WD-WMC4N0K3PFPS ONLINE 0 0 0
ata-WDC_WD30EFRX-68EUZN0_WD-WMC4N0M94AKN ONLINE 0 0 0
cache
sde ONLINE 0 0 0
errors: No known data errors
SMART data seems to tell me that everything is OK after running a short test, I'm in the middle of running the long self-test now to see if that comes up with anything. The only thing that jumps out is the UDMA_CRC_Error_Count
, but after I fixed that SATA cable it hasn't increased at all.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 195 175 021 Pre-fail Always - 5233
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 625
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0
9 Power_On_Hours 0x0032 069 069 000 Old_age Always - 22931
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 625
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 581
193 Load_Cycle_Count 0x0032 106 106 000 Old_age Always - 283773
194 Temperature_Celsius 0x0022 118 109 000 Old_age Always - 32
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 133 000 Old_age Always - 1801
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 22931 -
In addition to that, I'm also getting notifications about ZFS I/O errors, even though according to this it's just a bug related to drive idling/spin up time.
eid: 71
class: io
host: server
time: 2017-07-23 15:57:49-0500
vtype: disk
vpath: /dev/disk/by-id/ata-WDC_WD30EFRX-68EUZN0_WD-WMC4N1366685-part1
vguid: 0x979A2C1464C41735
cksum: 0
read: 0
write: 0
pool: sdb
My main question is how concerned should I be about that drive? I'm inclined to go replace it to be safe, but waned to know how soon I need to.
Here are the possibilities that I'm thinking might explain discrepancy between SMART data and ZFS/kernel:
- ZFS io error bug makes the kernel think that there's bad sectors, but according to SMART there aren't any.
- ZFS keeps repairing that drive (related to previous errors with faulty cable), which also might point to drive failure, despite SMART data.
- The error is a false alarm and related this unfixed bug in Ubuntu
EDIT: Now I just realized that the good drives are on firmware version 82.00A82, while the one that's getting the errors is 80.00A80. According to the Western Digital forum, there's no way to update this particular model's firmware. I'm sure that's not helping either.
EDIT 2: Forgot to update this a long time ago but this did end up being a hardware issue. After swapping multiple SATA cables, I finally realized that the issue the whole time was a failing power cable. The power flakiness was killing the drive, I but managed to get better drives and save the pool.