I woke up this morning to a notification email with some rather disturbing system log entries.
Dec 2 04:27:01 yeono kernel: [459438.816058] ata2.00: exception Emask 0x0 SAct 0xf SErr 0x0 action 0x6 frozen
Dec 2 04:27:01 yeono kernel: [459438.816071] ata2.00: failed command: WRITE FPDMA QUEUED
Dec 2 04:27:01 yeono kernel: [459438.816085] ata2.00: cmd 61/08:00:70:0d:ca/00:00:08:00:00/40 tag 0 ncq 4096 out
Dec 2 04:27:01 yeono kernel: [459438.816088] res 40/00:00:00:4f:c2/00:00:00:00:00/40 Emask 0x4 (timeout)
Dec 2 04:27:01 yeono kernel: [459438.816095] ata2.00: status: { DRDY }
(the above five lines were repeated a few times at a short interval)
Dec 2 04:27:01 yeono kernel: [459438.816181] ata2: hard resetting link
Dec 2 04:27:02 yeono kernel: [459439.920055] ata2: SATA link down (SStatus 0 SControl 300)
Dec 2 04:27:02 yeono kernel: [459439.932977] ata2: hard resetting link
Dec 2 04:27:09 yeono kernel: [459446.100050] ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Dec 2 04:27:09 yeono kernel: [459446.314509] ata2.00: configured for UDMA/133
Dec 2 04:27:09 yeono kernel: [459446.328037] ata2.00: device reported invalid CHS sector 0
("reported invalid CHS sector 0" repeated a few times at a short interval)
I make full nightly backups of my entire system to an external (USB-connected) drive, and the above happened right in the middle of that backup run. (The backup starts at 04:00 through cron, and tonight's logged completion just before 04:56.) The backup process itself claims to have completed without any errors.
There are two internally connected SATA drives and two externally (USB) connected drives on my system; one of the external drives is currently dormant. I don't recall off the top of my head which physical SATA ports are used for which of the internal drives.
When googling I found the AskUbuntu question Is this drive failure or something else? which indicates that a very similar error occured after 8-10 GB had been copied to a drive, but the actual failure mode was different as the drive switched to a read-only state. The only real similarity is that I did add on the order of 7-8 GB of data to my main storage last night, which would have been backed up around the time that the error occured.
smartd
is not reporting anything out of the ordinary on either of the internal drives. Unfortunately smartctl doesn't speak the language of the external backup drive's USB bridge, and simply complains about Unknown USB bridge [0x0bc2:0x3320 (0x100)]
. Googling for that specific error was distinctly unhelpful.
My main data storage as well as the backup is on ZFS and zpool status
reports 0 errors and no known data errors. Nevertheless I have initiated a full scrub on both the internal and external drives. It is currently slated to complete in about six hours for the internal drive (main storage pool) and 13-14 hours for the backup drive.
It seems that the next step should be to determine which drive was having trouble, and possibly replace it. The ata2.00
part probably tells me which drive was having problems, but how do I map that identifier to a physical drive?