1

I have a new system with debian (omv) a SSD hard drive for the OS and a software RAID 6 for the data.

I only saw now that I have very regular exceptions in my syslog. I'm worried now, what could cause those exceptions. Is it a software problem or is actually some hardware faulty? Can you actually read anything from those logs?

There is more exceptions in the syslog, but here an excerpt:

Jul 19 07:48:51 msa-nas1 kernel: [485174.166986] ata5.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6
Jul 19 07:48:51 msa-nas1 kernel: [485174.168522] ata5.01: failed command: WRITE MULTIPLE EXT
Jul 19 07:48:51 msa-nas1 kernel: [485174.170003] ata5.01: cmd 39/00:00:00:cc:89/00:04:08:00:00/f0 tag 0 pio 524288 out
Jul 19 07:48:51 msa-nas1 kernel: [485174.170003]          res 51/84:00:00:cd:89/84:03:08:00:00/f0 Emask 0x10 (ATA bus error)
Jul 19 07:48:51 msa-nas1 kernel: [485174.172996] ata5.01: status: { DRDY ERR }
Jul 19 07:48:51 msa-nas1 kernel: [485174.174500] ata5.01: error: { ICRC ABRT }
Jul 19 07:48:51 msa-nas1 kernel: [485174.176003] ata5: soft resetting link
Jul 19 07:48:51 msa-nas1 kernel: [485174.355492] ata5.00: configured for UDMA/33
Jul 19 07:48:51 msa-nas1 kernel: [485174.364550] ata5.01: configured for PIO0
Jul 19 07:48:51 msa-nas1 kernel: [485174.364574] ata5: EH complete
Jul 19 07:48:57 msa-nas1 kernel: [485180.175794] ata5.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6
Jul 19 07:48:57 msa-nas1 kernel: [485180.177436] ata5.01: failed command: WRITE MULTIPLE EXT
Jul 19 07:48:57 msa-nas1 kernel: [485180.179037] ata5.01: cmd 39/00:00:00:34:8a/00:04:08:00:00/f0 tag 0 pio 524288 out
Jul 19 07:48:57 msa-nas1 kernel: [485180.179037]          res 51/84:00:00:37:8a/84:01:08:00:00/f0 Emask 0x10 (ATA bus error)
Jul 19 07:48:57 msa-nas1 kernel: [485180.182279] ata5.01: status: { DRDY ERR }
Jul 19 07:48:57 msa-nas1 kernel: [485180.183907] ata5.01: error: { ICRC ABRT }
Jul 19 07:48:57 msa-nas1 kernel: [485180.185524] ata5: soft resetting link
Jul 19 07:48:57 msa-nas1 kernel: [485180.380318] ata5.00: configured for UDMA/33
Jul 19 07:48:57 msa-nas1 kernel: [485180.389391] ata5.01: configured for PIO0
Jul 19 07:48:57 msa-nas1 kernel: [485180.389407] ata5: EH complete
Jul 19 07:48:58 msa-nas1 kernel: [485180.939900] ata5.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6
Jul 19 07:48:58 msa-nas1 kernel: [485180.941736] ata5.01: failed command: WRITE MULTIPLE EXT
Jul 19 07:48:58 msa-nas1 kernel: [485180.943533] ata5.01: cmd 39/00:00:00:3c:8a/00:04:08:00:00/f0 tag 0 pio 524288 out
Jul 19 07:48:58 msa-nas1 kernel: [485180.943533]          res 51/84:00:00:3e:8a/84:02:08:00:00/f0 Emask 0x10 (ATA bus error)
Jul 19 07:48:58 msa-nas1 kernel: [485180.947169] ata5.01: status: { DRDY ERR }
Jul 19 07:48:58 msa-nas1 kernel: [485180.948998] ata5.01: error: { ICRC ABRT }
Jul 19 07:48:58 msa-nas1 kernel: [485180.950814] ata5: soft resetting link
Jul 19 07:48:58 msa-nas1 kernel: [485181.128420] ata5.00: configured for UDMA/33
Jul 19 07:48:58 msa-nas1 kernel: [485181.137482] ata5.01: configured for PIO0
Jul 19 07:48:58 msa-nas1 kernel: [485181.137505] ata5: EH complete

Thanks for any help with this.

EDIT:

Alright, I exchanged the cable of one of the drive where I thought it was ata5, now I realize there are two ata5 drives:

lrwxrwxrwx 1 root root 0 Jul 27 19:26 sde -> ../devices/pci0000:00/0000:00:14.1/ata5/host4/target4:0:0/4:0:0:0/block/sde
lrwxrwxrwx 1 root root 0 Jul 27 19:26 sdf -> ../devices/pci0000:00/0000:00:14.1/ata5/host4/target4:0:1/4:0:1:0/block/sdf

The second one is a SSD drive directly connected to the mainboard. Any idea what options I have? Did smartctl checks on both drives. Both without any errors.

EDIT2:

assuming it's not the SSD causing the trouble, I exchanged the other drive and SATA cable with parts that are working without errors in another system. I still get the errors.

How can a driver problem be identified, could the mainboard be faulty?

EDIT3: found something in the SMART log of the SSD drive:

212 SATA_PHY_Error          0x0032   100   100   ---    Old_age   Always       -       426

What does the SATA PHY Error stand for?

  • Have you had these errors from the start? They could be a hardware fault or a driver bug. Either way, you risk losing data. – Gilles Jul 20 '15 at 22:48
  • @Gilles: yes, had those errors from the start. Exchanged drive and cables with definitly working ones, but still get the error. A driver issue, could this be verified somehow? – Preexo Aug 10 '15 at 2:34
2

ATA errors like this are caused by faulty hardware for sure.

You can use "smartctl -a " to check for Reallocated sectors or for errors in the device journal. You can also check the attribute UDMA_CRC_Error_count to check for faulty cable or adapter.

If the ata device is part of a RAID, I would check it by itself by removing it from the RAID and using bonnie++ to check for performance issues or at least by using hdparm -t to compare it with other device.

If you need to find which device is ata5, try using this command: ls -l /sys/block/ | grep ata

It should give you a list of ata device which you can find yours in.

  • alright, will double check on the cables. Since I assembled them myself it might be possible that I left one a bit too loose. Thanks for helping, I might accept it as the answer after I went through the checks. – Preexo Jul 20 '15 at 3:36
  • Be sure to check for reallocated sectors on the device with smartctl, that would mean you have a faulty disk. If it is a new disk, that would mean you are eligible for a RMA. – TiCPU Jul 20 '15 at 3:41
  • changed the cable, since then it's fine, no more errors! thanks a lot! – Preexo Jul 27 '15 at 11:27
  • unfortunately I didn't look long or close enough... The errors are still there, I will update my question about it, since I found more than one ata5 drive, huh? smartctl checks went all fine. – Preexo Jul 28 '15 at 1:54
  • ok, swapped the cable with another faulty cable... oh dear... now got a proper one seems fine – Preexo Jul 29 '15 at 11:52
1

The steps I took to fix it:

  • updated BIOS
  • In the BIOS, diabled the SATA IDE Combined Mode with this help
  • reading the kernel documentation about kernel parameters, since every solution online was about adding parameters to that.
  • I found out that my SSD actually only supports SATA speed 3.0Gbps with a good shell script
    for i in `grep -l Gbps /sys/class/ata_link/*/sata_spd`; do
     echo Link "${i%/*}" Speed `cat $i`
     cat "${i%/*}"/device/dev*/ata_device/dev*/id | perl -nE 's/([0-9a-f]{2})/print chr hex $1/gie' | echo "    " Device `strings` | cut -f 1-3
    done
  • In the grub configuration, set the SATA port of the SSD drive to maximum speed 3.0
    vi /etc/default/grub

changed the parameter in this line to allow only 3Gbps for SATA port 7 (my SSD)

    GRUB_CMDLINE_LINUX_DEFAULT="libata.force=7:3.0G quiet"

update grub and reboot

    update-grub
    reboot

The solution to this has come a long long way for me. I basically approached the whole problem every other day from scratch.

The problems I found on the way where:

  • I checked my SMART stats every day and compared. The error count didn't increase even though the exceptions kept being thrown.
  • My SSD was actually the one causing the kernel exceptions, this script helped me lots to understand which ATA device was actually which hard drive in the case
  • My SSD and two other drives where on a completely wrong speed setting (UDMA)
root@msa-nas1:~# sudo hdparm -I /dev/sd{a,b,c,d,e,f,g} | grep -i udma
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
DMA: mdma0 mdma1 mdma2 udma0 udma1 *udma2 udma3 udma4 udma5 udma6
DMA: mdma0 mdma1 mdma2 udma0 *udma1 udma2 udma3 udma4 udma5 udma6
DMA: mdma0 mdma1 mdma2 udma0 udma1 *udma2 udma3 udma4 udma5 udma6
  • The dmesg log showed some strange messages about 40-wire cables, even though those don't really exist anymore, I bought two different NEW cables, nothing helped.
[    1.193091] ata5.01: ATA-8: SanDisk SD6SF1M128G1022I, X231200, max UDMA/133
[    1.193095] ata5.01: 250069680 sectors, multi 1: LBA48 NCQ (depth 0/32)
[    1.193743] ata5.00: limited to UDMA/33 due to 40-wire cable
[    1.193746] ata5.01: limited to UDMA/33 due to 40-wire cable
  • Grub loaded a funny kernel for the last two drives: pata_atiixp. I was expecting the AHCI driver.
[    1.022724] scsi4 : pata_atiixp
[    1.022834] scsi5 : pata_atiixp
[    1.022887] ata5: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0xf100 irq 14
[    1.022888] ata6: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0xf108 irq 15
  • I checked the power consumption and compared if it exceeded the power unit, it did not. Not even close.
  • I replaced the SSD with exactly the same model from another machine. Excactly the same model. Still the same errors.
  • The SSD!! was in fact incredibly slow, so the hdparm about the UDMA output was actually correct.
    root@msa-nas1:~# hdparm -t -T /dev/sdf

    /dev/sdf:
     Timing cached reads:   2144 MB in  2.00 seconds = 1072.18 MB/sec
     Timing buffered disk reads:   8 MB in  3.60 seconds =   2.22 MB/sec

I tried reaching out to SandDisk, it was their hard drive giving me the exceptions, without any success. I could really not find anyone with the exact same problem, but many people with similar problems, in the end I tried a few of those suggested solutions and it turned out to be a mix of a few things. Now it all makes perfectly sense to me, afterwards everyone knows better I guess.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.