3

I have a new system with debian (omv) a SSD hard drive for the OS and a software RAID 6 for the data.

I only saw now that I have very regular exceptions in my syslog. I'm worried now, what could cause those exceptions. Is it a software problem or is actually some hardware faulty? Can you actually read anything from those logs?

There is more exceptions in the syslog, but here an excerpt:

Jul 19 07:48:51 msa-nas1 kernel: [485174.166986] ata5.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6
Jul 19 07:48:51 msa-nas1 kernel: [485174.168522] ata5.01: failed command: WRITE MULTIPLE EXT
Jul 19 07:48:51 msa-nas1 kernel: [485174.170003] ata5.01: cmd 39/00:00:00:cc:89/00:04:08:00:00/f0 tag 0 pio 524288 out
Jul 19 07:48:51 msa-nas1 kernel: [485174.170003]          res 51/84:00:00:cd:89/84:03:08:00:00/f0 Emask 0x10 (ATA bus error)
Jul 19 07:48:51 msa-nas1 kernel: [485174.172996] ata5.01: status: { DRDY ERR }
Jul 19 07:48:51 msa-nas1 kernel: [485174.174500] ata5.01: error: { ICRC ABRT }
Jul 19 07:48:51 msa-nas1 kernel: [485174.176003] ata5: soft resetting link
Jul 19 07:48:51 msa-nas1 kernel: [485174.355492] ata5.00: configured for UDMA/33
Jul 19 07:48:51 msa-nas1 kernel: [485174.364550] ata5.01: configured for PIO0
Jul 19 07:48:51 msa-nas1 kernel: [485174.364574] ata5: EH complete
Jul 19 07:48:57 msa-nas1 kernel: [485180.175794] ata5.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6
Jul 19 07:48:57 msa-nas1 kernel: [485180.177436] ata5.01: failed command: WRITE MULTIPLE EXT
Jul 19 07:48:57 msa-nas1 kernel: [485180.179037] ata5.01: cmd 39/00:00:00:34:8a/00:04:08:00:00/f0 tag 0 pio 524288 out
Jul 19 07:48:57 msa-nas1 kernel: [485180.179037]          res 51/84:00:00:37:8a/84:01:08:00:00/f0 Emask 0x10 (ATA bus error)
Jul 19 07:48:57 msa-nas1 kernel: [485180.182279] ata5.01: status: { DRDY ERR }
Jul 19 07:48:57 msa-nas1 kernel: [485180.183907] ata5.01: error: { ICRC ABRT }
Jul 19 07:48:57 msa-nas1 kernel: [485180.185524] ata5: soft resetting link
Jul 19 07:48:57 msa-nas1 kernel: [485180.380318] ata5.00: configured for UDMA/33
Jul 19 07:48:57 msa-nas1 kernel: [485180.389391] ata5.01: configured for PIO0
Jul 19 07:48:57 msa-nas1 kernel: [485180.389407] ata5: EH complete
Jul 19 07:48:58 msa-nas1 kernel: [485180.939900] ata5.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6
Jul 19 07:48:58 msa-nas1 kernel: [485180.941736] ata5.01: failed command: WRITE MULTIPLE EXT
Jul 19 07:48:58 msa-nas1 kernel: [485180.943533] ata5.01: cmd 39/00:00:00:3c:8a/00:04:08:00:00/f0 tag 0 pio 524288 out
Jul 19 07:48:58 msa-nas1 kernel: [485180.943533]          res 51/84:00:00:3e:8a/84:02:08:00:00/f0 Emask 0x10 (ATA bus error)
Jul 19 07:48:58 msa-nas1 kernel: [485180.947169] ata5.01: status: { DRDY ERR }
Jul 19 07:48:58 msa-nas1 kernel: [485180.948998] ata5.01: error: { ICRC ABRT }
Jul 19 07:48:58 msa-nas1 kernel: [485180.950814] ata5: soft resetting link
Jul 19 07:48:58 msa-nas1 kernel: [485181.128420] ata5.00: configured for UDMA/33
Jul 19 07:48:58 msa-nas1 kernel: [485181.137482] ata5.01: configured for PIO0
Jul 19 07:48:58 msa-nas1 kernel: [485181.137505] ata5: EH complete

Thanks for any help with this.

EDIT:

Alright, I exchanged the cable of one of the drive where I thought it was ata5, now I realize there are two ata5 drives:

lrwxrwxrwx 1 root root 0 Jul 27 19:26 sde -> ../devices/pci0000:00/0000:00:14.1/ata5/host4/target4:0:0/4:0:0:0/block/sde
lrwxrwxrwx 1 root root 0 Jul 27 19:26 sdf -> ../devices/pci0000:00/0000:00:14.1/ata5/host4/target4:0:1/4:0:1:0/block/sdf

The second one is a SSD drive directly connected to the mainboard. Any idea what options I have? Did smartctl checks on both drives. Both without any errors.

EDIT2:

assuming it's not the SSD causing the trouble, I exchanged the other drive and SATA cable with parts that are working without errors in another system. I still get the errors.

How can a driver problem be identified, could the mainboard be faulty?

EDIT3: found something in the SMART log of the SSD drive:

212 SATA_PHY_Error          0x0032   100   100   ---    Old_age   Always       -       426

What does the SATA PHY Error stand for?

Improve this question
2
  • Have you had these errors from the start? They could be a hardware fault or a driver bug. Either way, you risk losing data.
    – Gilles 'SO- stop being evil'
    Jul 20, 2015 at 22:48
  • @Gilles: yes, had those errors from the start. Exchanged drive and cables with definitly working ones, but still get the error. A driver issue, could this be verified somehow?
    – Preexo
    Aug 10, 2015 at 2:34

2 Answers 2

Reset to default
4

The steps I took to fix it:

  • updated BIOS
  • In the BIOS, diabled the SATA IDE Combined Mode with this help
  • reading the kernel documentation about kernel parameters, since every solution online was about adding parameters to that.
  • I found out that my SSD actually only supports SATA speed 3.0Gbps with a good shell script 
    for i in `grep -l Gbps /sys/class/ata_link/*/sata_spd`; do
     echo Link "${i%/*}" Speed `cat $i`
     cat "${i%/*}"/device/dev*/ata_device/dev*/id | perl -nE 's/([0-9a-f]{2})/print chr hex $1/gie' | echo "    " Device `strings` | cut -f 1-3
    done
  • In the grub configuration, set the SATA port of the SSD drive to maximum speed 3.0
    vi /etc/default/grub

changed the parameter in this line to allow only 3Gbps for SATA port 7 (my SSD)

    GRUB_CMDLINE_LINUX_DEFAULT="libata.force=7:3.0G quiet"

update grub and reboot

    update-grub
    reboot

The solution to this has come a long long way for me. I basically approached the whole problem every other day from scratch.

The problems I found on the way where:

  • I checked my SMART stats every day and compared. The error count didn't increase even though the exceptions kept being thrown.
  • My SSD was actually the one causing the kernel exceptions, this script helped me lots to understand which ATA device was actually which hard drive in the case
  • My SSD and two other drives where on a completely wrong speed setting (UDMA)
root@msa-nas1:~# sudo hdparm -I /dev/sd{a,b,c,d,e,f,g} | grep -i udma
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
DMA: mdma0 mdma1 mdma2 udma0 udma1 *udma2 udma3 udma4 udma5 udma6
DMA: mdma0 mdma1 mdma2 udma0 *udma1 udma2 udma3 udma4 udma5 udma6
DMA: mdma0 mdma1 mdma2 udma0 udma1 *udma2 udma3 udma4 udma5 udma6
  • The dmesg log showed some strange messages about 40-wire cables, even though those don't really exist anymore, I bought two different NEW cables, nothing helped.
[    1.193091] ata5.01: ATA-8: SanDisk SD6SF1M128G1022I, X231200, max UDMA/133
[    1.193095] ata5.01: 250069680 sectors, multi 1: LBA48 NCQ (depth 0/32)
[    1.193743] ata5.00: limited to UDMA/33 due to 40-wire cable
[    1.193746] ata5.01: limited to UDMA/33 due to 40-wire cable
  • Grub loaded a funny kernel for the last two drives: pata_atiixp. I was expecting the AHCI driver.
[    1.022724] scsi4 : pata_atiixp
[    1.022834] scsi5 : pata_atiixp
[    1.022887] ata5: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0xf100 irq 14
[    1.022888] ata6: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0xf108 irq 15
  • I checked the power consumption and compared if it exceeded the power unit, it did not. Not even close.
  • I replaced the SSD with exactly the same model from another machine. Excactly the same model. Still the same errors.
  • The SSD!! was in fact incredibly slow, so the hdparm about the UDMA output was actually correct. 
    root@msa-nas1:~# hdparm -t -T /dev/sdf

    /dev/sdf:
     Timing cached reads:   2144 MB in  2.00 seconds = 1072.18 MB/sec
     Timing buffered disk reads:   8 MB in  3.60 seconds =   2.22 MB/sec

I tried reaching out to SandDisk, it was their hard drive giving me the exceptions, without any success. I could really not find anyone with the exact same problem, but many people with similar problems, in the end I tried a few of those suggested solutions and it turned out to be a mix of a few things. Now it all makes perfectly sense to me, afterwards everyone knows better I guess.

Improve this answer
2

ATA errors like this are caused by faulty hardware for sure.

You can use "smartctl -a " to check for Reallocated sectors or for errors in the device journal. You can also check the attribute UDMA_CRC_Error_count to check for faulty cable or adapter.

If the ata device is part of a RAID, I would check it by itself by removing it from the RAID and using bonnie++ to check for performance issues or at least by using hdparm -t to compare it with other device.

If you need to find which device is ata5, try using this command: ls -l /sys/block/ | grep ata

It should give you a list of ata device which you can find yours in.

Improve this answer
9
  • alright, will double check on the cables. Since I assembled them myself it might be possible that I left one a bit too loose. Thanks for helping, I might accept it as the answer after I went through the checks.
    – Preexo
    Jul 20, 2015 at 3:36
  • Be sure to check for reallocated sectors on the device with smartctl, that would mean you have a faulty disk. If it is a new disk, that would mean you are eligible for a RMA.
    – TiCPU
    Jul 20, 2015 at 3:41
  • changed the cable, since then it's fine, no more errors! thanks a lot!
    – Preexo
    Jul 27, 2015 at 11:27
  • unfortunately I didn't look long or close enough... The errors are still there, I will update my question about it, since I found more than one ata5 drive, huh? smartctl checks went all fine.
    – Preexo
    Jul 28, 2015 at 1:54
  • ok, swapped the cable with another faulty cable... oh dear... now got a proper one seems fine
    – Preexo
    Jul 29, 2015 at 11:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .