Root causes for failed command: WRITE FPDMA QUEUED

Question

On a brand new Intel NUC (10th gen) with brand new Samsung SSD 860 EVO 250GB RVT04B6Q that SSD yields some WRITE FPDMA QUEUED command failures such as this one:

Nov 28 21:25:26  ata3.00: exception Emask 0x10 SAct 0x60 SErr 0x400100 action 0x6 frozen
Nov 28 21:25:26  ata3.00: irq_stat 0x08000000, interface fatal error
Nov 28 21:25:26  ata3: SError: { UnrecovData Handshk }
Nov 28 21:25:26  ata3.00: failed command: WRITE FPDMA QUEUED
Nov 28 21:25:26  ata3.00: cmd 61/40:28:00:d7:31/00:00:00:00:00/40 tag 5 ncq dma 32768 out 
                          res 40/00:28:00:d7:31/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Nov 28 21:25:26  ata3.00: status: { DRDY }
Nov 28 21:25:26  ata3.00: failed command: WRITE FPDMA QUEUED
Nov 28 21:25:26  ata3.00: cmd 61/20:30:40:d7:31/00:00:00:00:00/40 tag 6 ncq dma 16384 out 
                          res 40/00:28:00:d7:31/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Nov 28 21:25:26  ata3.00: status: { DRDY }
Nov 28 21:25:26  ata3: hard resetting link
Nov 28 21:25:27  ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300) 
Nov 28 21:25:27  ata3.00: supports DRM functions and may not be fully accessible
Nov 28 21:25:27  ata3.00: supports DRM functions and may not be fully accessible
Nov 28 21:25:27  ata3.00: configured for UDMA/133
Nov 28 21:25:27  ata3: EH complete
Nov 28 21:25:27  ata3.00: Enabling discard_zeroes_data

These happen periodically in the order of 20 per day, while only relatively little IO happens. So far the Btrfs fileystem hasn't complained about checksum errors.

So is this just Samsung selling SSDs with poor quality assurance, possibly a bad cable or is a more systematic issue - such as Samsung not implementing commands in a standard conforming way, in its firmware?

---

Update 2020-01-09:

A brand new replacement Samsung SSD (same model) yields the same errors in that NUC. And with each burst of such errors the CRC_ERROR_COUNT SMART counter is incremented by one.

Opening the NUC shows that the SATA cable is folded (cf. bottom left corner):

Perhaps this sharp fold was deliberately applied by Intel during production. But it seems that it's unnecessary. Also if it was applied deliberately, why is there just one fold? I mean, two would make more sense then for symmetry (since the SSD is on top of the main board, i.e. integrated in the top cover). Also, is the 45 degree orientation of the fold optimal? I'm not an electrical engineer thus all this is perhaps completely irrelevant to this problem.

A good way to reproduce this error is to use fio. I mean it also happens during normal usage (e.g. when upgrading/installing packges), but with fio it's easier to reproduce. Example:

fio --rw=randrw --name=lol --bs=128k --direct=1 --filename=/dev/nvme0n1 \
    --numjobs=1 --ioengine=libaio --iodepth=32 --refill_buffers

If you run this - say - 2 times in a row and you don't see this error than this is a pretty good indication that your hardware is good and doesn't have this issue.

Answer 1

So I might be wrong here but I don't have enough rep to comment, so here's my 2cents.

I have a desktop with the same SSD, and get the exact same errors. It's due to a bug with the AMD SATA chipset & Samsung firmware - is it an AMD system? If so, you can disable NCQ with libata.force=noncq in the kernel parameters.

As you saw in the smart report, the CRC_ERROR_COUNT is (one?) of the parameters affected by this. It can either be caused by NCQ & this bug, or faulty SATA cables. So I'd try disabling NCQ first. This will hurt performance, though.

EDIT: Although, I do see some occurances of the problem with Intel controllers too: https://bugzilla.kernel.org/show_bug.cgi?id=203475#c14