nvme fstrim causing crash on linux, disabling with systemctl doesn't help

Question

I have a Linux (Ubuntu 18, kernel 4.15) desktop booting from an M2 nvme disk.

Once a week, it will crash around midnight. The relevant log file output from /var/log/syslog.* is below:

Jul 16 00:00:00 rabbitcruncher systemd[1]: Starting Discard unused blocks...

Jul 16 00:00:00 rabbitcruncher kernel: [559644.954267] nvme 0000:41:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0014 address=0x0000000000000000 flags=0x0000]

Jul 16 00:00:00 rabbitcruncher kernel: [559644.975805] nvme nvme0: async event result 00010300

Jul 16 00:00:30 rabbitcruncher kernel: [559675.338834] nvme nvme0: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x1010

Jul 16 00:00:31 rabbitcruncher kernel: [559675.621182] nvme 0000:41:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0014 address=0x0000000000000000 flags=0x0000]

Jul 16 00:01:01 rabbitcruncher kernel: [559706.346300] nvme nvme0: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x1010

Jul 16 00:01:01 rabbitcruncher kernel: [559706.378641] nvme nvme0: Identify namespace failed

Jul 16 13:39:24 rabbitcruncher systemd-fsck[962]: /dev/nvme0n1p1: 12 files, 1186/130812 clusters

Jul 16 13:39:24 rabbitcruncher kernel: [ 1.052853] nvme nvme0: pci function 0000:41:00.0

Jul 16 13:39:24 rabbitcruncher kernel: [ 1.285806] nvme0n1: p1 p2

Jul 16 13:39:24 rabbitcruncher kernel: [ 5.036910] EXT4-fs (nvme0n1p2): mounted filesystem with ordered data mode. Opts: (null)

Jul 16 13:39:24 rabbitcruncher kernel: [ 5.318742] EXT4-fs (nvme0n1p2): re-mounted. Opts: errors=remount-ro

I understand the "Discard unused blocks" means that Linux is trying to run fstrim. However, I have disabled fstrim using systemctl but it still happens!

systemctl status fstrim.service

● fstrim.service - Discard unused blocks

Loaded: loaded (/lib/systemd/system/fstrim.service; static; vendor preset: enabled)

Active: inactive (dead)

I'm at a loss for what to do to fix this problem. Could anyone offer advice?

Answer 1

Your Ubuntu is running inside a kvm virtual machine with AMD-Vi so it should not be running fstrim.

The fstrim service runs on a timer so as root:

rm /var/lib/systemd/timers/stamp-fstrim.timer
systemctl stop fstrim.service fstrim.timer
systemctl disable fstrim.service fstrim.timer
systemctl mask fstrim.service fstrim.timer

Answer 2

Your problem seems to be related to this one describing a crash every time a TRIMming is triggered by fstrim.timer. It seems to be a recent kernel issue that on AMD can be circumvented with iommu=soft (I found the prescription here with not much more details as regards the cause) - as per best practice with NVME SSDs, you can now enable fstrim.timer on your M.2 NVME disk back again.

I was bitten by (what I suppose is) the same problem on Intel - I reported it here - together with how I sidestepped the problem (intel_iommu=pt, which is the Intel equivalent of iommu=soft).

Once I made up my mind that the problem involved the IOMMU, I started looking for ways to change its behaviour; that is how I discovered here a Red Hat document regarding best practice for configuring a host for PCI passthrough - a somewhat only marginally related issue - mentioning intel_iommu=pt which, for me, solves the problem.

I'm currently not aware of side effects of such setting, if any.