3

I have a Linux (Ubuntu 18, kernel 4.15) desktop booting from an M2 nvme disk.

Once a week, it will crash around midnight. The relevant log file output from /var/log/syslog.* is below:

Jul 16 00:00:00 rabbitcruncher systemd[1]: Starting Discard unused blocks...

Jul 16 00:00:00 rabbitcruncher kernel: [559644.954267] nvme 0000:41:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0014 address=0x0000000000000000 flags=0x0000]

Jul 16 00:00:00 rabbitcruncher kernel: [559644.975805] nvme nvme0: async event result 00010300

Jul 16 00:00:30 rabbitcruncher kernel: [559675.338834] nvme nvme0: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x1010

Jul 16 00:00:31 rabbitcruncher kernel: [559675.621182] nvme 0000:41:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0014 address=0x0000000000000000 flags=0x0000]

Jul 16 00:01:01 rabbitcruncher kernel: [559706.346300] nvme nvme0: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x1010

Jul 16 00:01:01 rabbitcruncher kernel: [559706.378641] nvme nvme0: Identify namespace failed

Jul 16 13:39:24 rabbitcruncher systemd-fsck[962]: /dev/nvme0n1p1: 12 files, 1186/130812 clusters

Jul 16 13:39:24 rabbitcruncher kernel: [ 1.052853] nvme nvme0: pci function 0000:41:00.0

Jul 16 13:39:24 rabbitcruncher kernel: [ 1.285806] nvme0n1: p1 p2

Jul 16 13:39:24 rabbitcruncher kernel: [ 5.036910] EXT4-fs (nvme0n1p2): mounted filesystem with ordered data mode. Opts: (null)

Jul 16 13:39:24 rabbitcruncher kernel: [ 5.318742] EXT4-fs (nvme0n1p2): re-mounted. Opts: errors=remount-ro

I understand the "Discard unused blocks" means that Linux is trying to run fstrim. However, I have disabled fstrim using systemctl but it still happens!

systemctl status fstrim.service

● fstrim.service - Discard unused blocks

Loaded: loaded (/lib/systemd/system/fstrim.service; static; vendor preset: enabled)

Active: inactive (dead)

I'm at a loss for what to do to fix this problem. Could anyone offer advice?

1
  • You might need to systemctl mask fstrim if disable isn't enough. You seem to have a vendor preset that enables it, and I'm not sure if a disable is enough in this case.
    – meuh
    Jul 17, 2018 at 13:29

2 Answers 2

1

Your Ubuntu is running inside a kvm virtual machine with AMD-Vi so it should not be running fstrim.

The fstrim service runs on a timer so as root:

rm /var/lib/systemd/timers/stamp-fstrim.timer
systemctl stop fstrim.service fstrim.timer
systemctl disable fstrim.service fstrim.timer
systemctl mask fstrim.service fstrim.timer
1
  • Thanks so much... This helps me narrow it down. Ubuntu actually is the host OS on the machine and I didn't intend to have any virtual machines installed as far as I know. I looked through and the only thing I could think is that Ubuntu Snaps run a VM called Ubuntu-Core, and it seems that Ubuntu-Core occasionally calls fstrim even though it's a virtual machine. I uninstalled all Ubuntu Snap and I haven't had this crash since then... kind of crazy, right?
    – kipp
    Jul 25, 2018 at 20:04
1

Your problem seems to be related to this one describing a crash every time a TRIMming is triggered by fstrim.timer. It seems to be a recent kernel issue that on AMD can be circumvented with iommu=soft (I found the prescription here with not much more details as regards the cause) - as per best practice with NVME SSDs, you can now enable fstrim.timer on your M.2 NVME disk back again.

I was bitten by (what I suppose is) the same problem on Intel - I reported it here - together with how I sidestepped the problem (intel_iommu=pt, which is the Intel equivalent of iommu=soft).

Once I made up my mind that the problem involved the IOMMU, I started looking for ways to change its behaviour; that is how I discovered here a Red Hat document regarding best practice for configuring a host for PCI passthrough - a somewhat only marginally related issue - mentioning intel_iommu=pt which, for me, solves the problem.

I'm currently not aware of side effects of such setting, if any.

1
  • 1
    Thank you for this information. On StackExchange we prefer to have information directly in the answer, just in case the linked pages become unavailable. In this case you might like to provide enough detail here to help someone reading your answer even if the links disappear, but still provide the links for full cross-reference and credit. Mar 7, 2019 at 15:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .