0

Overview

For around the last two years, my desktop computer has been crashing intermittently due to filesystem errors forcing the SSD to become read-only. It might go well over a week without anything happening, and it might crash more than once on a given day. It seemed to often be triggered during large writes during backups and restorations and immediately after booting, but it could happen at other times. I have gone through periods of time thinking that something I did fixed the intermittent crashing issue only to have it happen again a while later.

Edit: These shutdowns often cause the UEFI boot entries to become corrupted, making multiple power cycles necessary to get a boot to happen.

I would also have files not copy correctly to USB drives using $ dd, but this seems to have been resolved after fixing my RAM configuration as described below.

This has happened across multiple installations of different distros (all debian-based) and on multiple filesystems (ext4 and btrfs).

My data is backed up on an external drive, and the most important parts are on other devices through syncthing. I recognize that the SSD could potentially die any day.


Specifications and Configuration

CPU: AMD Ryzen 9 3900x

SSD: 1 TB Samsung 970 EVO Plus

RAM: Crucial Ballistix 3200 MHz (16GB x 2) BL2K16G32C16U4B

Motherboard: ASRock B550 Phantom ITX

Current Distro: KDE Neon


Things That I Have Tried

I tested the ram using memtester (run under linux, not booted independently) a couple of months ago, and I might have tested it more extensively at some point over a year ago. I haven't had any memory errors come up during any tests I've ever done on this RAM.

I discovered that my RAM apparently requires a factory-overclocked voltage of 1.35V by default. I fixed this in the BIOS. This seemed to help faulty writes to USB drives and SD cards, but not the intermittent crashing. I've read about and adjusted the other settings previously, but everything seems to be set at the factory default currently.

I updated the UEFI firmware about a year ago. I updated the UEFI firmware to the most recent version.

I've attempted to upgrade the firmware on the SSD, but it seems to be at the latest version that the specific revision supports.

I've adjusted the ASPM setting on the boot parameters as described on this question with similar symptoms.

I've examined the SSD using smartmontools -> smartctl. Nothing seems to be that out of the ordinary.

    === START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        56 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    19,717,312 [10.0 TB]
Data Units Written:                 39,276,405 [20.1 TB]
Host Read Commands:                 117,854,095
Host Write Commands:                1,047,333,000
Controller Busy Time:               1,195
Power Cycles:                       702
Power On Hours:                     1,261
Unsafe Shutdowns:                   30
Media and Data Integrity Errors:    0
Error Information Log Entries:      3,591
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               56 Celsius
Temperature Sensor 2:               55 Celsius

Edit: These are the relevant lines output on dmesg using journalctl | egrep 'kernel.*nvme' immediately following a triggered shutdown using $ stress-ng --hdd $(nproc) and reboot.

Jul 03 23:38:51 $HOSTNAME kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-5.13.0-51-generic root=UUID=e624d68e-7ffe-4bdb-98e3-a349ac9cc3e0 ro quiet splash nvme_core.default_ps_max_latency_us=0 pcie_aspm=off vt.handoff=7
Jul 03 23:38:51 $HOSTNAME kernel: Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.13.0-51-generic root=UUID=e624d68e-7ffe-4bdb-98e3-a349ac9cc3e0 ro quiet splash nvme_core.default_ps_max_latency_us=0 pcie_aspm=off vt.handoff=7
Jul 03 23:38:51 $HOSTNAME kernel: nvme nvme0: pci function 0000:01:00.0
Jul 03 23:38:51 $HOSTNAME kernel: nvme nvme0: missing or invalid SUBNQN field.
Jul 03 23:38:51 $HOSTNAME kernel: nvme nvme0: Shutdown timeout set to 8 seconds
Jul 03 23:38:51 $HOSTNAME kernel: nvme nvme0: 32/0/0 default/read/poll queues
Jul 03 23:38:51 $HOSTNAME kernel:  nvme0n1: p1 p2 p3 p4 p5
Jul 03 23:38:51 $HOSTNAME kernel: EXT4-fs (nvme0n1p3): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
Jul 03 23:38:51 $HOSTNAME kernel: EXT4-fs (nvme0n1p3): re-mounted. Opts: (null). Quota mode: none.
Jul 03 23:38:51 $HOSTNAME kernel: Adding 67108860k swap on /dev/nvme0n1p2.  Priority:-2 extents:1 across:67108860k SSFS
Jul 03 23:38:51 $HOSTNAME kernel: EXT4-fs (nvme0n1p4): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
Jul 03 23:38:52 $HOSTNAME kernel: EXT4-fs (nvme0n1p5): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.

My Discovery

I finally found something that always makes a crash occur:

$ stress-ng --hdd $(nproc)

$ iotop -o shows all I/O activity stopping within 10 seconds due to the disk being forced to a read-only state, and a shutdown occurs within 20 seconds after that due to systemd-journald or others not being able to write to the disk.

According to the manual for stress-ng, temporary files are being written, read, and deleted. Other sources claim that the write() and unlink() calls are being used.

Other options such as

$ stress-ng --io $(nproc)
$ stress-ng --cpu $(nproc)
$ stress-ng --vm $(nproc)
$ stress-ng --iomix $(nproc)

don't seem to cause any trouble.


The Question

Does the unique ability of the stress test $ stress-ng --hdd $(nproc) to consistently crash my computer necessarily mean that the SSD hardware is to blame?

I've read that SSDs are known for failing catastrophically, so I feel uncertain about the issue really being the SSD. I am very interested in understanding the nature of the problem better, so I would really appreciate links to relevant documentation or any words of wisdom that can be offered on the subject.

5
  • It's probably the SSD, but it is worth checking if there's an updated BIOS for your motherboard - there have been several AGESA bug-fixes in the last year, and that can affect PCI-e devices incl. NVME drives. Also try removing and re-installing the drive, make sure it's seated correctly (probably not the problem, but doesn't hurt to try). 56° idle temp seems a little hot but still well under the max of ~ 85°, my pair of 970 EVOs (not Plus) idle at around 29°, but they're on a full-size ATX board in a well-ventilated case - try running without the M.2 heatsink that came with your m/b.
    – cas
    Commented Jul 3, 2022 at 5:36
  • Would be nice to see what kind of messages are logged in dmesg, e.g. journalctl | egrep 'kernel.*nvme' Commented Jul 3, 2022 at 10:41
  • @ArtemS.Tashkinov I added the logged messages that follow a crash.
    – tlake29
    Commented Jul 4, 2022 at 19:13
  • Unfortunately these are normal informational messages. No idea what's going on on your side. Might be related to high temperatures, newer SSDs require a heatsink. If 56C above is logged when your SSD was idle, it looks a tad too high for me. My SSD rests around 34C. Please check the advice given here: unix.stackexchange.com/questions/470778/… Commented Jul 4, 2022 at 21:34
  • Also, would be great if you run in single user mode and ran the test in console. That way you'd see what's going on right in front of your eyes. You could also run dmesg --console-on to see in real time whether the kernel is unhappy about something. Commented Jul 6, 2022 at 22:07

0

You must log in to answer this question.

Browse other questions tagged .