I'm running a small home server which runs Ubuntu 20.04 (19.04 before). This server only runs docker and generally writes its data to a ZFS Pool that I mount (not really relevant for this story, just for context).
As boot disk I'm using a Kingston A2000 512GB NVME drive with an EXT4 filesystem. This afternoon and a few times this week, the server stopped responding and I couldn't login to it remotely. After connecting a screen to it I found out the SSD had been mounted as Readonly because of an error. I didn't manage to find out what the error was. Upon reboot I decided to inspect the SSD for bad sectors or other issues I didn't find any issues. However it did stand out to me the SSD (half a year old) has 56TB written to it and only 6TB read.
This really bothers me because that's just way too much. I have set the noatime property and I am running trim.
The only things stored on the SSD are: +/- 30 Docker containers, Ubuntu 20.04 and some data from a few containers (Plex metadata, no videos/Duplicati databases that run daily backups/files for a Minecraft server with 5 unfrequent users in Docker).
I'm trying to get to the bottom of the high writes, but I have no idea how I could approach this in a smart or structured manner. I've found some commands to check all the files written to since boot, but these are just way too many files for me to go trough manually over for example a week.
I'm also still not sure why the drive keeps going into readonly mode but that might be a separate question.
Any help is much appreciated!
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-40-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Number: KINGSTON SA2000M8500G
Serial Number: XXXX
Firmware Version: S5Z42105
PCI Vendor/Subsystem ID: 0x2646
IEEE OUI Identifier: 0x0026b7
Controller ID: 1
Number of Namespaces: 1
Namespace 1 Size/Capacity: 500,107,862,016 [500 GB]
Namespace 1 Utilization: 29,767,180,288 [29.7 GB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: 0026b7 282536db15
Local Time is: Wed Jul 15 19:53:03 2020 CEST
Firmware Updates (0x14): 2 Slots, no Reset required
Optional Admin Commands (0x0017): Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005f): Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Maximum Data Transfer Size: 32 Pages
Warning Comp. Temp. Threshold: 75 Celsius
Critical Comp. Temp. Threshold: 80 Celsius
Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 9.00W - - 0 0 0 0 0 0
1 + 4.60W - - 1 1 1 1 0 0
2 + 3.80W - - 2 2 2 2 0 0
3 - 0.0450W - - 3 3 3 3 2000 2000
4 - 0.0040W - - 4 4 4 4 15000 15000
Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 0
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 46 Celsius
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 10%
Data Units Read: 12,031,713 [6.16 TB]
Data Units Written: 110,463,016 [56.5 TB]
Host Read Commands: 248,933,785
Host Write Commands: 1,467,111,619
Controller Busy Time: 9,524
Power Cycles: 101
Power On Hours: 4,515
Unsafe Shutdowns: 5
Media and Data Integrity Errors: 0
Error Information Log Entries: 0
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Error Information (NVMe Log 0x01, max 256 entries)
No Errors Logged
fstrimwith cron. I don't know if the large write count includes discard commands. You really need to capture the reason for going read-only somehow. Perhapsrsyslogthe kernel msgs over the network to another machine, or even to a file on a usb stick. It might be a pci bus error, for example.