1

I'm running a small home server which runs Ubuntu 20.04 (19.04 before). This server only runs docker and generally writes its data to a ZFS Pool that I mount (not really relevant for this story, just for context).

As boot disk I'm using a Kingston A2000 512GB NVME drive with an EXT4 filesystem. This afternoon and a few times this week, the server stopped responding and I couldn't login to it remotely. After connecting a screen to it I found out the SSD had been mounted as Readonly because of an error. I didn't manage to find out what the error was. Upon reboot I decided to inspect the SSD for bad sectors or other issues I didn't find any issues. However it did stand out to me the SSD (half a year old) has 56TB written to it and only 6TB read.

This really bothers me because that's just way too much. I have set the noatime property and I am running trim.

The only things stored on the SSD are: +/- 30 Docker containers, Ubuntu 20.04 and some data from a few containers (Plex metadata, no videos/Duplicati databases that run daily backups/files for a Minecraft server with 5 unfrequent users in Docker).

I'm trying to get to the bottom of the high writes, but I have no idea how I could approach this in a smart or structured manner. I've found some commands to check all the files written to since boot, but these are just way too many files for me to go trough manually over for example a week.

I'm also still not sure why the drive keeps going into readonly mode but that might be a separate question.

Any help is much appreciated!

smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-40-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       KINGSTON SA2000M8500G
Serial Number:                      XXXX
Firmware Version:                   S5Z42105
PCI Vendor/Subsystem ID:            0x2646
IEEE OUI Identifier:                0x0026b7
Controller ID:                      1
Number of Namespaces:               1
Namespace 1 Size/Capacity:          500,107,862,016 [500 GB]
Namespace 1 Utilization:            29,767,180,288 [29.7 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            0026b7 282536db15
Local Time is:                      Wed Jul 15 19:53:03 2020 CEST
Firmware Updates (0x14):            2 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Maximum Data Transfer Size:         32 Pages
Warning  Comp. Temp. Threshold:     75 Celsius
Critical Comp. Temp. Threshold:     80 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     9.00W       -        -    0  0  0  0        0       0
 1 +     4.60W       -        -    1  1  1  1        0       0
 2 +     3.80W       -        -    2  2  2  2        0       0
 3 -   0.0450W       -        -    3  3  3  3     2000    2000
 4 -   0.0040W       -        -    4  4  4  4    15000   15000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        46 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    10%
Data Units Read:                    12,031,713 [6.16 TB]
Data Units Written:                 110,463,016 [56.5 TB]
Host Read Commands:                 248,933,785
Host Write Commands:                1,467,111,619
Controller Busy Time:               9,524
Power Cycles:                       101
Power On Hours:                     4,515
Unsafe Shutdowns:                   5
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0

Error Information (NVMe Log 0x01, max 256 entries)
No Errors Logged
  • Are you using it for swap? – meuh Jul 15 at 20:11
  • No, I have swapiness set to 1 and when I setup the server I've entered swapoff -a. – Lars Jul 16 at 7:08
  • I've seen recommendations that SSD's shouldn't be mounted with discard/trim, but instead schedule a regular fstrim with cron. I don't know if the large write count includes discard commands. You really need to capture the reason for going read-only somehow. Perhaps rsyslog the kernel msgs over the network to another machine, or even to a file on a usb stick. It might be a pci bus error, for example. – meuh Jul 16 at 7:44
0

I've managed to track down the culprit using some feedback from a colleague. It turns out the issue was caused by Duplicati writing temporary files and removing them. In order to solve this I'm changing the location of these files to a harddisk rather than the SSD.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.