wear of NVME SSD disk (Debian)

Question

On my Debian/Sid desktop home computer (AMD Ryzen 2970WX, some MSI 399 motherboard) I have an SSD M2 NVME and I am getting -to root - automatic mails such as:

 The following warning/error was logged by the smartd daemon:

 Device: /dev/nvme0, number of Error Log entries increased from 423 to
 424

 Device info: Samsung SSD 970 EVO 2TB, S/N:S464NB0KA03837J,
 FW:2B2QEXE7, 2.00 TB

For details see host's SYSLOG.

That SSD disk hold the root partition (filled to 29% according to df -h) and /home (filled to 5%).

The desktop (powered on thru an UPS 24h/24, located near Paris in France) is mostly used to develop RefPerSys and usual software developer activity (software build, debugging, tests, and mail and web browsing, LaTeX, emacs, running ./refpersys, etc...).

My understanding is that it is normal wearout of SSD NVME disks.

After a smartctl --test=short /dev/nvme0 (run as root) the command smartctl -a /dev/nvme0 gives

smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-3-amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       Samsung SSD 970 EVO 2TB
Serial Number:                      S464NB0KA03837J
Firmware Version:                   2B2QEXE7
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 2,000,398,934,016 [2.00 TB]
Unallocated NVM Capacity:           0
Controller ID:                      4
Number of Namespaces:               1
Namespace 1 Size/Capacity:          2,000,398,934,016 [2.00 TB]
Namespace 1 Utilization:            258,943,426,560 [258 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            002538 5a81b50e6f
Local Time is:                      Mon Feb  3 10:28:49 2020 MET
Firmware Updates (0x16):            3 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Maximum Data Transfer Size:         512 Pages
Warning  Comp. Temp. Threshold:     82 Celsius
Critical Comp. Temp. Threshold:     82 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     6.20W       -        -    0  0  0  0        0       0
 1 +     4.30W       -        -    1  1  1  1        0       0
 2 +     2.10W       -        -    2  2  2  2        0       0
 3 -   0.0400W       -        -    3  3  3  3      210    1200
 4 -   0.0050W       -        -    4  4  4  4     2000    8000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        39 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    170,671,345 [87.3 TB]
Data Units Written:                 6,787,146 [3.47 TB]
Host Read Commands:                 1,072,794,583
Host Write Commands:                62,979,313
Controller Busy Time:               1,480
Power Cycles:                       196
Power On Hours:                     906
Unsafe Shutdowns:                   136
Media and Data Integrity Errors:    0
Error Information Log Entries:      427
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               39 Celsius
Temperature Sensor 2:               43 Celsius

Error Information (NVMe Log 0x01, max 64 entries)
No Errors Logged

Here is the output of smartctl -x /dev/nvme0 :

smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-3-amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       Samsung SSD 970 EVO 2TB
Serial Number:                      S464NB0KA03837J
Firmware Version:                   2B2QEXE7
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 2,000,398,934,016 [2.00 TB]
Unallocated NVM Capacity:           0
Controller ID:                      4
Number of Namespaces:               1
Namespace 1 Size/Capacity:          2,000,398,934,016 [2.00 TB]
Namespace 1 Utilization:            258,943,426,560 [258 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            002538 5a81b50e6f
Local Time is:                      Mon Feb  3 10:42:30 2020 MET
Firmware Updates (0x16):            3 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Maximum Data Transfer Size:         512 Pages
Warning  Comp. Temp. Threshold:     82 Celsius
Critical Comp. Temp. Threshold:     82 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     6.20W       -        -    0  0  0  0        0       0
 1 +     4.30W       -        -    1  1  1  1        0       0
 2 +     2.10W       -        -    2  2  2  2        0       0
 3 -   0.0400W       -        -    3  3  3  3      210    1200
 4 -   0.0050W       -        -    4  4  4  4     2000    8000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        39 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    170,671,351 [87.3 TB]
Data Units Written:                 6,787,156 [3.47 TB]
Host Read Commands:                 1,072,794,690
Host Write Commands:                62,980,162
Controller Busy Time:               1,480
Power Cycles:                       196
Power On Hours:                     906
Unsafe Shutdowns:                   136
Media and Data Integrity Errors:    0
Error Information Log Entries:      427
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               39 Celsius
Temperature Sensor 2:               42 Celsius

Error Information (NVMe Log 0x01, max 64 entries)
No Errors Logged

---

I find such a test and the no errors logged message reassuring.

question:

When should I become concerned?

In an ideal world I would prefer to have the SSD changed before it crashes unexpectedly. I heard a rumor that an SSD fails totally (not gradually, like rotating hard disk do). I am not a hardware guy at all.

What Linux command should I run (e.g. monthly) to assess the wear out of the the SSD NVME disk?

Answer 1

Try sudo nvme error-log /dev/nvme0 (requires the package nvme-cli to be installed via apt, synaptic or similar).

Answer 2

The lifetime of a SSD is usually limited by the number of writes the memory cells can handle. The memory cells in an SSD/NVMe can only endure a limited number of write operations before they fail.

Generally the memory cells in solid state devices experience wear during each write operations and each memory cells tolerate only a limited (finite) number of overwrite passes. (This value is usually smaller and smaller as bits stored in each cells increase: SLC->MLC->TLC->QLC). Most solid state disks (SSD devices) report the overall health of the memory cells by various attributes.

You can use smartctl -a /dev/nvme0n1 to see the health information about your nvme disk.

The percentage_used shows the calculated number by disk manufacturer about disk lifespan. The available_spare parameter which indicates how many of the spare memory cells are available. Unless the "available_spare" is down to 1% and the "percentage_used" is going up to 100% the SSD is still fully functional.

Once you reach those limits though, you should think about replacing the SSD. Another indicator is Critical Warning which should be always 0x00 check the NVMe specification for other numbers.

Truncated example output of a failed/wearedout NVMe:

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
- available spare has fallen below threshold
- media has been placed in read only mode

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x09
Temperature:                        54 Celsius
Available Spare:                    0%
Available Spare Threshold:          10%
Percentage Used:                    3%
Data Units Read:                    152,049,475 [77.8 TB]
Data Units Written:                 123,071,212 [63.0 TB]

Example of a healthy NVMe:

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        38 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    1%
Data Units Read:                    8,602,182 [4.40 TB]
Data Units Written:                 13,527,143 [6.92 TB]

Example of a NVMe that is has used some spares but is still healthy:

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        45 Celsius
Available Spare:                    74%
Available Spare Threshold:          10%
Percentage Used:                    3%
Data Units Read:                    435,391,613 [222 TB]
Data Units Written:                 47,171,668 [24.1 TB]

Good reads: how-long-do-nvme-drives-last