1

I have been using pop os (currently I have the latest version based on Ubuntu 20.04) with my lenovo x1 extreme gen 1 very happily for a while (2-3 years) but recently ran into what is possibly a ssd related hardware issue (laptop crashes randomly giving ext4-fs and systemd-journald errors) which persist after a fresh install. I am attaching some screen shots below but I will also put a log of errors I could find in the logs directory below

enter image description here

enter image description here

Diagnosis:

fdisk, fsck:

pop-os@pop-os:~$ sudo fdisk -l
Disk /dev/loop0: 2.24 GiB, 2400944128 bytes, 4689344 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/nvme0n1: 953.89 GiB, 1024209543168 bytes, 2000409264 sectors
Disk model: SAMSUNG MZVLB1T0HALR-000L7        
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5FCEDA12-D1BA-4EEF-B174-C7F4C4F7ACFC


Device              Start        End    Sectors   Size Type

/dev/nvme0n1p1       4096    1023998    1019903   498M EFI System
/dev/nvme0n1p2    1024000    9412606    8388607     4G Microsoft basic data
/dev/nvme0n1p3    9412608 1992016558 1982603951 945.4G Linux filesystem
/dev/nvme0n1p4 1992016560 2000405166    8388607     4G Linux swap

(Note: Microsoft basic data is the recovery partition remaining from windows)

sudo fsck -CvMf /dev/nvme0n1p3
fsck from util-linux 2.34
e2fsck 1.45.5 (07-Jan-2020)

Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure                                     
Pass 3: Checking directory connectivity                                   
Pass 4: Checking reference counts
Pass 5: Checking group summary information                               
                                                                         

     366095 inodes used (0.59%, out of 61964288)
       2849 non-contiguous files (0.8%)
        412 non-contiguous directories (0.1%)
            # of inodes with ind/dind/tind blocks: 0/0/0
            Extent depth histogram: 326954/107
   11214214 blocks used (4.53%, out of 247825493)
          0 bad blocks
          2 large files

     287132 regular files
      36849 directories
          7 character device files
          0 block device files
          0 fifos
      91242 links
      42092 symbolic links (39013 fast symbolic links)
          6 sockets
------------
     457328 files

SMART test:

=== START OF INFORMATION SECTION ===

Model Number:                       SAMSUNG MZVLB1T0HALR-000L7
Firmware Version:                   5L2QEXA7
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 1,024,209,543,168 [1.02 TB]
Unallocated NVM Capacity:           0
Controller ID:                      4
Number of Namespaces:               1
Namespace 1 Size/Capacity:          1,024,209,543,168 [1.02 TB]
Namespace 1 Utilization:            47,027,638,272 [47.0 GB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            002538 8881b2cb9e
Local Time is:                      Mon Aug  3 16:34:10 2020 UTC
Firmware Updates (0x16):            3 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x001f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Maximum Data Transfer Size:         512 Pages
Warning  Comp. Temp. Threshold:     81 Celsius
Critical Comp. Temp. Threshold:     82 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
0 +     7.02W       -        -    0  0  0  0        0       0
1 +     6.30W       -        -    1  1  1  1        0       0
2 +     3.50W       -        -    2  2  2  2        0       0
3 -   0.0760W       -        -    3  3  3  3      210    1200
4 -   0.0050W       -        -    4  4  4  4     2000    8000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
0 +     512       0         0

=== START OF SMART DATA SECTION ===

SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        39 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    1%
Data Units Read:                    22,730,197 [11.6 TB]
Data Units Written:                 39,001,161 [19.9 TB]
Host Read Commands:                 280,072,901
Host Write Commands:                496,008,535
Controller Busy Time:               1,454
Power Cycles:                       2,705
Power On Hours:                     1,567
Unsafe Shutdowns:                   226
Media and Data Integrity Errors:    0
Error Information Log Entries:      2,071
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               39 Celsius
Temperature Sensor 2:               41 Celsius

Error Information (NVMe Log 0x01, max 64 entries)
No Errors Logged

To get more detailed information I went and looked at the logs for keywords. I looked at nvme, ext4-fs in the logs. Of note are the entries like

/var/log/kern.log:Aug  3 19:01:43 pop-os kernel: [  237.251085] blk_update_request: I/O error, dev nvme0n1, sector 1209397344 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 0

...

/var/log/kern.log:Aug  3 22:39:08 pop-os kernel: [    2.115859]  nvme0n1: p1 p2 p3 p4
/var/log/kern.log:Aug  3 22:39:08 pop-os kernel: [    3.868483] EXT4-fs (nvme0n1p3): INFO: recovery required on readonly filesystem
/var/log/kern.log:Aug  3 22:39:08 pop-os kernel: [    3.868483] EXT4-fs (nvme0n1p3): write access will be enabled during recovery
/var/log/kern.log:Aug  3 22:39:08 pop-os kernel: [    3.894018] EXT4-fs (nvme0n1p3): orphan cleanup on readonly fs
/var/log/kern.log:Aug  3 22:39:08 pop-os kernel: [    3.904196] EXT4-fs (nvme0n1p3): 227 orphan inodes deleted
/var/log/kern.log:Aug  3 22:39:08 pop-os kernel: [    3.904197] EXT4-fs (nvme0n1p3): recovery complete
/var/log/kern.log:Aug  3 22:39:08 pop-os kernel: [    3.916157] EXT4-fs (nvme0n1p3): mounted filesystem with ordered data mode. Opts: (null)
/var/log/kern.log:Aug  3 22:39:08 pop-os kernel: [    4.235950] EXT4-fs (nvme0n1p3): re-mounted. Opts: errors=remount-ro
/var/log/kern.log:Aug  3 22:39:08 pop-os kernel: [    5.580150] FAT-fs (nvme0n1p2): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
/var/log/kern.log:Aug  3 22:39:08 pop-os kernel: [    5.580956] FAT-fs (nvme0n1p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
/var/log/kern.log:Aug  3 22:39:48 pop-os kernel: [   47.658007] blk_update_request: I/O error, dev nvme0n1, sector 1209399024 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
/var/log/kern.log:Aug  5 07:16:47 pop-os kernel: [    2.018779]  nvme0n1: p1 p2 p3 p4
/var/log/kern.log:Aug  5 07:16:47 pop-os kernel: [    3.839434] EXT4-fs (nvme0n1p3): mounted filesystem with ordered data mode. Opts: (null)
/var/log/kern.log:Aug  5 07:16:47 pop-os kernel: [    4.149146] EXT4-fs (nvme0n1p3): re-mounted. Opts: errors=remount-ro
/var/log/kern.log:Aug  5 07:16:47 pop-os kernel: [    5.006306] FAT-fs (nvme0n1p2): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
/var/log/kern.log:Aug  5 07:16:47 pop-os kernel: [    5.006685] FAT-fs (nvme0n1p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
/var/log/kern.log:Aug  9 15:03:31 pop-os kernel: [    2.105116]  nvme0n1: p1 p2 p3 p4
/var/log/kern.log:Aug  9 15:03:31 pop-os kernel: [    3.892947] EXT4-fs (nvme0n1p3): mounted filesystem with ordered data mode. Opts: (null)
/var/log/kern.log:Aug  9 15:03:31 pop-os kernel: [    4.183333] EXT4-fs (nvme0n1p3): re-mounted. Opts: errors=remount-ro
/var/log/kern.log:Aug  9 15:03:31 pop-os kernel: [    4.682363] FAT-fs (nvme0n1p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
/var/log/kern.log:Aug  9 15:03:31 pop-os kernel: [    4.683046] FAT-fs (nvme0n1p2): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
/var/log/kern.log:Aug 10 13:35:55 pop-os kernel: [    2.111633]  nvme0n1: p1 p2 p3 p4
/var/log/kern.log:Aug 10 13:35:55 pop-os kernel: [    3.817532] EXT4-fs (nvme0n1p3): INFO: recovery required on readonly filesystem
/var/log/kern.log:Aug 10 13:35:55 pop-os kernel: [    3.817532] EXT4-fs (nvme0n1p3): write access will be enabled during recovery
/var/log/kern.log:Aug 10 13:35:55 pop-os kernel: [    3.827850] EXT4-fs (nvme0n1p3): recovery complete
/var/log/kern.log:Aug 10 13:35:55 pop-os kernel: [    3.832040] EXT4-fs (nvme0n1p3): mounted filesystem with ordered data mode. Opts: (null)
/var/log/kern.log:Aug 10 13:35:55 pop-os kernel: [    4.169487] EXT4-fs (nvme0n1p3): re-mounted. Opts: errors=remount-ro
/var/log/kern.log:Aug 10 13:35:55 pop-os kernel: [    5.442449] FAT-fs (nvme0n1p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
/var/log/kern.log:Aug 10 13:35:55 pop-os kernel: [    5.444632] FAT-fs (nvme0n1p2): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
/var/log/kern.log:Aug 11 00:03:10 pop-os kernel: [    2.078927]  nvme0n1: p1 p2 p3 p4
/var/log/kern.log:Aug 11 00:03:10 pop-os kernel: [    3.845395] EXT4-fs (nvme0n1p3): INFO: recovery required on readonly filesystem
/var/log/kern.log:Aug 11 00:03:10 pop-os kernel: [    3.845396] EXT4-fs (nvme0n1p3): write access will be enabled during recovery
/var/log/kern.log:Aug 11 00:03:10 pop-os kernel: [    4.026435] EXT4-fs (nvme0n1p3): orphan cleanup on readonly fs
/var/log/kern.log:Aug 11 00:03:10 pop-os kernel: [    4.026557] EXT4-fs (nvme0n1p3): 16 orphan inodes deleted
/var/log/kern.log:Aug 11 00:03:10 pop-os kernel: [    4.026557] EXT4-fs (nvme0n1p3): recovery complete
/var/log/kern.log:Aug 11 00:03:10 pop-os kernel: [    4.037091] EXT4-fs (nvme0n1p3): mounted filesystem with ordered data mode. Opts: (null)
/var/log/kern.log:Aug 11 00:03:10 pop-os kernel: [    4.352561] EXT4-fs (nvme0n1p3): re-mounted. Opts: errors=remount-ro
/var/log/kern.log:Aug 11 00:03:10 pop-os kernel: [    5.140268] FAT-fs (nvme0n1p2): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
/var/log/kern.log:Aug 11 00:03:10 pop-os kernel: [    5.176295] FAT-fs (nvme0n1p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
/var/log/kern.log:Aug 11 10:12:22 pop-os kernel: [    2.063656]  nvme0n1: p1 p2 p3 p4
/var/log/kern.log:Aug 11 10:12:22 pop-os kernel: [    3.861041] EXT4-fs (nvme0n1p3): INFO: recovery required on readonly filesystem
/var/log/kern.log:Aug 11 10:12:22 pop-os kernel: [    3.861041] EXT4-fs (nvme0n1p3): write access will be enabled during recovery
/var/log/kern.log:Aug 11 10:12:22 pop-os kernel: [    3.876059] EXT4-fs (nvme0n1p3): recovery complete
/var/log/kern.log:Aug 11 10:12:22 pop-os kernel: [    3.880170] EXT4-fs (nvme0n1p3): mounted filesystem with ordered data mode. Opts: (null)
/var/log/kern.log:Aug 11 10:12:22 pop-os kernel: [    4.200170] EXT4-fs (nvme0n1p3): re-mounted. Opts: errors=remount-ro
/var/log/kern.log:Aug 11 10:12:22 pop-os kernel: [    5.109084] FAT-fs (nvme0n1p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
/var/log/kern.log:Aug 11 10:12:22 pop-os kernel: [    5.131469] FAT-fs (nvme0n1p2): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
grep: /var/log/private: Is a directory
grep: /var/log/speech-dispatcher: Is a directory
/var/log/syslog:Aug  3 18:58:00 pop-os kernel: [    2.092722]  nvme0n1: p1 p2 p3 p4
/var/log/syslog:Aug  3 18:58:00 pop-os kernel: [    3.780347] EXT4-fs (nvme0n1p3): mounted filesystem with ordered data mode. Opts: (null)
/var/log/syslog:Aug  3 18:58:00 pop-os kernel: [    4.089493] EXT4-fs (nvme0n1p3): re-mounted. Opts: errors=remount-ro
Binary file /var/log/syslog matches

In dmesg, I also saw a bunch of errors related to temperature though I don't know how serious that is (though given that the threshold is 81C that is a bit worrying)

[    3.417048] kernel: mce: CPU3: Core temperature above threshold, cpu clock throttled (total events = 1)
[    3.417049] kernel: mce: CPU9: Core temperature above threshold, cpu clock throttled (total events = 1)
[    3.417050] kernel: mce: CPU3: Package temperature above threshold, cpu clock throttled (total events = 1)
[    3.417050] kernel: mce: CPU9: Package temperature above threshold, cpu clock throttled (total events = 1)
[    3.417091] kernel: mce: CPU6: Package temperature above threshold, cpu clock throttled (total events = 1)
[    3.417093] kernel: mce: CPU7: Package temperature above threshold, cpu clock throttled (total events = 1)
[    3.417094] kernel: mce: CPU1: Package temperature above threshold, cpu clock throttled (total events = 1)
[    3.417095] kernel: mce: CPU0: Package temperature above threshold, cpu clock throttled (total events = 1)
[    3.417096] kernel: mce: CPU4: Package temperature above threshold, cpu clock throttled (total events = 1)
[    3.417097] kernel: mce: CPU2: Package temperature above threshold, cpu clock throttled (total events = 1)
[    3.417098] kernel: mce: CPU5: Package temperature above threshold, cpu clock throttled (total events = 1)
[    3.417099] kernel: mce: CPU10: Package temperature above threshold, cpu clock throttled (total events = 1)
[    3.417100] kernel: mce: CPU8: Package temperature above threshold, cpu clock throttled (total events = 1)
[    3.417101] kernel: mce: CPU11: Package temperature above threshold, cpu clock throttled (total events = 1)

Finally, when I am installing popos (which I did quite a lot of times over the last month due to this issue), maybe one out of every two attempts the installer fails at extraction stage. It works when I retry a couple of times without changing anything with the live usb or any of the installation settings so it also seems like a random read/write error. The installation log also seems to reveal an input/output error. Of note are entries like:

Jul 31 21:30:16 pop-os kernel: [  163.161995] nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0xffff
Jul 31 21:30:16 pop-os kernel: [  163.254016] nvme 0000:71:00.0: Refused to change power state, currently in D3
Jul 31 21:30:16 pop-os kernel: [  163.254502] nvme nvme0: Removing after probe failure status: -19
Jul 31 21:30:16 pop-os kernel: [  163.346070] blk_update_request: I/O error, dev nvme0n1, sector 38805760 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 0
Jul 31 21:30:16 pop-os kernel: [  163.347594] EXT4-fs warning (device nvme0n1p3): ext4_end_bio:309: I/O error 10 writing to inode 33423367 (offset 9043968 size 2306048 starting block 1219744)
Jul 31 21:30:16 pop-os kernel: [  163.347601] Buffer I/O error on device nvme0n1p3, logical block 43168
Jul 31 21:30:16 pop-os kernel: [  163.347610] Buffer I/O error on device nvme0n1p3, logical block 43169

Extra info:

  1. I also did some of the memory diagnostic tests that come packaged with the laptop none of which returned any errors (not posting them here unless someone asks for it).

  2. After each re-installation of the Linux, the problem reappears after sometime (though I think it takes longer to appear if I try to keep the disk as empty as possible). Also the "refresh" option in the installation does not seem to help I have to completely reinstall Linux.

FIX ATTEMPTS:

  1. When I do a fresh install of the Linux this problem usually reappears after a few days and seems to be related how much stuff I put on my hard disk. If I intentionally try to keep it at a minimum it seems to take longer before the crash happens. The last crash happened when I was doing some semi-intensive (couple of 100 MBs) read and write operations through python.

  2. There is one hint in arch-linux wiki (https://wiki.archlinux.org/index.php/Solid_state_drive/NVMe) which says

    Samsung drive errors on Linux 4.10

    On Linux 4.10, drive errors can occur and causing system instability. This seems to be the result of a power saving state that the drive cannot use. Adding the kernel parameter nvme_core.default_ps_max_latency_us=5500[4][5] disables the lowest power saving state, preventing write errors.

    Mine is also Samsung (see below for more details) so I did what is suggested but this does not seem to help.

  3. I updated all my firmware through the lfvs though there were no ssd updates there, mainly bios. This resolved some other problems but not this particular problem.

I don't have much ideas about how to proceed though as I have almost zero knowledge about hardware and don't want to put in random kernel parameters that I know nothing about.
I can update complete logs if requested.

2
  • This sounds like it might be a hardware problem. Have you tried using smartctl to test your disk?
    – bk2204
    Aug 13, 2020 at 0:56
  • yes the results of smartctl are above and it gave no warnings and says all the tests have passed?
    – Sina
    Aug 13, 2020 at 8:14

1 Answer 1

0

It sounds like you have a problem with the NVMe hardware device. You could try booting from a recovery USB image and running badblocks on the NVMe device, or if there is a Samsung diagnostic tool to erase/test the device you could run that?

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .