0

Overview & Symptoms

My system freezes almost completely whenever my system runs out of RAM and starts hitting the swap partition heavily. Everything freezes including the mouse and keyboard with a few exceptions:

  • The hard drive light appears to indicate some background activity
  • The fan sometimes spins up and down indicating some CPU activity
  • nmap -sT (TCP handshake) from another machine reveals open ports indicating that the NIC is responding at the OSI transport layer

Nothing is logged indicating what causes this.

On one rare occasion I remember the mouse was able to move a bit after about a minute or two of the system being frozen. This issue does not appear to occur whenever there is plenty of free RAM available, it only seems to occur when the swap partition starts experiencing significant load.

Here is the output of free that indicates free RAM and swap storage, right now there is mild swap usage. This is typically entering the danger zone where the system would freeze, although I've witnessed up to 12MB of swap used without an issue.

Total RAM: 32GB

Total Swap: 24GB

              total        used        free      shared  buff/cache   available
Mem:           31Gi        26Gi       1.7Gi       1.4Gi       3.5Gi       3.5Gi
Swap:          22Gi       3.9Gi        18Gi

What Might be Causing it

I've had this machine for 5 years, but this behaviour started occurring within the past year since the following changes:

  • Upgraded the processor from Intel i5 to Intel Core i7 4790K
  • Upgraded my GPU from an Asus 960 GTX to an EVGA 2070 RTX

Reproducing this behavior is fairly consistent, I wrote a script that spins-up background Python processes that sends requests until the system runs out of memory. I was able to reproduce the system freeze twice in a row doing this.

Troubleshooting & Mitigation

This old thread on LinuxQuestions.org almost exactly mirrors my issue, and I have done the following in attempt to mitigate this issue without any success:

  • Adjust the RAM timing to lower the voltage
  • Replaced all DIMMs with 1600MHz frequency and 1.5 voltage spec (no overclocking)
  • Updated the BIOS firmware

Other things I have tried:

  • S.M.A.R.T. long and short tests of the swap partition
  • fsck scan of the swap partition

System Details

Kernel: Linux 5.4.0-73-generic #82-Ubuntu SMP / x86_64

Disks and Partitions:

NAME                       MAJ:MIN RM   SIZE RO TYPE   MOUNTPOINT
sda                          8:0    0 238.5G  0 disk   
├─sda1                       8:1    0   953M  0 part   /boot/efi
├─sda2                       8:2    0    28G  0 part   /
└─sda3                       8:3    0 209.6G  0 part   /usr
sdb                          8:16   0   1.8T  0 disk   
├─sdb1                       8:17   0  22.4G  0 part   
├─sdb2                       8:18   0 144.4G  0 part   
├─sdb4                       8:20   0   9.3G  0 part   
├─sdb5                       8:21   0   1.7T  0 part   
└─isw_dhciiffhhj_Groovy    253:0    0   1.8T  0 dmraid 
  ├─isw_dhciiffhhj_Groovy1 253:1    0  22.4G  0 part   [SWAP]
  ├─isw_dhciiffhhj_Groovy2 253:2    0 144.4G  0 part   /var
  ├─isw_dhciiffhhj_Groovy4 253:3    0   9.3G  0 part   /srv
  └─isw_dhciiffhhj_Groovy5 253:4    0   1.7T  0 part   /home
sdc                          8:32   0   1.8T  0 disk   
├─sdc1                       8:33   0  22.4G  0 part   
├─sdc2                       8:34   0 144.4G  0 part   
├─sdc4                       8:36   0   9.3G  0 part   
├─sdc5                       8:37   0   1.7T  0 part   
└─isw_dhciiffhhj_Groovy    253:0    0   1.8T  0 dmraid 
  ├─isw_dhciiffhhj_Groovy1 253:1    0  22.4G  0 part   [SWAP]
  ├─isw_dhciiffhhj_Groovy2 253:2    0 144.4G  0 part   /var
  ├─isw_dhciiffhhj_Groovy4 253:3    0   9.3G  0 part   /srv
  └─isw_dhciiffhhj_Groovy5 253:4    0   1.7T  0 part   /home
sdd                          8:48   0 465.8G  0 disk   /opt

System:

H/W path          Device       Class          Description
=========================================================
                               system         All Series (All)
/0                             bus            Z97-PRO GAMER
/0/0                           memory         64KiB BIOS
/0/45                          memory         32GiB System Memory
/0/45/0                        memory         8GiB DIMM DDR3 Synchronous 1333 MHz (0.8 ns)
/0/45/1                        memory         8GiB DIMM DDR3 Synchronous 1333 MHz (0.8 ns)
/0/45/2                        memory         8GiB DIMM DDR3 Synchronous 1333 MHz (0.8 ns)
/0/45/3                        memory         8GiB DIMM DDR3 Synchronous 1333 MHz (0.8 ns)
/0/54                          processor      Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
/0/54/55                       memory         256KiB L1 cache
/0/54/56                       memory         1MiB L2 cache
/0/54/57                       memory         8MiB L3 cache
/0/100                         bridge         4th Gen Core Processor DRAM Controller
/0/100/1                       bridge         Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller
/0/100/1.1                     bridge         Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller
/0/100/1.1/0                   display        TU104 [GeForce RTX 2070 SUPER]
/0/100/1.1/0.1                 multimedia     TU104 HD Audio Controller
/0/100/1.1/0.2                 bus            TU104 USB 3.1 Host Controller
/0/100/1.1/0.2/0  usb5         bus            xHCI Host Controller
/0/100/1.1/0.2/1  usb6         bus            xHCI Host Controller
/0/100/1.1/0.3                 bus            TU104 USB Type-C UCSI Controller
/0/100/14                      bus            9 Series Chipset Family USB xHCI Controller
/0/100/14/0       usb3         bus            xHCI Host Controller
/0/100/14/0/4                  input          Back-UPS NS 1350M2 FW:954.e3 .D USB FW:e3
/0/100/14/0/9                  input          Gaming Mouse G502
/0/100/14/0/a                  input          Corsair K70 RGB Gaming Keyboard
/0/100/14/0/d                  multimedia     Blue Microphones
/0/100/14/0/e                  bus            USB2.0 Hub
/0/100/14/0/e/2                multimedia     Logitech Wireless Headset
/0/100/14/0/e/4                multimedia     C922 Pro Stream Webcam
/0/100/14/1       usb4         bus            xHCI Host Controller
/0/100/16                      communication  9 Series Chipset Family ME Interface #1
/0/100/19         eno1         network        Ethernet Connection (2) I218-V
/0/100/1a                      bus            9 Series Chipset Family USB EHCI Controller #2
/0/100/1a/1       usb1         bus            EHCI Host Controller
/0/100/1a/1/1                  bus            USB hub
/0/100/1b                      multimedia     9 Series Chipset Family HD Audio Controller
/0/100/1c                      bridge         9 Series Chipset Family PCI Express Root Port 1
/0/100/1c.3                    bridge         82801 PCI Bridge
/0/100/1c.3/0                  bridge         ASM1083/1085 PCIe to PCI Bridge
/0/100/1d                      bus            9 Series Chipset Family USB EHCI Controller #1
/0/100/1d/1       usb2         bus            EHCI Host Controller
/0/100/1d/1/1                  bus            USB hub
/0/100/1f                      bridge         Z97 Chipset LPC Controller
/0/100/1f.2                    storage        9 Series Chipset Family SATA Controller [AHCI Mode]
/0/100/1f.3                    bus            9 Series Chipset Family SMBus Controller
/0/1                           system         PnP device PNP0c01
/0/2                           system         PnP device PNP0c02
/0/3                           system         PnP device PNP0b00
/0/4                           generic        PnP device INT3f0d
/0/5                           system         PnP device PNP0c02
/0/6                           system         PnP device PNP0c02
/0/7                           communication  PnP device PNP0501
/0/8                           system         PnP device PNP0c02
/0/9              scsi0        storage        
/0/9/0.0.0        /dev/sda     disk           256GB Samsung SSD 850
/0/9/0.0.0/1      /dev/sda1    volume         952MiB Windows FAT volume
/0/9/0.0.0/2      /dev/sda2    volume         27GiB EFI partition
/0/9/0.0.0/3      /dev/sda3    volume         209GiB EFI partition
/0/a              scsi2        storage        
/0/a/0.0.0        /dev/sdb     disk           2TB ST2000DM001-1ER1
/0/a/0.0.0/1                   volume         22GiB Linux swap volume
/0/a/0.0.0/2                   volume         144GiB EXT4 volume
/0/a/0.0.0/4                   volume         9537MiB EFI partition
/0/a/0.0.0/5                   volume         1686GiB EXT4 volume
/0/b              scsi3        storage        
/0/b/0.0.0        /dev/sdc     disk           2TB ST2000DM001-1ER1
/0/b/0.0.0/1                   volume         22GiB Linux swap volume
/0/b/0.0.0/2                   volume         144GiB EXT4 volume
/0/b/0.0.0/4                   volume         9537MiB EFI partition
/0/b/0.0.0/5                   volume         1686GiB EXT4 volume
/0/c              scsi4        storage        
/0/c/0.0.0        /dev/sdd     volume         465GiB Samsung SSD 860
/1                             power          To Be Filled By O.E.M.
/2                vethc2afe35  network        Ethernet interface

This experience has left me feeling demoralized and deflated, it occurs often enough to significantly impact my productivity. I am tempted to replace the entire system top to bottom but I am suspicious this issue would follow me to the new system too.

2
  • How much RAM? Is it all shown correctly in system? What does this show? free-h My old laptop only had 1.5GB of RAM and if I loaded two large apps, it would go gray for several seconds as it went to swap. Swap is orders of magnitude slower than RAM and generally you do not want to use swap.Its there more to prevent system crash if too much RAM used. Are you loading lots of apps or tabs? Editing very large photos or videos? Not sure what else to check, try 'top` to see whats running. My z97 Asus runs well, but I do not do anything that uses a lot of RAM.
    – oldfred
    Jun 4, 2021 at 19:27
  • Updated my question to answer yours, I've waited for up to 5 / 10 minutes for the system to unfreeze without luck. I'm a developer so I typically have a lot of browser tabs open, developer tools, background Java processes, and IntelliJ uses ~5GB alone. My PC will run fine as long as I don't touch swap, I think as long as a month or two until I need to restart for other reasons.
    – Coder Guy
    Jun 4, 2021 at 22:59

1 Answer 1

0

System "freezes" are often caused by running too many, too large programs and running out of available memory. Use free to see if you have swap space, read man mkswap swapon fstab to create some. Swap space must be contiguous. use mkswap or fallocate, not dd. Traditionally, swap space of 1.5 × RAM has been recommended, but YMMV. If you don't plan to hibernate your system, you can have less than 1.0 × RAM.

1
  • I'm going to update my question with the output of free, and I didn't notice the mkswap -c option, I will try that later.
    – Coder Guy
    Jun 4, 2021 at 22:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.