Overview & Symptoms
My system freezes almost completely whenever my system runs out of RAM and starts hitting the swap partition heavily. Everything freezes including the mouse and keyboard with a few exceptions:
- The hard drive light appears to indicate some background activity
- The fan sometimes spins up and down indicating some CPU activity
nmap -sT(TCP handshake) from another machine reveals open ports indicating that the NIC is responding at the OSI transport layer
Nothing is logged indicating what causes this.
On one rare occasion I remember the mouse was able to move a bit after about a minute or two of the system being frozen. This issue does not appear to occur whenever there is plenty of free RAM available, it only seems to occur when the swap partition starts experiencing significant load.
Here is the output of free that indicates free RAM and swap storage, right now there is mild swap usage. This is typically entering the danger zone where the system would freeze, although I've witnessed up to 12MB of swap used without an issue.
Total RAM: 32GB
Total Swap: 24GB
total used free shared buff/cache available
Mem: 31Gi 26Gi 1.7Gi 1.4Gi 3.5Gi 3.5Gi
Swap: 22Gi 3.9Gi 18Gi
What Might be Causing it
I've had this machine for 5 years, but this behaviour started occurring within the past year since the following changes:
- Upgraded the processor from Intel i5 to Intel Core i7 4790K
- Upgraded my GPU from an Asus 960 GTX to an EVGA 2070 RTX
Reproducing this behavior is fairly consistent, I wrote a script that spins-up background Python processes that sends requests until the system runs out of memory. I was able to reproduce the system freeze twice in a row doing this.
Troubleshooting & Mitigation
This old thread on LinuxQuestions.org almost exactly mirrors my issue, and I have done the following in attempt to mitigate this issue without any success:
- Adjust the RAM timing to lower the voltage
- Replaced all DIMMs with 1600MHz frequency and 1.5 voltage spec (no overclocking)
- Updated the BIOS firmware
Other things I have tried:
- S.M.A.R.T. long and short tests of the swap partition
fsckscan of the swap partition
System Details
Kernel: Linux 5.4.0-73-generic #82-Ubuntu SMP / x86_64
Disks and Partitions:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 238.5G 0 disk
├─sda1 8:1 0 953M 0 part /boot/efi
├─sda2 8:2 0 28G 0 part /
└─sda3 8:3 0 209.6G 0 part /usr
sdb 8:16 0 1.8T 0 disk
├─sdb1 8:17 0 22.4G 0 part
├─sdb2 8:18 0 144.4G 0 part
├─sdb4 8:20 0 9.3G 0 part
├─sdb5 8:21 0 1.7T 0 part
└─isw_dhciiffhhj_Groovy 253:0 0 1.8T 0 dmraid
├─isw_dhciiffhhj_Groovy1 253:1 0 22.4G 0 part [SWAP]
├─isw_dhciiffhhj_Groovy2 253:2 0 144.4G 0 part /var
├─isw_dhciiffhhj_Groovy4 253:3 0 9.3G 0 part /srv
└─isw_dhciiffhhj_Groovy5 253:4 0 1.7T 0 part /home
sdc 8:32 0 1.8T 0 disk
├─sdc1 8:33 0 22.4G 0 part
├─sdc2 8:34 0 144.4G 0 part
├─sdc4 8:36 0 9.3G 0 part
├─sdc5 8:37 0 1.7T 0 part
└─isw_dhciiffhhj_Groovy 253:0 0 1.8T 0 dmraid
├─isw_dhciiffhhj_Groovy1 253:1 0 22.4G 0 part [SWAP]
├─isw_dhciiffhhj_Groovy2 253:2 0 144.4G 0 part /var
├─isw_dhciiffhhj_Groovy4 253:3 0 9.3G 0 part /srv
└─isw_dhciiffhhj_Groovy5 253:4 0 1.7T 0 part /home
sdd 8:48 0 465.8G 0 disk /opt
System:
H/W path Device Class Description
=========================================================
system All Series (All)
/0 bus Z97-PRO GAMER
/0/0 memory 64KiB BIOS
/0/45 memory 32GiB System Memory
/0/45/0 memory 8GiB DIMM DDR3 Synchronous 1333 MHz (0.8 ns)
/0/45/1 memory 8GiB DIMM DDR3 Synchronous 1333 MHz (0.8 ns)
/0/45/2 memory 8GiB DIMM DDR3 Synchronous 1333 MHz (0.8 ns)
/0/45/3 memory 8GiB DIMM DDR3 Synchronous 1333 MHz (0.8 ns)
/0/54 processor Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
/0/54/55 memory 256KiB L1 cache
/0/54/56 memory 1MiB L2 cache
/0/54/57 memory 8MiB L3 cache
/0/100 bridge 4th Gen Core Processor DRAM Controller
/0/100/1 bridge Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller
/0/100/1.1 bridge Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller
/0/100/1.1/0 display TU104 [GeForce RTX 2070 SUPER]
/0/100/1.1/0.1 multimedia TU104 HD Audio Controller
/0/100/1.1/0.2 bus TU104 USB 3.1 Host Controller
/0/100/1.1/0.2/0 usb5 bus xHCI Host Controller
/0/100/1.1/0.2/1 usb6 bus xHCI Host Controller
/0/100/1.1/0.3 bus TU104 USB Type-C UCSI Controller
/0/100/14 bus 9 Series Chipset Family USB xHCI Controller
/0/100/14/0 usb3 bus xHCI Host Controller
/0/100/14/0/4 input Back-UPS NS 1350M2 FW:954.e3 .D USB FW:e3
/0/100/14/0/9 input Gaming Mouse G502
/0/100/14/0/a input Corsair K70 RGB Gaming Keyboard
/0/100/14/0/d multimedia Blue Microphones
/0/100/14/0/e bus USB2.0 Hub
/0/100/14/0/e/2 multimedia Logitech Wireless Headset
/0/100/14/0/e/4 multimedia C922 Pro Stream Webcam
/0/100/14/1 usb4 bus xHCI Host Controller
/0/100/16 communication 9 Series Chipset Family ME Interface #1
/0/100/19 eno1 network Ethernet Connection (2) I218-V
/0/100/1a bus 9 Series Chipset Family USB EHCI Controller #2
/0/100/1a/1 usb1 bus EHCI Host Controller
/0/100/1a/1/1 bus USB hub
/0/100/1b multimedia 9 Series Chipset Family HD Audio Controller
/0/100/1c bridge 9 Series Chipset Family PCI Express Root Port 1
/0/100/1c.3 bridge 82801 PCI Bridge
/0/100/1c.3/0 bridge ASM1083/1085 PCIe to PCI Bridge
/0/100/1d bus 9 Series Chipset Family USB EHCI Controller #1
/0/100/1d/1 usb2 bus EHCI Host Controller
/0/100/1d/1/1 bus USB hub
/0/100/1f bridge Z97 Chipset LPC Controller
/0/100/1f.2 storage 9 Series Chipset Family SATA Controller [AHCI Mode]
/0/100/1f.3 bus 9 Series Chipset Family SMBus Controller
/0/1 system PnP device PNP0c01
/0/2 system PnP device PNP0c02
/0/3 system PnP device PNP0b00
/0/4 generic PnP device INT3f0d
/0/5 system PnP device PNP0c02
/0/6 system PnP device PNP0c02
/0/7 communication PnP device PNP0501
/0/8 system PnP device PNP0c02
/0/9 scsi0 storage
/0/9/0.0.0 /dev/sda disk 256GB Samsung SSD 850
/0/9/0.0.0/1 /dev/sda1 volume 952MiB Windows FAT volume
/0/9/0.0.0/2 /dev/sda2 volume 27GiB EFI partition
/0/9/0.0.0/3 /dev/sda3 volume 209GiB EFI partition
/0/a scsi2 storage
/0/a/0.0.0 /dev/sdb disk 2TB ST2000DM001-1ER1
/0/a/0.0.0/1 volume 22GiB Linux swap volume
/0/a/0.0.0/2 volume 144GiB EXT4 volume
/0/a/0.0.0/4 volume 9537MiB EFI partition
/0/a/0.0.0/5 volume 1686GiB EXT4 volume
/0/b scsi3 storage
/0/b/0.0.0 /dev/sdc disk 2TB ST2000DM001-1ER1
/0/b/0.0.0/1 volume 22GiB Linux swap volume
/0/b/0.0.0/2 volume 144GiB EXT4 volume
/0/b/0.0.0/4 volume 9537MiB EFI partition
/0/b/0.0.0/5 volume 1686GiB EXT4 volume
/0/c scsi4 storage
/0/c/0.0.0 /dev/sdd volume 465GiB Samsung SSD 860
/1 power To Be Filled By O.E.M.
/2 vethc2afe35 network Ethernet interface
This experience has left me feeling demoralized and deflated, it occurs often enough to significantly impact my productivity. I am tempted to replace the entire system top to bottom but I am suspicious this issue would follow me to the new system too.
free-hMy old laptop only had 1.5GB of RAM and if I loaded two large apps, it would go gray for several seconds as it went to swap. Swap is orders of magnitude slower than RAM and generally you do not want to use swap.Its there more to prevent system crash if too much RAM used. Are you loading lots of apps or tabs? Editing very large photos or videos? Not sure what else to check, try 'top` to see whats running. My z97 Asus runs well, but I do not do anything that uses a lot of RAM.