CPU overheats and PC shuts down when swap is full

Question

This doesn't necessarily have to be a Linux problem but I'll ask it here anyway. I'm using a workstation mainly for training deep learning and machine learning models. I run training codes on both CPU and GPU.

CPU: AMD Ryzen 9 5950X 16-Core Processor

GPU: NVIDIA GeForce RTX 3090

OS: Ubuntu 22.04 LTS

The libraries that I use (PyTorch, XGBoost, LightGBM and etc.) utilize swap memory a lot for data loading. While working on big datasets, swap memory accumulates slowly and exceeds the limit (2GB). When that happens, all of the cores go crazy and CPU overheats. Workstation shuts down itself couple seconds later.

I'm a data scientist and I'm not good with hardware. It took couple weeks for me to figure out why my workstation was keep shutting itself down. I have to find a way to prevent this since I can't progress on my own tasks anymore. What are your suggestions?

To give you more details, this wasn't happening 3-4 months ago. It started very recently.

Edit: Added nvidia-smi and sensors outputs while training two models (UNet and YOLOv6) simultaneously.

nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.73.05    Driver Version: 510.73.05    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:0A:00.0 Off |                  N/A |
|100%   79C    P2   338W / 350W |  14171MiB / 24576MiB |    100%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1361      G   /usr/lib/xorg/Xorg                 56MiB |
|    0   N/A  N/A      1568      G   /usr/bin/gnome-shell               10MiB |
|    0   N/A  N/A     27955      C   python                           2743MiB |
|    0   N/A  N/A     31692      C   python                          11355MiB |
+-----------------------------------------------------------------------------+

sensors

nvme-pci-0300
Adapter: PCI adapter
Composite:    +74.8°C  (low  = -273.1°C, high = +84.8°C)
                       (crit = +84.8°C)
Sensor 1:     +74.8°C  (low  = -273.1°C, high = +65261.8°C)
Sensor 2:     +74.8°C  (low  = -273.1°C, high = +65261.8°C)

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1:        +57.0°C  

k10temp-pci-00c3
Adapter: PCI adapter
Tctl:         +87.8°C  
Tccd1:        +89.2°C  
Tccd2:        +79.5°C

Answer 1

First, absolutely make sure your PSU is powerful enough - instant shutdowns like yours could indicate an issue with it. Maybe replace it. RTX 3090 can have spikes up to 500W and that means, along with your CPU, that your PSU must be rated at the very least 850W.

---

Speaking of your temps.

Your CPU is running close to its rated maximum, which is 90C, which means you'd better improve your case cooling by installing case fans e.g. 120mm (140mm are beter - quieter and more powerful) and probably installing a better cooler on your CPU along with changing thermal paste - my preferred one is Arctic MX-4 (MX-5 in theory provides better performance but it's a lot more cumbersome to apply).

Installing proper case cooling might prove enough since your GPU is definitely increasing your CPU temps.

Don't forget to update your EFI BIOS as well.

You can also use a software only solution: enter your BIOS and

• either decrease your CPU PPT (maximum wattage)
• or set the maximum temperature for it, e.g. 85C

Both will result in decreased multithreaded performance but not so much. You may get more help here: https://www.reddit.com/r/Amd/

Answer 2

Presuming for the sake of discussion that the CPUs are overheating, that is a cooling issue, not a memory management issue. It's hypothetically possible that the additional I/O your system storage has to do when swap gets full and the system has to do more moving of data between swap and RAM is making the storage hardware toasty enough to overwhelm the thermal management hardware. Monitor your system temperature rather than system load when under this computational load to correlate shutdown with a temperature spike rather than a computational one. Ensure that your system fans are working cooperatively to keep air flowing rather than all pulling air into or exhausting air from the chassis. Ensure the heat sinks are properly installed.

Answer 3

I'll answer my own question to tell you how I solved this problem. It might help others to check stuff like this before jumping into conclusions.

Yes, it was a cooling problem. I found that a motherboard fan connector was loose. Fans were working but when the CPU was overheating, that information wasn't passed to fans. When I plugged that connector back, temperatures got back to normal.