10

This doesn't necessarily have to be a Linux problem but I'll ask it here anyway. I'm using a workstation mainly for training deep learning and machine learning models. I run training codes on both CPU and GPU.

CPU: AMD Ryzen 9 5950X 16-Core Processor

GPU: NVIDIA GeForce RTX 3090

OS: Ubuntu 22.04 LTS

The libraries that I use (PyTorch, XGBoost, LightGBM and etc.) utilize swap memory a lot for data loading. While working on big datasets, swap memory accumulates slowly and exceeds the limit (2GB). When that happens, all of the cores go crazy and CPU overheats. Workstation shuts down itself couple seconds later.

I'm a data scientist and I'm not good with hardware. It took couple weeks for me to figure out why my workstation was keep shutting itself down. I have to find a way to prevent this since I can't progress on my own tasks anymore. What are your suggestions?

To give you more details, this wasn't happening 3-4 months ago. It started very recently.

Edit: Added nvidia-smi and sensors outputs while training two models (UNet and YOLOv6) simultaneously.

nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.73.05    Driver Version: 510.73.05    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:0A:00.0 Off |                  N/A |
|100%   79C    P2   338W / 350W |  14171MiB / 24576MiB |    100%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1361      G   /usr/lib/xorg/Xorg                 56MiB |
|    0   N/A  N/A      1568      G   /usr/bin/gnome-shell               10MiB |
|    0   N/A  N/A     27955      C   python                           2743MiB |
|    0   N/A  N/A     31692      C   python                          11355MiB |
+-----------------------------------------------------------------------------+

sensors

nvme-pci-0300
Adapter: PCI adapter
Composite:    +74.8°C  (low  = -273.1°C, high = +84.8°C)
                       (crit = +84.8°C)
Sensor 1:     +74.8°C  (low  = -273.1°C, high = +65261.8°C)
Sensor 2:     +74.8°C  (low  = -273.1°C, high = +65261.8°C)

iwlwifi_1-virtual-0
Adapter: Virtual device
temp1:        +57.0°C  

k10temp-pci-00c3
Adapter: PCI adapter
Tctl:         +87.8°C  
Tccd1:        +89.2°C  
Tccd2:        +79.5°C
17
  • 11
    Being somehow surprised reading the swap "exceeding the limit"… if you are confident in the cause to effect relationship, what about… increasing swap space. Under linux, you can do that easily at run time using mkswap and swapon utilities.
    – MC68020
    Jun 29 at 13:54
  • 6
    Yes, exactly, so if the problem only occurs when swap is full, wouldn't adding more swap solve the issue? You can also try cleaning the machine: literally cleaning, remove dust.
    – terdon
    Jun 29 at 14:03
  • 6
    @gunesevitan why would you want an alternative? Adding swap is trivial, it's just editing one file (/etc/fstab) and running one command to generate a new swapfile. It should take around 5 minutes, at most. If that is the problem, then you're very lucky. It's just extremely unlikely that this is the problem. It's far more likely that when you are swapping is when you are doing most of the number crunching, so your CPU is overheating and the swapping is incidental.
    – terdon
    Jun 29 at 15:06
  • 4
    Is the temperature you show the very last value just before the crash? If not, please try and get the last value or as close as you can. You could try something like while true; do date > log; sensors >> log; sleep 1; done to run sensors every second and save the output. Then, when you crash and reboot you can see what the temperature was at most a second before the crash.
    – terdon
    Jun 29 at 15:56
  • 6
    Your CPU is way too hot. Again, check your fan/thermal paste. Apply something like Arctic MX4 if you're using a noname vendor thermal paste. The GPU temperature is tolerable. Your SSD is too hot. Looks like you've got a closed case with no decent cooling. Either remove the left panel or install 120/140mm case fans. With decent case cooling you may as well not touch the CPU at all. Jun 29 at 16:13

3 Answers 3

20

First, absolutely make sure your PSU is powerful enough - instant shutdowns like yours could indicate an issue with it. Maybe replace it. RTX 3090 can have spikes up to 500W and that means, along with your CPU, that your PSU must be rated at the very least 850W.


Speaking of your temps.

Your CPU is running close to its rated maximum, which is 90C, which means you'd better improve your case cooling by installing case fans e.g. 120mm (140mm are beter - quieter and more powerful) and probably installing a better cooler on your CPU along with changing thermal paste - my preferred one is Arctic MX-4 (MX-5 in theory provides better performance but it's a lot more cumbersome to apply).

Installing proper case cooling might prove enough since your GPU is definitely increasing your CPU temps.

Don't forget to update your EFI BIOS as well.

You can also use a software only solution: enter your BIOS and

  • either decrease your CPU PPT (maximum wattage)
  • or set the maximum temperature for it, e.g. 85C

Both will result in decreased multithreaded performance but not so much. You may get more help here: https://www.reddit.com/r/Amd/

1
  • Thank you. I accepted this as the answer since it is more detailed than the other one. Jun 30 at 8:12
13

Presuming for the sake of discussion that the CPUs are overheating, that is a cooling issue, not a memory management issue. It's hypothetically possible that the additional I/O your system storage has to do when swap gets full and the system has to do more moving of data between swap and RAM is making the storage hardware toasty enough to overwhelm the thermal management hardware. Monitor your system temperature rather than system load when under this computational load to correlate shutdown with a temperature spike rather than a computational one. Ensure that your system fans are working cooperatively to keep air flowing rather than all pulling air into or exhausting air from the chassis. Ensure the heat sinks are properly installed.

3
  • I added sensors output. What's your opinion? Jun 29 at 15:48
  • 1
    @gunesevitan too hot. This answer is entirely right. You have a hardware problem. It might be exacerbated by your workload, but it's a hardware problem.
    – hobbs
    Jun 30 at 2:38
  • It may be a cooling issue, but it's also a memory-management issue. Linux has an unfortunate habit of going crazy when it's almost out of memory. (Actually running out of memory invokes the OOM killer and lets the system recover quickly, but it's not uncommon for the system to be unusably slow for hours or days before memory actually runs out.)
    – Mark
    Jun 30 at 23:26
1

I'll answer my own question to tell you how I solved this problem. It might help others to check stuff like this before jumping into conclusions.

Yes, it was a cooling problem. I found that a motherboard fan connector was loose. Fans were working but when the CPU was overheating, that information wasn't passed to fans. When I plugged that connector back, temperatures got back to normal.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.