This is not so much a performance problem as it is a desire to understand what is happening and how it works. I have a system with lots of resources, including 128 GB of RAM. What I have discovered (which happens every time given enough up time) is that the system will start using the entire swap space available to it when there is still 86 GB of memory left.
Here is a screenshot of htop demonstrating the system state:
Swappiness is set to 5:
$ sysctl vm.swappiness
vm.swappiness = 5
vmstat
shows mostly 0s, but one event:
$ vmstat 60
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 8305756 5218320 31872 83460448 0 0 0 1411 11432 17991 1 1 98 0 0
2 0 8305616 5042872 31872 83571392 66 0 1193 2068 12097 18650 2 1 98 0 0
System details:
Key | Value |
---|---|
Linux Distro | Fedora 37 |
Kernel version | 6.4.11-100.fc37.x86_64 |
Top processes using swap (sorted highest at the bottom)
PID User Command Swap USS PSS RSS
...
2835417 ben /usr/lib64/firefox/firefox 60104 703736 710125 818252
2835448 ben /usr/lib64/firefox/firefox 63260 1179992 1185457 1288392
2833473 ben /opt/google/chrome/chrome - 64748 32896 34897 146000
3090226 100998 /usr/lib/chromium/chromium 67612 101820 104321 136248
2835394 ben /usr/lib64/firefox/firefox 72476 1177840 1187600 1295372
12678 100998 /usr/local/bin/python /usr/ 75788 330820 330927 332536
3682935 ben /opt/google/chrome/chrome - 87196 207280 210962 350268
1516354 ben /home/ben/.rubies/ruby-3.0. 91728 28868 29182 32820
3621825 ben /opt/google/chrome/chrome - 108724 175184 178064 310136
2833257 ben /opt/google/chrome/chrome 110132 513256 545048 710296
599920 ben /usr/lib64/erlang/erts-13.2 113620 27728 28951 34700
1281109 ben /opt/google/chrome/chrome - 139388 24860 27513 153668
1013489 qemu /usr/bin/qemu-system-x86_64 774020 16138492 16138790 16150960
1013519 root /usr/libexec/virtiofsd --fd 816020 24 154 1880
2863657 ben /usr/lib/slack/slack 1504064 124068 143980 222196
EDIT - January 2024: Still don't have the answer, but adding some additional information that I've picked up from continued usage. The most common processes that seem to accumulate swap are usually the same, and are KVM/qemu related, and Google Chrome/Electron related
for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | less