I have a real-time networking application that is extremely sensitive to any kind of "blocking" and needs to be extremely responsive most of the time with perfect latency.
The app (multithreaded) itself is unfortunately quite memory hungry, it needs about 6GB of RAM.
The system had 8GB RAM with 2GB swap. I observed some performance issues with increased app latency that I connected to swapping events (OS is monitored using grafana and node_exporter every 15 seconds), exactly in same time as bad latency happened, the OS was swapping out some pages (while still having 2GB available memory).
I understand Linux kernel VM subsystem fairly well (I read Mel Gorman's "Understanding the Linux® Virtual Memory Manager" book) so I am associating this with the OS proactively swapping out inactive pages to get more space for FS cache / buffers, which I wouldn't really mind, if it wasn't crippling the latency of this application.
vm.swappiness = 0 in order to get kernel to use swap only as last resort when low mem watermark is reached, but to no avail - it keeps swapping out tiny chunks of memory approx. every 30 minutes, crippling the latency of application.
Only thing that helped was disabling swap entirely - ever since I did that latency issues never reoccurred. But that is also a bit of problem, because if some unfortunate situation happened when app would temporarily need more memory approaching the 8GB of total usage, OOM killer would now kill it, instead of swapping out portion of inactive pages, which would be even worse scenario than crippled latency.
So I have 2 questions:
- Why are these small swapout events crippling the latency of application? My understanding is that swapper is evicting inactive memory pages anyway - is the process frozen when swapper moves its memory, even if it doesn't try to access the pages that aren't in RAM or that are being moved?
- Is there any way to enforce kernel to swap ONLY when necessary? Eg. - don't swap when > 1GB of RAM is still available?