1

I have a real-time networking application that is extremely sensitive to any kind of "blocking" and needs to be extremely responsive most of the time with perfect latency.

The app (multithreaded) itself is unfortunately quite memory hungry, it needs about 6GB of RAM.

The system had 8GB RAM with 2GB swap. I observed some performance issues with increased app latency that I connected to swapping events (OS is monitored using grafana and node_exporter every 15 seconds), exactly in same time as bad latency happened, the OS was swapping out some pages (while still having 2GB available memory).

I understand Linux kernel VM subsystem fairly well (I read Mel Gorman's "Understanding the Linux® Virtual Memory Manager" book) so I am associating this with the OS proactively swapping out inactive pages to get more space for FS cache / buffers, which I wouldn't really mind, if it wasn't crippling the latency of this application.

I set vm.swappiness = 0 in order to get kernel to use swap only as last resort when low mem watermark is reached, but to no avail - it keeps swapping out tiny chunks of memory approx. every 30 minutes, crippling the latency of application.

Only thing that helped was disabling swap entirely - ever since I did that latency issues never reoccurred. But that is also a bit of problem, because if some unfortunate situation happened when app would temporarily need more memory approaching the 8GB of total usage, OOM killer would now kill it, instead of swapping out portion of inactive pages, which would be even worse scenario than crippled latency.

So I have 2 questions:

  • Why are these small swapout events crippling the latency of application? My understanding is that swapper is evicting inactive memory pages anyway - is the process frozen when swapper moves its memory, even if it doesn't try to access the pages that aren't in RAM or that are being moved?
  • Is there any way to enforce kernel to swap ONLY when necessary? Eg. - don't swap when > 1GB of RAM is still available?
0

The app (multithreaded) itself is unfortunately quite memory hungry, it needs about 6GB of RAM.
(...)
The system had 8GB RAM

Your system is short in memory; I think the only solution to your problem would be to add some RAM.

Keep in mind that pages swapped out of memory are written to disk, which offers much slower read/write access than RAM. This must be why your application is slowing down.

Linux has been observed to swap even when swappiness is set to 0. Related question: Does Linux perform "opportunistic swapping", or is it a myth?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.