We have a Linux server running Debian 4.0.5 (Kernel 4.0.0-2) with 32G RAM installed and 16G Swap configured. The system uses lxc containers for compartmentalisation, but that shouldn't matter here. The issue exists inside and out of different containers.
Here's a typical
total used free shared buff/cache available Mem: 28G 2.1G 25G 15M 936M 26G Swap: 15G 1.4G 14G
Committed_AS: 12951172 kB
So there's plenty of free memory, even if everything allocated was actually used at once. However, the system is instantly paging even running processes.
This is most notable with Gitlab, a Rails application using Unicorn: newly forked Unicorn workers are instantly swapped, and when a request comes in need to be read from disk at ~1400kB/s (data from
iotop) and runs into timeouts (30s for now, to get it restarted in time. No normal request should take more than 5s) before it gets loaded into memory completely, thus getting instantly killed. Note that this is just an example, I have seen this happen to redis, amavis, postgres, mysql, java(openjdk) and others.
The system is otherwise in a low-load situation with about 5% CPU utilization and a loadavg around 2 (on 8 cores).
What we tried (in no particular order):
swapoff -a: fails at about 800M still swapped
- Reducing swappiness (in steps) using
sysctl vm.swappiness=NN. This seems to have no impact at all, we went down to 0% and still exactly the same behaviour exists
- Stopping non-essential services (Gitlab, a Jetty-based webapp...), freeing ca. 8G of committed-but-not-mapped memory and bringing Committed_AS down to about 5G. No change at all.
- Clearing system caches using
sync && echo 3 > /proc/sys/vm/drop_caches. This frees up memory, but does nothing to the swap situation.
- Combinations of the above
Restarting the machine to completely disable swap via fstab as a test is not really an option, as some services have availability issues and need planned downtimes, not "poking around"... and also we don't really want to disable swap as a fallback.
I don't see why there is any swapping occuring here. Any ideas what may be going on?
This problem has existed for a while now, but it showed up first during a period of high IO load (long background data processing task), so I can't pinpoint a specific event. This task is done for some days and the problem persists, hence this question.