Over the course of about a week or so, our Ubuntu 16.04 server's swap space fills up and never releases it, causing other cascading problems on the server. Namely, an instance of Tomcat running Solr will periodically fail.
The server is a four core VM, with
16g RAM, and
16g swap. Currently has a
swappiness value of
We run this server fairly hard, with a large amount of internal HTTP calls to local servers for image processing, etc., much of that through Apache. As we have been encountering this problem of swap space slowly filling up and never releasing, we have streamlined other areas, but to no apparent effect. A reboot fixes things, until it fills back up.
I could attach graphs numbers of the swap declining, but not sure how helpful that would be (been tracking with nmon). For example, after a reboot yesterday our
free -m output is:
total used free shared buff/cache available Mem: 16047 4888 7621 32 3536 10729 Swap: 16380 0 16380
As can be seen,
0 used for swap. But once the system dips into it -- for reasons still yet to be determined -- it never really recovers and you get a decline like this:
And jump forward a few days until it bottoms out around 1:30pm:
Any suggestions? or advice on how to continue investigating?
Based on some feedback in comments, updating question a bit.
Doesn't appear to be heap size for Tomcat, our current JVM settings for Tomcat are
JAVA_OPTS=' -Xms4096m -Xmx4096m', which we've bumped up considerably (usually run around 2g). Tomcat's
catalina.out is telling us
Native memory allocation (mmap) failed to map 2863661056 bytes for committing reserved memory.