Over the course of about a week or so, our Ubuntu 16.04 server's swap space fills up and never releases it, causing other cascading problems on the server. Namely, an instance of Tomcat running Solr will periodically fail.
The server is a four core VM, with 16g
RAM, and 16g
swap. Currently has a swappiness
value of 60
.
We run this server fairly hard, with a large amount of internal HTTP calls to local servers for image processing, etc., much of that through Apache. As we have been encountering this problem of swap space slowly filling up and never releasing, we have streamlined other areas, but to no apparent effect. A reboot fixes things, until it fills back up.
I could attach graphs numbers of the swap declining, but not sure how helpful that would be (been tracking with nmon). For example, after a reboot yesterday our free -m
output is:
total used free shared buff/cache available
Mem: 16047 4888 7621 32 3536 10729
Swap: 16380 0 16380
As can be seen, 0
used for swap. But once the system dips into it -- for reasons still yet to be determined -- it never really recovers and you get a decline like this:
And jump forward a few days until it bottoms out around 1:30pm:
Any suggestions? or advice on how to continue investigating?
Update Question
Based on some feedback in comments, updating question a bit.
Doesn't appear to be heap size for Tomcat, our current JVM settings for Tomcat are JAVA_OPTS=' -Xms4096m -Xmx4096m'
, which we've bumped up considerably (usually run around 2g). Tomcat's catalina.out
is telling us Native memory allocation (mmap) failed to map 2863661056 bytes for committing reserved memory.
ps
says about the memory usage of various services, and consider if you can restart just those from time to time. (Or, preferably, fix the leaks.) – ilkkachu Jun 15 '18 at 12:38ps -aux
at time of failures – ghukill Jun 15 '18 at 12:42top -o VIRT
and see if any process becomes really large. – Mark Plotnick Jun 15 '18 at 14:58VIRT
: getting what seem like huge numbers for Tomcat -->0.25t
then fell slightly to99.696g
, as I watch up to.409t
. Been using Tomcat for a long time, but never paid much attention toVIRT
, are these in the realm of normal? – ghukill Jun 15 '18 at 15:07