I have a server running Red Hat Enterprise Linux Server release 6.7. The server runs a number of java processes including a tomcat server with nginx webserver. The server has 12 GB of ram allocated to it.
Upon immediate startup the server shows to be using 1289 MB of ram (free -m -/+ buffers/cache line)... output below.
A "ps aux" on server startup shows the memory usage of all processes being a total of 1,290,996 k of memory (The RSS column)... The 2 numbers are pretty close.
After 2 days of uptime free -m now shows us using 10,698 MB of memory while "ps aux" totals up to 8,345,352 k for all processes.
We've attempted to stop/kill each process one by one and even with an absolute bare minimum number of things running (No Tomcat, no Java, no Nginx) we never recover the "Missing" ~2GB of ram.. The only thing we've found to recover the missing memory is to reboot the server. If the server goes more than 3-4 days without a reboot it starts eating into swap space which eventually causes high CPU usage and the java applications running on it to crawl to a standstill.
I should also mention that this server is running as a virtual machine using VMWare.
Free -m output on startup:
total used free shared buffers cached Mem: 11852 1787 10064 0 28 468 -/+ buffers/cache: 1289 10562 Swap: 8191 0 8191
Free -m output after 2 days uptime
total used free shared buffers cached Mem: 11852 11625 226 0 146 779 -/+ buffers/cache: 10698 1153 Swap: 8191 43 8148
What can I do to help diagnose where the "Missing" memory goes to so that I can resolve the issue. Any suggestions are appreciated.