I run a medium VPS (4GB RAM) as a web server. I host some private Ruby on Rails sites with Apache and mod_passenger. I also have two legacy apps which run with "passenger standalone" (two Nginx servers with passenger on ports 3000 and 4000). Apache acts as a proxy for those.
Most of the time this is works okay, but every few months the whole system goes berserk:
- RAM usage goes up to 100%
- the log files get filled with "Cannot allocate memory" & reverse proxy errors
- soon enough the whole system is unresponsive
Sometimes this clears up by itself, sometimes I have to ask my provider to reboot the system because I can't even SSH into it.
This always happens at night, when traffic is very low. Here are two graphs:
How do I get to the bottom of this? Which steps can I take to debug this situation? Is it possible that I have a bad neighbor on the machine?
Other processes on the major machine: MySQL, PostgreSQL, Elasticsearch, Redis
UPDATE to clarify
I'm looking for a tool that records processes with their memory & cpu consumption so I can track down the trouble maker(s). Is something like that out there or do I have to roll my own?