I run a medium VPS (4GB RAM) as a web server. I host some private Ruby on Rails sites with Apache and mod_passenger. I also have two legacy apps which run with "passenger standalone" (two Nginx servers with passenger on ports 3000 and 4000). Apache acts as a proxy for those.
Most of the time this is works okay, but every few months the whole system goes berserk:
- RAM usage goes up to 100%
- the log files get filled with "Cannot allocate memory" & reverse proxy errors
- soon enough the whole system is unresponsive
Sometimes this clears up by itself, sometimes I have to ask my provider to reboot the system because I can't even SSH into it.
This always happens at night, when traffic is very low. Here are two graphs:
How do I get to the bottom of this? Which steps can I take to debug this situation? Is it possible that I have a bad neighbor on the machine?
Other processes on the major machine: MySQL, PostgreSQL, Elasticsearch, Redis
UPDATE to clarify
I'm looking for a tool that records processes with their memory & cpu consumption so I can track down the trouble maker(s). Is something like that out there or do I have to roll my own?
psoutput. Run it every two seconds over night and see what's listed as the highest memory consumer(s). Once you figure out the offending process you can look deeper into it to see what it's doing.