Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I run a medium VPS (4GB RAM) as a web server. I host some private Ruby on Rails sites with Apache and mod_passenger. I also have two legacy apps which run with "passenger standalone" (two Nginx servers with passenger on ports 3000 and 4000). Apache acts as a proxy for those.

Most of the time this is works okay, but every few months the whole system goes berserk:

  • RAM usage goes up to 100%
  • the log files get filled with "Cannot allocate memory" & reverse proxy errors
  • soon enough the whole system is unresponsive

Sometimes this clears up by itself, sometimes I have to ask my provider to reboot the system because I can't even SSH into it.

This always happens at night, when traffic is very low. Here are two graphs:

Apache accesses

enter image description here

Memory consumption

enter image description here

How do I get to the bottom of this? Which steps can I take to debug this situation? Is it possible that I have a bad neighbor on the machine?

Other processes on the major machine: MySQL, PostgreSQL, Elasticsearch, Redis

UPDATE to clarify

I'm looking for a tool that records processes with their memory & cpu consumption so I can track down the trouble maker(s). Is something like that out there or do I have to roll my own?

share|improve this question
1  
Probably a cron job. Check /etc/cron.daily/ and /etc/cron.d/. –  bahamat Jan 2 at 8:39
    
Yes, I had that suspicion too, but I found nothing out of the ordinary: MySQL dumper, munin, etc. –  Wukerplank Jan 2 at 9:32
    
There doesn't have to be anything out of the ordinary, just that there is something wrong that causes those jobs to fail occasionally and do this. I also suspect the cron jobs for something that occurs always at the same time. –  Chris Down Jan 2 at 9:37
    
OK, assuming a cron job messes up - how do I find the culprit? — I've updated my question. –  Wukerplank Jan 2 at 10:00
    
Start with a log of ps output. Run it every two seconds over night and see what's listed as the highest memory consumer(s). Once you figure out the offending process you can look deeper into it to see what it's doing. –  bahamat Jan 2 at 20:52
add comment

1 Answer

up vote 1 down vote accepted

From my personal experience of managing RoR apps, memory leaks can easily creep in and can catch you at most unexpected time. I would recommend doing two things to catch the culprit:

  1. Use sar to record the system activities.
  2. Since memory is specific target, use free | grep "Mem:" | awk '{print $4}' and send out a mail to yourself using mailx if the threshold exceeds a certain limit of your choice.

Usually there is a bad sql query or a rouge gem that cause this. If anything has recently changed with respect to these, then you may want to look into that as well.

share|improve this answer
    
Thanks! I didn't know sar, I'll look into it. For now I'm dumping ps outputs into a file, but I will pick up your idea with mailx. –  Wukerplank Jan 3 at 9:51
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.