1

I run a medium VPS (4GB RAM) as a web server. I host some private Ruby on Rails sites with Apache and mod_passenger. I also have two legacy apps which run with "passenger standalone" (two Nginx servers with passenger on ports 3000 and 4000). Apache acts as a proxy for those.

Most of the time this is works okay, but every few months the whole system goes berserk:

  • RAM usage goes up to 100%
  • the log files get filled with "Cannot allocate memory" & reverse proxy errors
  • soon enough the whole system is unresponsive

Sometimes this clears up by itself, sometimes I have to ask my provider to reboot the system because I can't even SSH into it.

This always happens at night, when traffic is very low. Here are two graphs:

Apache accesses

enter image description here

Memory consumption

enter image description here

How do I get to the bottom of this? Which steps can I take to debug this situation? Is it possible that I have a bad neighbor on the machine?

Other processes on the major machine: MySQL, PostgreSQL, Elasticsearch, Redis

UPDATE to clarify

I'm looking for a tool that records processes with their memory & cpu consumption so I can track down the trouble maker(s). Is something like that out there or do I have to roll my own?

Improve this question
5
  • 1
    Probably a cron job. Check /etc/cron.daily/ and /etc/cron.d/.
    – bahamat
    Jan 2, 2014 at 8:39
  • Yes, I had that suspicion too, but I found nothing out of the ordinary: MySQL dumper, munin, etc.
    – Wukerplank
    Jan 2, 2014 at 9:32
  • There doesn't have to be anything out of the ordinary, just that there is something wrong that causes those jobs to fail occasionally and do this. I also suspect the cron jobs for something that occurs always at the same time.
    – Chris Down
    Jan 2, 2014 at 9:37
  • OK, assuming a cron job messes up - how do I find the culprit? — I've updated my question.
    – Wukerplank
    Jan 2, 2014 at 10:00
  • Start with a log of ps output. Run it every two seconds over night and see what's listed as the highest memory consumer(s). Once you figure out the offending process you can look deeper into it to see what it's doing.
    – bahamat
    Jan 2, 2014 at 20:52

1 Answer 1

Reset to default
1

From my personal experience of managing RoR apps, memory leaks can easily creep in and can catch you at most unexpected time. I would recommend doing two things to catch the culprit:

  1. Use sar to record the system activities.
  2. Since memory is specific target, use free | grep "Mem:" | awk '{print $4}' and send out a mail to yourself using mailx if the threshold exceeds a certain limit of your choice.

Usually there is a bad sql query or a rouge gem that cause this. If anything has recently changed with respect to these, then you may want to look into that as well.

Improve this answer
1
  • Thanks! I didn't know sar, I'll look into it. For now I'm dumping ps outputs into a file, but I will pick up your idea with mailx.
    – Wukerplank
    Jan 3, 2014 at 9:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.