Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

The main server at my company has recently been having a lot of downtime. For reasons that neither I nor the other admins can determine, it has random (VERY sudden) explosions in memory. It becomes unresponsive because it exhausts all the memory, and then we have to reboot it. Very annoying. It's a Debian system, we haven't upgraded to Squeeze or anything, it's been perfectly stable for a long time.

The problem is that the logs are totally useless. They don't seem to indicate that anything is going wrong. I'm guessing that some process is buggy and hogging all of the memory, but I have NO way of proving that at the moment. Remote logging is no help, because it's not complaining about anything -- it thinks everything is peachy.

So my question is: how would you approach this problem? Any insight is appreciated. Thanks.

share|improve this question
    
How about a strategy of logging memory usage for every process after it exceeds memory more than X (for some fixed memory usage X) at any given point in tiem? I don't know how difficult it would be to implement such an approach though. –  Faheem Mitha May 9 '11 at 20:53
    
What evidence do you have that memory is exhausted? If you can see that memory is exhausted, why cannot you see what is using it? –  camh May 9 '11 at 21:27
    
We use cacti, and right before the server goes down the memory goes to 100% and so does the swap. This usually happens within a 5-10 minute window. And we're talking about roughly 40G (24G RAM + 16G swap) of total virtual space, this ain't no small potatoes. –  Chris May 9 '11 at 22:03
    
God can I pick multiple answers? These are all good... –  Chris May 9 '11 at 22:37
add comment

5 Answers 5

up vote 3 down vote accepted

atop is pretty good at monitoring and logging resource usage. It can be used interactively or as a service; the debian package sets it to log to /var/log/atop.log every ten minutes (edit /etc/init.d/atop for something more precise). You can then replay the logs with atop -r /var/log/atop.log -b hh:mm -mM; mM selects a view and a sort appropriate for memory problems, hh:mm should be a few minutes before the incident, use tT to navigate. Also try the A sort.

share|improve this answer
    
Atop rocks. I've used it to successfully solve a similar problem. –  pboin May 10 '11 at 12:03
    
Excellent call. The explosion finally happened again -- it was emacs! Some weird emacs 23 X forwarding bug that no one seems to know about. We've "solved" the problem by turning off X forwarding. Ugh. Anyways atop saved the day, so I'm going to go with yours as the answer. –  Chris May 18 '11 at 18:16
add comment

You could try to use a combination of limiting address space (as) per process and number of processes (nproc) per user through /etc/security/limits.conf (pam_limits), hopefully this would help avoiding that your box become completely unresponsive and that whatever is causing the problem dies noisily (or whomever causes it complains that their stuff doesn't work anylonger) so you can figure out the actual cause.

share|improve this answer
add comment

The first step would be to monitor memory usage. top can do that, but the Linux version is clumsy to set up. Make a copy of the top executable called top-mem-chris. Run it and set some reasonable parameters, at least sort by memory usage (M) and display only the first N tasks (n 15). Then have it generate a configuration file (W) ~/.top-mem-chrisrc. For monitoring, run top-mem-chris -b -d 5 >~/log/top-mem-mainserver.log. Alternatively, perhaps try atsar, which looks suitable but I don't know beyond the description.

share|improve this answer
add comment

Do you log number of active processes? If the system doesn't kill off memory hogs, it's more likely you're seeing a fork bomb (whether malicious or programming error).

share|improve this answer
    
Yes we log that, it's not a fork bomb. Good idea though. –  Chris May 10 '11 at 3:07
    
So if nr. of processes isn't the culprit and you cannot pinpoint a single process eating the memory. Leaves the kernel. NFS? –  Mel May 10 '11 at 3:21
    
Or possibly a single process. Right now cassandra is the culprit. –  Chris May 10 '11 at 3:28
add comment

If there's a real head on the machine (unlikely), try having your friendly datacenter facility tech pressing caps lock on a normal, direct-attached keyboard. If a x86 processor is still servicing interrupts, should will work. If this does not work, the machine maybe borked. I imagine there are corner cases with lights-out cards, ip kvms, ps/2 vs usb, etc.

Check the usual: free disk space, crons, mail spool, rootkits, etc. Also, I've seen bad hardware, i.e., PSUs, mem, NFS/iSCSI/FC interruptions, do all sorts of things. (Hint: hardware self-tests are nearly always useless.)

In case not mentioned elsewhere:

  • periodic trending sar
  • disk io iostat ALL -p ALL
  • disk usage df -h
  • network activity iftop
  • network packets sudo tcpdump
  • mem free -lmt
  • memory burn-in testing: memtest86+
  • enhanced top htop
  • kernel goodness slabtop

top's Prettier Keys:

  • 10 updates/s d0.1enter
  • Toggle SMP view 1
  • Color toggle z
  • Toggle sort highlight x
  • Toggle highlight running tasks y
  • Sort by next LEFT column < (typically shift+,)
  • Sort by next RIGHT column > (typically shift+.)
  • Toggle sort bold shift+b
  • Toggle bold b
  • Toggle sort asc -> desc shift+r

Partitioning vital directory into separate mount points / /tmp /var /var/log /var/tmp /var/spool /usr /usr/local /opt /home ... are important to not take down systems at some point in the future if anticipated reasonably.

share|improve this answer
    
Yeah it's got a head. Not sure about the caps lock thing, that's a good idea though. Even if it is still servicing interrupts the rest of the machine is completely unresponsive. Even using a real monitor and keyboard I can't even pull it out of power-saving mode (i.e. pressing a key won't even power the monitor on). Thanks for the great ideas. –  Chris May 10 '11 at 13:53
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.