The main server at my company has recently been having a lot of downtime. For reasons that neither I nor the other admins can determine, it has random (VERY sudden) explosions in memory. It becomes unresponsive because it exhausts all the memory, and then we have to reboot it. Very annoying. It's a Debian system, we haven't upgraded to Squeeze or anything, it's been perfectly stable for a long time.

The problem is that the logs are totally useless. They don't seem to indicate that anything is going wrong. I'm guessing that some process is buggy and hogging all of the memory, but I have NO way of proving that at the moment. Remote logging is no help, because it's not complaining about anything -- it thinks everything is peachy.

So my question is: how would you approach this problem? Any insight is appreciated. Thanks.

  • How about a strategy of logging memory usage for every process after it exceeds memory more than X (for some fixed memory usage X) at any given point in tiem? I don't know how difficult it would be to implement such an approach though. May 9, 2011 at 20:53
  • What evidence do you have that memory is exhausted? If you can see that memory is exhausted, why cannot you see what is using it?
    – camh
    May 9, 2011 at 21:27
  • We use cacti, and right before the server goes down the memory goes to 100% and so does the swap. This usually happens within a 5-10 minute window. And we're talking about roughly 40G (24G RAM + 16G swap) of total virtual space, this ain't no small potatoes. May 9, 2011 at 22:03
  • God can I pick multiple answers? These are all good... May 9, 2011 at 22:37

5 Answers 5


atop is pretty good at monitoring and logging resource usage. It can be used interactively or as a service; the debian package sets it to log to /var/log/atop.log every ten minutes (edit /etc/init.d/atop for something more precise). You can then replay the logs with atop -r /var/log/atop.log -b hh:mm -mM; mM selects a view and a sort appropriate for memory problems, hh:mm should be a few minutes before the incident, use tT to navigate. Also try the A sort.

  • Atop rocks. I've used it to successfully solve a similar problem.
    – pboin
    May 10, 2011 at 12:03
  • Excellent call. The explosion finally happened again -- it was emacs! Some weird emacs 23 X forwarding bug that no one seems to know about. We've "solved" the problem by turning off X forwarding. Ugh. Anyways atop saved the day, so I'm going to go with yours as the answer. May 18, 2011 at 18:16

You could try to use a combination of limiting address space (as) per process and number of processes (nproc) per user through /etc/security/limits.conf (pam_limits), hopefully this would help avoiding that your box become completely unresponsive and that whatever is causing the problem dies noisily (or whomever causes it complains that their stuff doesn't work anylonger) so you can figure out the actual cause.


The first step would be to monitor memory usage. top can do that, but the Linux version is clumsy to set up. Make a copy of the top executable called top-mem-chris. Run it and set some reasonable parameters, at least sort by memory usage (M) and display only the first N tasks (n 15). Then have it generate a configuration file (W) ~/.top-mem-chrisrc. For monitoring, run top-mem-chris -b -d 5 >~/log/top-mem-mainserver.log. Alternatively, perhaps try atsar, which looks suitable but I don't know beyond the description.


Do you log number of active processes? If the system doesn't kill off memory hogs, it's more likely you're seeing a fork bomb (whether malicious or programming error).

  • Yes we log that, it's not a fork bomb. Good idea though. May 10, 2011 at 3:07
  • So if nr. of processes isn't the culprit and you cannot pinpoint a single process eating the memory. Leaves the kernel. NFS?
    – Mel
    May 10, 2011 at 3:21
  • Or possibly a single process. Right now cassandra is the culprit. May 10, 2011 at 3:28

If there's a real head on the machine (unlikely), try having your friendly datacenter facility tech pressing caps lock on a normal, direct-attached keyboard. If a x86 processor is still servicing interrupts, should will work. If this does not work, the machine maybe borked. I imagine there are corner cases with lights-out cards, ip kvms, ps/2 vs usb, etc.

Check the usual: free disk space, crons, mail spool, rootkits, etc. Also, I've seen bad hardware, i.e., PSUs, mem, NFS/iSCSI/FC interruptions, do all sorts of things. (Hint: hardware self-tests are nearly always useless.)

In case not mentioned elsewhere:

  • periodic trending sar
  • disk io iostat ALL -p ALL
  • disk usage df -h
  • network activity iftop
  • network packets sudo tcpdump
  • mem free -lmt
  • memory burn-in testing: memtest86+
  • enhanced top htop
  • kernel goodness slabtop

top's Prettier Keys:

  • 10 updates/s d0.1enter
  • Toggle SMP view 1
  • Color toggle z
  • Toggle sort highlight x
  • Toggle highlight running tasks y
  • Sort by next LEFT column < (typically shift+,)
  • Sort by next RIGHT column > (typically shift+.)
  • Toggle sort bold shift+b
  • Toggle bold b
  • Toggle sort asc -> desc shift+r

Partitioning vital directory into separate mount points / /tmp /var /var/log /var/tmp /var/spool /usr /usr/local /opt /home ... are important to not take down systems at some point in the future if anticipated reasonably.

  • Yeah it's got a head. Not sure about the caps lock thing, that's a good idea though. Even if it is still servicing interrupts the rest of the machine is completely unresponsive. Even using a real monitor and keyboard I can't even pull it out of power-saving mode (i.e. pressing a key won't even power the monitor on). Thanks for the great ideas. May 10, 2011 at 13:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .