Unpredictable memory explosions

Question

The main server at my company has recently been having a lot of downtime. For reasons that neither I nor the other admins can determine, it has random (VERY sudden) explosions in memory. It becomes unresponsive because it exhausts all the memory, and then we have to reboot it. Very annoying. It's a Debian system, we haven't upgraded to Squeeze or anything, it's been perfectly stable for a long time.

The problem is that the logs are totally useless. They don't seem to indicate that anything is going wrong. I'm guessing that some process is buggy and hogging all of the memory, but I have NO way of proving that at the moment. Remote logging is no help, because it's not complaining about anything -- it thinks everything is peachy.

So my question is: how would you approach this problem? Any insight is appreciated. Thanks.

Answer 1

atop is pretty good at monitoring and logging resource usage. It can be used interactively or as a service; the debian package sets it to log to /var/log/atop.log every ten minutes (edit /etc/init.d/atop for something more precise). You can then replay the logs with atop -r /var/log/atop.log -b hh:mm -mM; mM selects a view and a sort appropriate for memory problems, hh:mm should be a few minutes before the incident, use tT to navigate. Also try the A sort.

Answer 2

You could try to use a combination of limiting address space (as) per process and number of processes (nproc) per user through /etc/security/limits.conf (pam_limits), hopefully this would help avoiding that your box become completely unresponsive and that whatever is causing the problem dies noisily (or whomever causes it complains that their stuff doesn't work anylonger) so you can figure out the actual cause.

Answer 3

The first step would be to monitor memory usage. top can do that, but the Linux version is clumsy to set up. Make a copy of the top executable called top-mem-chris. Run it and set some reasonable parameters, at least sort by memory usage (M) and display only the first N tasks (n 15). Then have it generate a configuration file (W) ~/.top-mem-chrisrc. For monitoring, run top-mem-chris -b -d 5 >~/log/top-mem-mainserver.log. Alternatively, perhaps try atsar, which looks suitable but I don't know beyond the description.

Answer 4

Do you log number of active processes? If the system doesn't kill off memory hogs, it's more likely you're seeing a fork bomb (whether malicious or programming error).

Answer 5

If there's a real head on the machine (unlikely), try having your friendly datacenter facility tech pressing caps lock on a normal, direct-attached keyboard. If a x86 processor is still servicing interrupts, should will work. If this does not work, the machine maybe borked. I imagine there are corner cases with lights-out cards, ip kvms, ps/2 vs usb, etc.

Check the usual: free disk space, crons, mail spool, rootkits, etc. Also, I've seen bad hardware, i.e., PSUs, mem, NFS/iSCSI/FC interruptions, do all sorts of things. (Hint: hardware self-tests are nearly always useless.)

In case not mentioned elsewhere:

• periodic trending sar
• disk io iostat ALL -p ALL
• disk usage df -h
• network activity iftop
• network packets sudo tcpdump
• mem free -lmt
• memory burn-in testing: memtest86+
• enhanced top htop
• kernel goodness slabtop

top's Prettier Keys:

• 10 updates/s d0.1enter
• Toggle SMP view 1
• Color toggle z
• Toggle sort highlight x
• Toggle highlight running tasks y
• Sort by next LEFT column < (typically shift+,)
• Sort by next RIGHT column > (typically shift+.)
• Toggle sort bold shift+b
• Toggle bold b
• Toggle sort asc -> desc shift+r

Partitioning vital directory into separate mount points / /tmp /var /var/log /var/tmp /var/spool /usr /usr/local /opt /home ... are important to not take down systems at some point in the future if anticipated reasonably.