We build a system that's intended to be on all the time - it collects and displays graphs of data. If we leave it without changing anything for long enough, we end up with an oom-killer event. That kills our main process (it's got the high oom-score) and our software gets restarted.
Basics: The system is CentOS 6, kernel is 22.214.171.124. The system has 2G of ram and 4G of swap. The application is written in C++ w/Qt 3.
I've set a cron job to grab the contents of /proc/meminfo and /proc/slabinfo every minute. Here's the traces I find most interesting from the meminfo data (the most recent oom-killer is on the right side of the graph):
Note SUnreclaim grows until the oom-killer hits. The change in slope on SUnreclaim is where I switched displays.
What this looks like to me is that something's leaking or fragmenting. Whatever it is does seem to get cleaned up when my processes die, but I honestly have no idea what's going on here.
How do I figure out what's leaking?
Updated: Early on in this process, I started with ps output (not shown here). All of our processes RSS values ramp up quickly to their 'normal' level and then stay put. If this was a process running away with normal memory, I wouldn't need assistance. Instead, there's something we're doing that's causing unswappable memory to be allocated.
As to the upgrade suggestion: The codebase has a lot of dependencies on old libraries, and I can't make a transition to even a 3 series kernel right now.