Here's what the memory graph looks like on a VPS running CentOS with 512MB of RAM and nginx/php-fpm/mysqld serving (mostly static) content to a couple thousand visitors per day.

Weekly memory graph

(those are days on the x-axis)

As you can see, it's quite jumpy in the cache and buffer area. The memory cache is purged at irregular intervals (ruling out a responsible cron job). It's usually, but not always, purged at the point where it can grow no larger. Sometimes it clears almost entirely, at other times only halfway down.

I'm trying to understand the logic behind these purges. I would expect file data to be cached much longer & don't see any other programs using more memory than usual when the memory cache is cleared.

Is this normal behavior, or am I missing something?

UPDATE: A memory upgrade seems to have stabilized the graph. Still seeing small drops, but nowhere as significant as it was prior to the upgrade.

After memory upgrade

  • Is this an OpenVZ/Virtuozzo container or a real VM such as XEN or KVM? – jordanm Oct 23 '12 at 22:30
  • 1
    Can't explain what they are, but I have a VPS which displays the same behaviour. dl.dropbox.com/u/1578899/memory-week.png – EightBitTony Oct 23 '12 at 22:32
  • @jordanm It's a Xen-based virtual machine. – redburn Oct 24 '12 at 17:00
  • @EightBitTony Thanks for sharing. Yours looks a little bit more 'natural', but I do clearly see a similar (but perhaps more predictable) pattern of drops in memory cache. – redburn Oct 24 '12 at 17:02
  • I did wonder if Munin 2 was graphing / collecting the data differently enough to result in some of the differences (smoother graph on yours), but even mine shows a drop in the middle of a cycle rather than daily. It's odd, for sure. – EightBitTony Oct 25 '12 at 11:33

Could be a lot of things. Maybe one of the programs you're running is occasionally and briefly using lots of RAM. If that's really weeks on the x-axis, you should sample at a much higher resolution (e.g. once per minute or even second) to get more information about what's going on that is causing the cache to get dropped. ps and top output (including load average) during that time would be useful too.

  • Yes, I guess we could theorize that a very short, sudden burst in memory use which happens in a minute or so, and isn't spotted by Munin could dump the cache, which of course, is spotted because it persists. – EightBitTony Oct 25 '12 at 11:34
  • The header is a bit confusing, in that it's actually showing one week of data, so those are days on the x-as, not weeks. As for the polling frequency: Munin fetches data every 5 minutes, and I don't think that frequency can be altered. I'm only running nginx, mysql, php-fpm and munin-node. Can it have anything to do with the mysql cache, perhaps? – redburn Oct 26 '12 at 20:59
  • I had top (sorted by memory usage) write its output to a file every 5 seconds, then I analyzed that file and found no processes showing any unusual behavior at the point where cached memory suddenly dropped. Unless a process can use this much memory and still escape that 5 second window, I'm not convinced this could be the cause. But if there's no processes running amok, what could it be? – redburn Nov 3 '12 at 10:46
  • This thread is a bit stale, but a quick observation about methodology: how much a process contributes to the system memory cache will not (easily) be reflected in top, since a very active process may pile stuff into the cache very rapidly without its own allocated memory increasing much. For example, reading a very large file in small chunks for transmission -- while the process might never use more than a few MB allocated, those few MB will constantly change and accumulate references in the cache. So the thing to look at in the top output would be a sudden accumulation of CPU time. – goldilocks Dec 6 '12 at 21:04
  • You might also be interested in this: cognitivedissonance.ca/cogware/plog – goldilocks Dec 6 '12 at 21:04

One possible reason would be a growing file, like a say a log, being either removed, compressed or sent elsewhere when it reaches a given size.

In either case, its cached size, possibly the whole if there is no memory pressure in your OS, would be freed from the cache as soon as the original file is removed.

  • An interesting idea, but the most active log files rarely exceed a file size of 25MB before they are rotated, and cache/buffer usage tends to drop by about 200MB. – redburn Jan 22 '13 at 20:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.