14

Here's what the memory graph looks like on a VPS running CentOS with 512MB of RAM and nginx/php-fpm/mysqld serving (mostly static) content to a couple thousand visitors per day.

Weekly memory graph

(those are days on the x-axis)

As you can see, it's quite jumpy in the cache and buffer area. The memory cache is purged at irregular intervals (ruling out a responsible cron job). It's usually, but not always, purged at the point where it can grow no larger. Sometimes it clears almost entirely, at other times only halfway down.

I'm trying to understand the logic behind these purges. I would expect file data to be cached much longer & don't see any other programs using more memory than usual when the memory cache is cleared.

Is this normal behavior, or am I missing something?

UPDATE: A memory upgrade seems to have stabilized the graph. Still seeing small drops, but nowhere as significant as it was prior to the upgrade.

After memory upgrade

Improve this question
8
  • Is this an OpenVZ/Virtuozzo container or a real VM such as XEN or KVM?
    – jordanm
    Oct 23, 2012 at 22:30
  • 1
    Can't explain what they are, but I have a VPS which displays the same behaviour. dl.dropbox.com/u/1578899/memory-week.png
    – EightBitTony
    Oct 23, 2012 at 22:32
  • @jordanm It's a Xen-based virtual machine.
    – redburn
    Oct 24, 2012 at 17:00
  • @EightBitTony Thanks for sharing. Yours looks a little bit more 'natural', but I do clearly see a similar (but perhaps more predictable) pattern of drops in memory cache.
    – redburn
    Oct 24, 2012 at 17:02
  • I did wonder if Munin 2 was graphing / collecting the data differently enough to result in some of the differences (smoother graph on yours), but even mine shows a drop in the middle of a cycle rather than daily. It's odd, for sure.
    – EightBitTony
    Oct 25, 2012 at 11:33

2 Answers 2

Reset to default
4

Could be a lot of things. Maybe one of the programs you're running is occasionally and briefly using lots of RAM. If that's really weeks on the x-axis, you should sample at a much higher resolution (e.g. once per minute or even second) to get more information about what's going on that is causing the cache to get dropped. ps and top output (including load average) during that time would be useful too.

Improve this answer
5
  • Yes, I guess we could theorize that a very short, sudden burst in memory use which happens in a minute or so, and isn't spotted by Munin could dump the cache, which of course, is spotted because it persists.
    – EightBitTony
    Oct 25, 2012 at 11:34
  • The header is a bit confusing, in that it's actually showing one week of data, so those are days on the x-as, not weeks. As for the polling frequency: Munin fetches data every 5 minutes, and I don't think that frequency can be altered. I'm only running nginx, mysql, php-fpm and munin-node. Can it have anything to do with the mysql cache, perhaps?
    – redburn
    Oct 26, 2012 at 20:59
  • I had top (sorted by memory usage) write its output to a file every 5 seconds, then I analyzed that file and found no processes showing any unusual behavior at the point where cached memory suddenly dropped. Unless a process can use this much memory and still escape that 5 second window, I'm not convinced this could be the cause. But if there's no processes running amok, what could it be?
    – redburn
    Nov 3, 2012 at 10:46
  • This thread is a bit stale, but a quick observation about methodology: how much a process contributes to the system memory cache will not (easily) be reflected in top, since a very active process may pile stuff into the cache very rapidly without its own allocated memory increasing much. For example, reading a very large file in small chunks for transmission -- while the process might never use more than a few MB allocated, those few MB will constantly change and accumulate references in the cache. So the thing to look at in the top output would be a sudden accumulation of CPU time.
    – goldilocks
    Dec 6, 2012 at 21:04
  • You might also be interested in this: cognitivedissonance.ca/cogware/plog
    – goldilocks
    Dec 6, 2012 at 21:04
2

One possible reason would be a growing file, like a say a log, being either removed, compressed or sent elsewhere when it reaches a given size.

In either case, its cached size, possibly the whole if there is no memory pressure in your OS, would be freed from the cache as soon as the original file is removed.

Improve this answer
1
  • An interesting idea, but the most active log files rarely exceed a file size of 25MB before they are rotated, and cache/buffer usage tends to drop by about 200MB.
    – redburn
    Jan 22, 2013 at 20:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .