I am looking for a counter provided by the Linux kernel counting the number of memory allocations performed by tasks on the system. I want to watch for high velocity changes in this counter, using Prometheus, in order to detect when some task on the machine does something stupid like allocating memory in loops.
I have found a bunch of different metrics that seem to be gauges, that is numbers representing the current state of the machine. Examples include
kbhugused. These measure the current amount of something being available or used, but since one task allocating for example 1 page and then deallocating it again will result in an unchanged gauge these are of little use to me.
One thought I came across on IRC was if there was a counter for the number of times
brk(2) was called, but I soon found that it was not the only system call used to allocate memory.
Right now I'm looking at
/proc/vmstat, but have yet to figure out exactly what it is a measure of.
Why do we want to look for huge rates of memory allocation you ask? Because memory allocation is costly. Not only do you have to switch into kernel space and back, the kernel also has a number of locks that can bring a system from having 2 CPUs with 80 execution threads processing data in parallel to just 1 thread allocating memory. This is a real world scenario we have encountered and want to watch for.