1

I am looking for a counter provided by the Linux kernel counting the number of memory allocations performed by tasks on the system. I want to watch for high velocity changes in this counter, using Prometheus, in order to detect when some task on the machine does something stupid like allocating memory in loops.

I have found a bunch of different metrics that seem to be gauges, that is numbers representing the current state of the machine. Examples include nr_free_pages and kbhugused. These measure the current amount of something being available or used, but since one task allocating for example 1 page and then deallocating it again will result in an unchanged gauge these are of little use to me.

One thought I came across on IRC was if there was a counter for the number of times brk(2) was called, but I soon found that it was not the only system call used to allocate memory.

Right now I'm looking at pgalloc_normal in /proc/vmstat, but have yet to figure out exactly what it is a measure of.

Why do we want to look for huge rates of memory allocation you ask? Because memory allocation is costly. Not only do you have to switch into kernel space and back, the kernel also has a number of locks that can bring a system from having 2 CPUs with 80 execution threads processing data in parallel to just 1 thread allocating memory. This is a real world scenario we have encountered and want to watch for.

2
  • What's stupid about allocating memory in a loop? Surely it depends on other things happening in the loop? – Kusalananda Jan 20 at 11:54
  • It is usually a better idea to allocate the memory you will need outside the loop, and then merely use it inside the loop. Why? Because memory allocation is costly. Not only do you have to switch into kernel space and back, the kernel also has a number of locks that can take a system from having 2 CPUs with 80 execution threads processing data in parallel to just 1 thread allocating memory. This is a real world scenario we have encountered and want to watch for. – Rovanion Jan 20 at 11:59
1

There’s no such counter ready-made for you as far as I’m aware, but there are a number of ways to track system calls. I think the most straightforward is to use perf:

sudo perf stat -e syscalls:sys_enter_mmap -e syscalls:sys_enter_brk -I 1000 -a

This will show the number of mmap and brk calls per second, every second.

You can track all system calls with this variant:

sudo perf stat -e 'syscalls:sys_enter_*' -I 1000 -a

You can also monitor specific processes, using -p and the relevant pid, instead of -a.

4
  • I know this question is hard to answer, but: On a scale from "run this only when debugging" to "run this in production", how would you rate the performance impact of having this running on a machine? Though I like it, since we can run it on a per batch-job granularity rather than per machine. – Rovanion Jan 21 at 11:00
  • There’s very little impact, this can be run in production. – Stephen Kitt Jan 21 at 11:08
  • BTW in relation to this and your pgalloc question, note that page allocation is separate from memory allocation; mmap and brk counters measure address space allocation, whereas pgalloc measures page allocation — if a program allocates more memory than it uses, it can call brk etc. without actually allocating pages. – Stephen Kitt Jan 21 at 11:10
  • Thank you for your help! Have been thinking and testing around your comment today. Would you say the following statement is true? One memory allocation call using mmap or brk for a large chunk of memory may allocate many pages, thus increasing pgalloc by a lot, while not costing significantly more CPU time than a call to allocate a small amount of memory. – Rovanion Jan 22 at 11:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.