I appear to have a memory leak in the kernel space, slab shows kmalloc-4096 perpetually growing an an even clip until it monopolizes all of the ram resources on the system and forces swap.

Free shows much of this memory usage as cache, but it refuses to free on need, or even when flagged to manually clear. An example of what we're seeing:

$ sudo su -c "free -h && sync && echo 3 > /proc/sys/vm/drop_caches && free -h"
              total        used        free      shared  buff/cache   available
Mem:            15G        4.4G        166M        280M         10G        104M
Swap:           15G        7.8G        8.1G
              total        used        free      shared  buff/cache   available
Mem:            15G        4.4G        186M        280M         10G        115M
Swap:           15G        7.8G        8.1G

Should I be profiling kmalloc to determine where the leak is? If so, how might I go about doing that?

This is a stock Ubuntu 16.04 install on an Intel i5 Skylake.

$ uname -a
Linux fire 4.4.0-78-generic #99-Ubuntu SMP Thu Apr 27 15:29:09 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

The 'sync' isn't going to reduce the buff/cache, it merely requests that the data goes to the block devices. The point of cache is that the data remains in the RAM caches. Sending a 3 to drop_caches does not wait for the caches to get dropped either. Could you add a 'sleep 60' before the 'free'? This might allow some time for the memory to clear before reporting from free.

Leaving a 'watch -n60 cat /proc/meminfo' might help show the area of memory that is growing, but cache and buff are not leaks alone, by consuming memory they're fulfilling their purpose and making the best use of your system's available resource.

If a process is swapped to disk this does not automatically mean that something bad has happened. The kernel is doing the right thing if that process is not using all its pages and has gone idle, there is a reasonable chance that on a web server the content in the www root is going to be needed before a copy of mutt left running in GNU screen when the user has logged out, for example.

If you still consider that the kernel is consuming the RAM through modules, you can inspect this a little more using:

awk '{ print $2" "$1 }' /proc/modules  | sort -n

Are you using ZFS? That's pretty hungry for RAM, but like cache, it tries to keep disk IO in RAM just in case it's needed.

  • I agree that normally cache isn't an issue. On this particular server however, the swapiness is set to 0 and we're still having active processes forced onto swap with 10 - 12 GB of ram held by cache (per free) after a few days of usage. Because of this, I'm struggling to profile what is the cause of the usage. smem shows it's overwhelmingly "kernel dynamic memory" in the "used" column. slabtop shows the only thing that grows regardless of usage patterns to be kmalloc-4096, in the above case, it was holding ~12 GB. Would you recommend looking elsewhere to do a root cause? – DivinusVox May 31 '17 at 23:15
  • Edited answer for some more detail – Ed Neville Jun 1 '17 at 18:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.