A procedure of caching data in linux kernel, especially on LLC

Question

I wonder how kernel puts data on LLC or L3 caches in detail. I have been searching the information about using cache in kernel but many results talk about page cache or buffered cache but in this question it is only related to cache above the main memory.

Assume that I make a working set or data structure (30MB) in the kernel context by using kmalloc, vmalloc or whatever. What I am thinking is that there might be a different procedure such as protection or extra optimization to prevent any memory violation in kernel unlike using malloc or mmap in user context.

Therefore, if I make a same-size working set(30MB) using user library(malloc) in user mode, I would expect that I might get different performance when I access this data in the working set.

Question
1. How kernel uses LLC cache?
2. Can I get different performance when I allocate memory using malloc?

Answer 1

LLC means only "last level cache". It is strongly architecture-dependent, exactly which level of cache is it. In recent x86/amd64 architectures, it is typically L3 cache.

Caching happens mostly independently from the kernel. There is none to little function to manipulate CPU-memory caching, including L3 cache, even on the asm level. It might be dubious, if it is even the task of the kernel to deal with it.

On Intel (AMD) CPUs, there are the mtrr registers which can affect, how the kernel can see the cache-ability of memory ranges. On newer intel CPUs, there is also a functionality to partition the L3 cache, which can be tuned by tools (since kernel 4.10).¹

The kernel API doesn't have a malloc() function. It has kmalloc(), which is essentially a malloc(). It has also vmalloc(), which allocates always pages and not bytes. Note, the kernel VM deals mostly with pages, kmalloc() is essentially only a wrapper around vmalloc() to get a more effective, malloc-like functionality.

There are various optimizations in the kernel algorithms to make it better tuned in a caching environment, although they deal mainly with lower cache levels (for example, it is better if concurrent processes/kernel threads work least possible on the same page or cache line). Using vmalloc() can help a lot to deal with cache in a multi-cpu environment.

The kernel is written mainly in C, it handles memory addresses and caching is handled by the cpu memory handler mechanism transparently.

¹ _{Thanks for Stephen Kitt for the valuable comment.}