0

I wonder how kernel puts data on LLC or L3 caches in detail. I have been searching the information about using cache in kernel but many results talk about page cache or buffered cache but in this question it is only related to cache above the main memory.

Assume that I make a working set or data structure (30MB) in the kernel context by using kmalloc, vmalloc or whatever. What I am thinking is that there might be a different procedure such as protection or extra optimization to prevent any memory violation in kernel unlike using malloc or mmap in user context.

Therefore, if I make a same-size working set(30MB) using user library(malloc) in user mode, I would expect that I might get different performance when I access this data in the working set.

Question
1. How kernel uses LLC cache?
2. Can I get different performance when I allocate memory using malloc?

2

LLC means only "last level cache". It is strongly architecture-dependent, exactly which level of cache is it. In recent x86/amd64 architectures, it is typically L3 cache.

Caching happens mostly independently from the kernel. There is none to little function to manipulate CPU-memory caching, including L3 cache, even on the asm level. It might be dubious, if it is even the task of the kernel to deal with it.

On Intel (AMD) CPUs, there are the mtrr registers which can affect, how the kernel can see the cache-ability of memory ranges. On newer intel CPUs, there is also a functionality to partition the L3 cache, which can be tuned by tools (since kernel 4.10).1

The kernel API doesn't have a malloc() function. It has kmalloc(), which is essentially a malloc(). It has also vmalloc(), which allocates always pages and not bytes. Note, the kernel VM deals mostly with pages, kmalloc() is essentially only a wrapper around vmalloc() to get a more effective, malloc-like functionality.

There are various optimizations in the kernel algorithms to make it better tuned in a caching environment, although they deal mainly with lower cache levels (for example, it is better if concurrent processes/kernel threads work least possible on the same page or cache line). Using vmalloc() can help a lot to deal with cache in a multi-cpu environment.

The kernel is written mainly in C, it handles memory addresses and caching is handled by the cpu memory handler mechanism transparently.

1 Thanks for Stephen Kitt for the valuable comment.

  • “It might be dubious, if it is even the task of the kernel to deal with it.” +1. For extra fun and games, the LLC on some CPUs is now partitionable ;-). – Stephen Kitt Jan 7 at 6:44
  • @peterh Thanks for your answer. My ultimate question is actually related to the second one. In the second question what I really mean is that can I get same performance (e.g. cache miss rate) when I make a working set in user application using malloc like I made same set in the kernel using kmalloc. From the part of your answer, "Caching happens mostly independently from the kernel", I think I would get the same performance no matter where data resides (user or kernel) – Mr.Nobody Jan 7 at 6:55
  • @StephenKitt Wow, it seems it is more complex as I knew. There is even kernel support for that ( manpages.debian.org/testing/intel-cmt-cat/pqos.8.en.html ). There are also mtrr since decades ago. – peterh Jan 7 at 6:57
  • @vincentc Doing it from the kernel will be faster, because it will avoid a lot of permission checking and memory mapping mechanism, and context switches. But developing it would be much harder and the performance gain won't be too big in most scenarios. Permission check happens before the cache (well, not entirely, google for "spectre" and "meltdown", but these were cpu bugs), from the view of the memory access there is no gain. – peterh Jan 7 at 7:07
  • @peterh Thanks for your clarification. I think I need to delve into performance matter in kernel. – Mr.Nobody Jan 7 at 7:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.