1

This is a follow-up question to dentry/inode caching on multi-cpu machines / memory allocator configuration, but here I try to put the question differently.

My problem is that I have a dual socket machine, and memory for the kernel caches (dentry/inode/buffer) are allocated from bank0 (cpu0's memory bank), and that eventually gets consumed. However, bank1 is never used for caches, so there is plenty of free memory in the overall system. So in this state the memory allocator gets memory from bank1, regardless of where my process is running (even if I set memory affinity). Due to the different memory latency when accessing memory from different banks, this means that my process (which is somewhat memory access bound with a low cache-hit ratio) will run much slower when scheduled on the cores in cpu0 than when scheduled on the cores in cpu1. (I'd like to schedule two processes, one for each cpu, and a process should use all cores of its cpu. I don't want to waste half the cores.)

What could I do to ensure that my process can get memory from the local bank, no matter on which cpu it gets scheduled on?

I tried playing with the kernel VM parameters, but they don't really do anything. After all, half the memory is free! These caches in kernel simply do not seem to take NUMA issues into account. I tried to look into cgroups, but as far as I can understand, I can't really control the kernel that way. I did not really find anything that would address my issue :-(.

I can, of course, drop all caches before starting my processes, but that is a bit heavy handed. A cleaner solution would be, for example, to limit the total cache size (say 8GB). True, cpu0 would still have a bit less memory than cpu1 (I have 64GB in both banks), but I can live with that.

I'd be grateful for any suggestions... Thanks!

6
  • Can you please append to your question the output of tree -P "read*|write*" /sys/devices/system/node/nodeX/access0/initiators/ for X=0 and X=1 ?
    – MC68020
    Commented Feb 3, 2023 at 6:53
  • Sorry, there is no access0 directory under nodeX. The machines are running centos 7.9.2009, the kernel is 3.10.0-1160.el7.x86_64. Do you think this issue would be solved with a newer kernel? Commented Feb 3, 2023 at 17:47
  • 3.10 ? OMG ! I think you'd need at least a 5.2 (since the first time I see this path referenced elixir.bootlin.com/linux/v5.2.21/source/drivers/base/node.c#L88 ) BUT BEWARE I CANNOT TELL IF centos 7 CAN GET ITS KERNEL VERSION UPGRADED TO WHATEVER 5.2+ Cont...
    – MC68020
    Commented Feb 3, 2023 at 18:28
  • ... But what we can understand from kernel.org/doc/html/v5.8/admin-guide/mm/numaperf.html is definitely that "Applications may wish to consider which node they want their memory to be allocated from based on the node’s performance characteristics. If the system provides these attributes…"
    – MC68020
    Commented Feb 3, 2023 at 18:31
  • Interesting, thanks for the link! However, this link just describes how the memory hierarchy is built up and how can information be extracted about it. But I still don't see anything about how to control (or limit) where the kernel caches get the memory from. As long as the vfs part of the kernel always gets scheduled on cpu0 and the allocator gives it "near memory", the issue I described will be present. Cont... Commented Feb 3, 2023 at 19:18

1 Answer 1

0
+50

"What is going on with kernel caching on NUMA architecture" under your linux-3.10 is governed by the zone_reclaim_mode sysctl which allows to select the appropriate action to be taken when a zone runs out of memory.
In other words, determine if the page allocator will reclaim easily reusable pages before allocating off node pages or the right opposite. (refer to the official documentation here-above linked for more details)

Several patches came in linux-4 times regarding default settings, then in linux-5 times, that global (nodes-wide) setting became a per node setting : node_reclaim_mode.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .