2

I'm struggling to understand the inner workings of page frame reclamation algorithm in RHEL 6.

More specifically, I want to understand why we are seeing non zero values of si/so in vmstat and other signs of swapping when free memory doesn't go anywhere below pages_low (or even pages_high).

From vmstat:

  procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu-----
 r     b swpd        free          buff       cache        si   so   bi      bo      in          cs        us sy id wa st
 13 4 2476036 1533508 486264 10396996 18 22 9674 2790 59364 114558 7 8 81 4 0

i.e. there's 1533508 kilobytes of free memory on the system.

From /proc/zoneinfo

Node 0, zone Normal
…
 min 130364
 low 162955
 high 195546 

The fact that we see non-zero swap-in and swap-out activity (si>0, so>0) while free memory (equivalent of about 375k pages) is well above both low and high memory thresholds seems to be at odds with how swapping activity is described in documentation and literature.

E.g. “Understanding Linux Virtual Memory” by Mel Gorman:

“Historically, kswapd used to wake up every 10 seconds but now it is only woken by the physical page allocator when the pages_low number of free pages in a zone is reached”

Later on the book offers one possible explanation to what we are seeing:

“Under extreme memory pressure, processes will do the work of kswapd synchronously by calling balance_classzone() which calls try_to_free_pages_zone()”

i.e. when memory allocation requests fail or are slow, processes can initiate zone balancing themselves. However, it’s not clear whether this can account for swapping as try_to_free_pages_zone seems to be focused around shrinking various caches.

Also, we often see kswapd in top when observing signs of swapping, which also seems to be at odds with the direct reclamation theory.

Is there something I'm missing here?

Update I specifically checked ExaWatcher ps output taken during a period of swapping and I can see kswapd0 process in the "R" state during these times. I.e. this rules out the direct reclamation scenario.

Best regards, Nikolai

  • Not that I can answer this, but could you edit to specify the version of the rhel6 franken-kernel(s) (or any other kernels) you have seen this behaviour on? – sourcejedi Jul 16 at 8:13
  • could it be similar to unix.stackexchange.com/questions/224958/… ? i.e. are you using docker (or directly using cgroups) and setting memory limits? – sourcejedi Jul 16 at 8:20
  • The kernel version is 4.1.12-94.8.10.el6uek.x86_64. I don't see any restrictive limits on memory in cgroup: memory.kmem.max_usage_in_bytes: 0 memory.use_hierarchy: 0 memory.swappiness: 1 memory.memsw.failcnt: 0 memory.limit_in_bytes: 9223372036854771712 memory.memsw.max_usage_in_bytes: 0 memory.usage_in_bytes: 84498038784 memory.memsw.limit_in_bytes: 9223372036854771712 memory.failcnt: 0 memory.kmem.limit_in_bytes: 9223372036854771712 – Nikolai Jul 16 at 8:54
1

I was able to find at least one scenario that can lead to swapping pages out of main memory while free memory is well above any zone watermarks. The scenario has to do with zone compaction, one of algorithms for VM defragmentation.

The basic idea behind the process is to move pages around to create large continuous chunks of virtual addresses. "Moving around" refers to updating pages' PTEs, not physically moving them.

The compaction algorithm runs two scanners from opposite ends of a zone, working their way towards each other. One scanner searches for pages to move, the other one for free pages where they could be moved to, and eventually they are supposed to meet somewhere in the middle.

The thing is, during zone compaction, it is possible to find a page that cannot be moved, but yet can be reclaimed. When this happens, the algorithm may try to reclaim it by swapping it out.

The important thing here is that zone compaction is not triggered by any watermarks. Rather, it happens whenever a high order allocation fails, i.e. it can happen when there's still plenty of free memory left, if this memory is fragmented enough.

1

I found another answer which is probably a better fit. As it turns out, modern versions of Linux kernel apart from direct and periodic reclaims have slowpath allocations, where kswapd is woken up after failing to allocate a contiguous chunk of memory.

When woken up, kswapd does check zone watermarks. However, as it turns out, watermarks are not static zone-level numbers they may have been once. Rather, they are specific to the allocation order.

I.e. when deciding whether or not a zone should be rebalanced, kswapd takes into account the order of the failed allocation request that triggered it. So if memory is fragmented enough, kswapd will have work to do.

When rebalancing a zone, kswapd will have a choice between shrinking file cache and stealing anonymous pages from user processes (unless swapping is completely disabled). So the remaining question is -- why does kswapd go for the latter option. I think the answer is once again fragmentation -- I think the reclamation algorithm might have a way of knowing that pages obtained from shrinking the file cache may not physically contiguous.

More generally speaking, not only recent versions of kernel have memory defragmentation added, but also the line between page frame reclaiming and compacting memory is somewhat blurred.

Unfortunately, all classic Linux kernel textbooks were written based on 2.6 or earlier kernel versions, so they can be quite misleading.

  • I don't think the watermarks started varying depending on the size of the allocation. The function I found just says "For high-order checks it will return true of the order-0 watermark is reached and there is at least one free page of a suitable size." I think the real change is that RHEL enables transparent huge pages for all applications. This means it attempts to allocate non-file-backed memory in 2MiB pageblocks if possible. But you're right in the sense this ends up being a pretty significant change, compared to all the descriptions of VM behaviour around 2.6. – sourcejedi Aug 3 at 22:32
  • 1
    Which function are you referring to? Watermark checking is handled by __zone_watermark_ok and its wrappers, and it does take order allocation as a parameter. Also, the comment says: /* * Return true if free pages are above 'mark'. This takes into account the order * of the allocation. */ static bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark, int classzone_idx, int alloc_flags, long free_pages) and in the body you can actually see it looping up to "order", so its result does depend on the allocation order that triggered its call. – Nikolai Aug 5 at 3:47
  • Sorry, I'm doing the thing of looking at the latest code, even though I asked you about the version. I.e. being lazy rather than downloading the exact RHEL code, or even comparing the base version of the RHEL kernel. Here's the code I have been looking at: elixir.bootlin.com/linux/v5.3-rc1/source/mm/page_alloc.c#L3397 The loop over order counts up, it means that a request with a relatively low order can be satisfied by breaking up a higher-order free block. – sourcejedi Aug 5 at 9:05
  • 1
    I think you should be able to navigate to a specific release tag on that website using the panel on the left. It's intereseting that 5.3 behavior is different, but the difference isn't really huge. In both 4.12 and 5.3 a) zone balancing is not just triggered by low memory, but by memory fragmentation as well and b) failed high order allocations can lead to scenarios other than direct reclaim/compaction. I.e they can wake up kswapd processes (and possibly kcompactd in newer kernel versions). Which explains why you can see swapping, active kswapd & high pgscank/s with memfree > pages_high – Nikolai Aug 6 at 9:36
  • linux-4.1.12/mmpage_alloc.c:1852 I see it now. Thanks! As you say, there's a change in how that function behaves, even though both versions have a loop over order. And the overall answer does not rely on that exact detail. (There are other changes one could look at, but I probably shouldn't wonder about them on a question about RHEL 6 :-) – sourcejedi Aug 6 at 10:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.