Can kswapd be active if free memory well exceeds pages_high watermark?

Question

I'm struggling to understand the inner workings of page frame reclamation algorithm in RHEL 6.

More specifically, I want to understand why we are seeing non zero values of si/so in vmstat and other signs of swapping when free memory doesn't go anywhere below pages_low (or even pages_high).

From vmstat:

  procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu-----
 r     b swpd        free          buff       cache        si   so   bi      bo      in          cs        us sy id wa st
 13 4 2476036 1533508 486264 10396996 18 22 9674 2790 59364 114558 7 8 81 4 0

i.e. there's 1533508 kilobytes of free memory on the system.

From /proc/zoneinfo

Node 0, zone Normal
…
 min 130364
 low 162955
 high 195546 

The fact that we see non-zero swap-in and swap-out activity (si>0, so>0) while free memory (equivalent of about 375k pages) is well above both low and high memory thresholds seems to be at odds with how swapping activity is described in documentation and literature.

E.g. “Understanding Linux Virtual Memory” by Mel Gorman:

“Historically, kswapd used to wake up every 10 seconds but now it is only woken by the physical page allocator when the pages_low number of free pages in a zone is reached”

Later on the book offers one possible explanation to what we are seeing:

“Under extreme memory pressure, processes will do the work of kswapd synchronously by calling balance_classzone() which calls try_to_free_pages_zone()”

i.e. when memory allocation requests fail or are slow, processes can initiate zone balancing themselves. However, it’s not clear whether this can account for swapping as try_to_free_pages_zone seems to be focused around shrinking various caches.

Also, we often see kswapd in top when observing signs of swapping, which also seems to be at odds with the direct reclamation theory.

Is there something I'm missing here?

Update I specifically checked ExaWatcher ps output taken during a period of swapping and I can see kswapd0 process in the "R" state during these times. I.e. this rules out the direct reclamation scenario.

Best regards, Nikolai

Answer 1

I was able to find at least one scenario that can lead to swapping pages out of main memory while free memory is well above any zone watermarks. The scenario has to do with zone compaction, one of algorithms for VM defragmentation.

The basic idea behind the process is to move pages around to create large continuous chunks of virtual addresses. "Moving around" refers to updating pages' PTEs, not physically moving them.

The compaction algorithm runs two scanners from opposite ends of a zone, working their way towards each other. One scanner searches for pages to move, the other one for free pages where they could be moved to, and eventually they are supposed to meet somewhere in the middle.

The thing is, during zone compaction, it is possible to find a page that cannot be moved, but yet can be reclaimed. When this happens, the algorithm may try to reclaim it by swapping it out.

The important thing here is that zone compaction is not triggered by any watermarks. Rather, it happens whenever a high order allocation fails, i.e. it can happen when there's still plenty of free memory left, if this memory is fragmented enough.

Answer 2

I found another answer which is probably a better fit. As it turns out, modern versions of Linux kernel apart from direct and periodic reclaims have slowpath allocations, where kswapd is woken up after failing to allocate a contiguous chunk of memory.

When woken up, kswapd does check zone watermarks. However, as it turns out, watermarks are not static zone-level numbers they may have been once. Rather, they are specific to the allocation order.

I.e. when deciding whether or not a zone should be rebalanced, kswapd takes into account the order of the failed allocation request that triggered it. So if memory is fragmented enough, kswapd will have work to do.

When rebalancing a zone, kswapd will have a choice between shrinking file cache and stealing anonymous pages from user processes (unless swapping is completely disabled). So the remaining question is -- why does kswapd go for the latter option. I think the answer is once again fragmentation -- I think the reclamation algorithm might have a way of knowing that pages obtained from shrinking the file cache may not physically contiguous.

More generally speaking, not only recent versions of kernel have memory defragmentation added, but also the line between page frame reclaiming and compacting memory is somewhat blurred.

Unfortunately, all classic Linux kernel textbooks were written based on 2.6 or earlier kernel versions, so they can be quite misleading.