I'm struggling to understand the inner workings of page frame reclamation algorithm in RHEL 6.
More specifically, I want to understand why we are seeing non zero values of si/so in vmstat and other signs of swapping when free memory doesn't go anywhere below pages_low (or even pages_high).
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 13 4 2476036 1533508 486264 10396996 18 22 9674 2790 59364 114558 7 8 81 4 0
i.e. there's 1533508 kilobytes of free memory on the system.
Node 0, zone Normal … min 130364 low 162955 high 195546
The fact that we see non-zero swap-in and swap-out activity (si>0, so>0) while free memory (equivalent of about 375k pages) is well above both low and high memory thresholds seems to be at odds with how swapping activity is described in documentation and literature.
E.g. “Understanding Linux Virtual Memory” by Mel Gorman:
“Historically, kswapd used to wake up every 10 seconds but now it is only woken by the physical page allocator when the pages_low number of free pages in a zone is reached”
Later on the book offers one possible explanation to what we are seeing:
“Under extreme memory pressure, processes will do the work of kswapd synchronously by calling balance_classzone() which calls try_to_free_pages_zone()”
i.e. when memory allocation requests fail or are slow, processes can initiate zone balancing themselves. However, it’s not clear whether this can account for swapping as try_to_free_pages_zone seems to be focused around shrinking various caches.
Also, we often see kswapd in top when observing signs of swapping, which also seems to be at odds with the direct reclamation theory.
Is there something I'm missing here?
Update I specifically checked ExaWatcher ps output taken during a period of swapping and I can see kswapd0 process in the "R" state during these times. I.e. this rules out the direct reclamation scenario.
Best regards, Nikolai