Every Saturday, for reasons unknown to me, one of my servers experiences a spike in swap utilisation. It's not necessarily a problem as there is plenty of free memory available, but I'd still like to understand what's going on.
In particular, I'm perplexed by a situation where swap goes from 2% used to 100% used over a period of around 20 minutes. When this happens though, there doesn't seem to be any increase in the swap rate. The server has an 8 GB swap partition, so I would expect a page out rate of thousands of pages a second in order to fill the available space.
Furthermore, there doesn't seem to be any spike in memory utilisation at the time that causes the server to swap.
Can anyone give an explanation for the observed behaviour? The server is RHEL 4.8 (old, I know) with a 2.6.9 Kernel. I've included some output from sar below.
Swap utilisation jumps from 2% to 100% in around 20 minutes:
$ sar -r -f sa12
....
06:00:01 PM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
06:10:01 PM 4583856 11847032 72.10 79676 319804 8178056 207864 2.48 19816
06:20:01 PM 4720904 11709984 71.27 38840 225108 8178400 207520 2.47 19124
06:30:02 PM 4839160 11591728 70.55 1404 144948 1968152 6417768 76.53 6227068
06:40:02 PM 4827016 11603872 70.62 1640 145484 10232 8375688 99.88 8182968
06:50:02 PM 4836376 11594512 70.57 844 123304 60 8385860 100.00 8193044
07:00:01 PM 4825764 11605124 70.63 920 128108 4516 8381404 99.95 8188680
....
Here are the swapping statistics for the same period.
$ sar -W -f sa12
....
06:00:01 PM pswpin/s pswpout/s
06:10:01 PM 0.00 0.00
06:20:01 PM 0.00 0.00
06:30:02 PM 0.68 0.00
06:40:02 PM 1.85 0.00
06:50:02 PM 5.07 0.00
07:00:01 PM 8.62 0.00
....
cron
at saturdays?/proc/sys/vm/swappiness
?