Spike in swap utilisation without high swap rate

Question

Every Saturday, for reasons unknown to me, one of my servers experiences a spike in swap utilisation. It's not necessarily a problem as there is plenty of free memory available, but I'd still like to understand what's going on.

In particular, I'm perplexed by a situation where swap goes from 2% used to 100% used over a period of around 20 minutes. When this happens though, there doesn't seem to be any increase in the swap rate. The server has an 8 GB swap partition, so I would expect a page out rate of thousands of pages a second in order to fill the available space.

Furthermore, there doesn't seem to be any spike in memory utilisation at the time that causes the server to swap.

Can anyone give an explanation for the observed behaviour? The server is RHEL 4.8 (old, I know) with a 2.6.9 Kernel. I've included some output from sar below.

Swap utilisation jumps from 2% to 100% in around 20 minutes:

$ sar -r -f sa12
....
06:00:01 PM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
06:10:01 PM   4583856  11847032     72.10     79676    319804   8178056    207864      2.48     19816
06:20:01 PM   4720904  11709984     71.27     38840    225108   8178400    207520      2.47     19124
06:30:02 PM   4839160  11591728     70.55      1404    144948   1968152   6417768     76.53   6227068
06:40:02 PM   4827016  11603872     70.62      1640    145484     10232   8375688     99.88   8182968
06:50:02 PM   4836376  11594512     70.57       844    123304        60   8385860    100.00   8193044
07:00:01 PM   4825764  11605124     70.63       920    128108      4516   8381404     99.95   8188680
....

Here are the swapping statistics for the same period.

$ sar -W -f sa12
....
06:00:01 PM  pswpin/s pswpout/s
06:10:01 PM      0.00      0.00
06:20:01 PM      0.00      0.00
06:30:02 PM      0.68      0.00
06:40:02 PM      1.85      0.00
06:50:02 PM      5.07      0.00
07:00:01 PM      8.62      0.00
....

Answer 1

Interesting fact: swap usage increased from 0 to 8GB... meanwhile, not a single byte were written to disk (sar's pswpout/s show 0). So my assumption is that the swap was allocated/reserved, but not consumed/used.

My best guess is that your server uses vm.overcommit_memory=2 (read vm.overcommit_memory documentation, vm overcommit-accounting). In which case, each byte allocated is counted as used. [I didn't checked].

You may also want to read the thread Linux: Total swap used = swap used by processes + ??.

Answer 2

This doesn't directly answer your question, but the following script may be useful in your investigation. It lets you know how much swap space each individual process is using:

Find out what processes are using swap

If you have a rough idea of when the swapping starts, you could set cron to run this script around that time.