Every few days, at differing times my system starts swapping like crazy and the load goes extremely high such that the system is very slow to respond. Sometimes I have waited 4 hours for it to recover, other times I have just rebooted or shutdown using the Magic SysRq key (so yes the kernel still responds ok and it responds promptly). The swap space and operating system is on a pair of mirrored SSDs.
When the system is going crazy the
kswapd is always using the most CPU and the rest of the tasks near the top vary.
Here's a sample of the
top output when the system is going crazy. The memory used at around 4.5GB is about normal for the system.
Sometimes drop caches fixes the problem. Other times it does not. Sometimes stopping new processes from starting works to restore the system, other time not - eg stopping
cron (which I have scheduling several regular processes) or
nagios (which fires off regular plugins)
Sometimes I've found that the OOM killer has killed some processes to recover some memory but this does not always fix the system either.
The load on this system has not really changed for a long time and suddenly this has started happening. I thought it might have started when I upgraded to kernel 4.16.* but I reverted to kernel 4.15.* and that did not fix it.
I did write a script to collect various pieces of information on a regular basis and from that I can clearly see the times when the system starts going crazy. I don't have that data available at the moment, I'll have to provide that the next time the system goes crazy.
Here's how the load progresses, as recorded by a script I wrote: load progression history
when it says "Trying to remediate high load" it is trying to drop the caches by using
sync;echo 3 > /proc/sys/vm/drop_caches
Any ideas what I should be looking at? I need some help trying to work out what is going on. Thanks