1

Every few days, at differing times my system starts swapping like crazy and the load goes extremely high such that the system is very slow to respond. Sometimes I have waited 4 hours for it to recover, other times I have just rebooted or shutdown using the Magic SysRq key (so yes the kernel still responds ok and it responds promptly). The swap space and operating system is on a pair of mirrored SSDs.

When the system is going crazy the kswapd is always using the most CPU and the rest of the tasks near the top vary.
Here's a sample of the top output when the system is going crazy. The memory used at around 4.5GB is about normal for the system.

top output

Sometimes drop caches fixes the problem. Other times it does not. Sometimes stopping new processes from starting works to restore the system, other time not - eg stopping cron (which I have scheduling several regular processes) or nagios (which fires off regular plugins)

Sometimes I've found that the OOM killer has killed some processes to recover some memory but this does not always fix the system either.

The load on this system has not really changed for a long time and suddenly this has started happening. I thought it might have started when I upgraded to kernel 4.16.* but I reverted to kernel 4.15.* and that did not fix it.

I did write a script to collect various pieces of information on a regular basis and from that I can clearly see the times when the system starts going crazy. I don't have that data available at the moment, I'll have to provide that the next time the system goes crazy.

Here's how the load progresses, as recorded by a script I wrote: load progression history

when it says "Trying to remediate high load" it is trying to drop the caches by using sync;echo 3 > /proc/sys/vm/drop_caches

Any ideas what I should be looking at? I need some help trying to work out what is going on. Thanks

closed as too broad by Rui F Ribeiro, Romeo Ninov, andcoz, schily, Tomasz Jun 20 '18 at 13:19

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    Please, don't post images of text. – Kusalananda Jun 16 '18 at 11:28
  • systemd-cgtop (and press m) will show the memory usage according to cgroups which will be roughly broken down by service. This will include the disk cache memory usage. (disk cache pages are considered owned by a single specific cgroup). You could use this to get a high-level overview, of which services are using memory. At the same time, make sure to check df -h -t tmpfs. If you have big files in tmpfs, bad things happen and they don't show up in certain common tools like top :). – sourcejedi Jun 16 '18 at 11:55
  • If I interpret your "load progression history" link correctly, it looks to me like drop_caches is finding very little it can drop in this log. Note that drop_caches does not drop any file which is mapped by a currently running process. (I.e., any running executable, or mmap()ed file). I think you might expect from this that all the significant memory usage will show up as RESident in top PLUS any tmpfs files, PLUS other fun types of "shmem" such as memfd. – sourcejedi Jun 16 '18 at 12:00
  • Sorry, the systemd-cgtop feature only started working with relatively recent versions of systemd (which enable the cgroup memory controller), I'm not sure if it will show on Fedora 27. Works on Fedora 28. You can enable it manually with DefaultMemoryAccounting= in systemd-system.conf (and restart the system, I guess). – sourcejedi Jun 16 '18 at 12:23
  • Please provide the contents of /proc/meminfo while the issue is occurring. As text, not as an image. – Patrick Jun 16 '18 at 15:03
2

The system is obviously RAM starved at several points in time, the OOM killer events and the swap use should tell you that.

Nevertheless, at 16GB RAM and running MythTV+MySQL+Nagios+Apache, pserver, CVS and God knows what else (am only guessing things from your top output) in the same server might be too much for it. Both in RAM and I/O.

We also do not know how many events you have in that Nagios, and how is their scheduling. Too much events in a too short time, and they will start being launched before they go full round, and will eat any machine no matter how powerful it is. And probably they won´t have time to go full round when the machine starts to get starved, and suddenly you have a boatload of Nagios checks going on. And on top of that you say you have cronjobs...

I would think of having more resources, and running a couple of machines and/or VMs with all those services. A consumer machine won´t also withstand so much I/O, at some point if you are doing some serious work, you have to go server grade.

Obviously at some point, you also have to do some Nagios and MySQL proper configuration management/DBA intervention for managing your available resources.

How to organize all of this is out of scope of this answer. Just as the old adage says, do not put all your eggs in one basket.

PS. read your top image here and see how much RAM each process is roughly using (they could be using more swap, but you know they are at least using that. Do your math). That top is a very crude indicator your RAM calculations are way off bellow what the system needs.

PS2. I am mostly guessing things, and obviously do not know your specific configuration. Take the text as a general recommendation guide.

  • I'm not sure how valid your comments are, about what this server is running. This server has been running the same load just fine for years. Obviously something has changed just recently. I don't think it is the load it is running, but let me think some more about that. I don't understand what you mean in your "PS". – user3421823 Jun 16 '18 at 12:00
  • 1
    Sorry, I don't know where your numbers are coming from. For example, you mention 2G for Myth. However, the top I pasted shows that Myth has about 8k left as resident and a 5G virtual mem size. Even now as I write this I see about 100M resident and 5G virtual size. Another example I don't understand is you mention +4G of swap. Where does that number come from? The top shows I have 10G of swap space – user3421823 Jun 16 '18 at 21:42
  • Indeed I made a mistake. You need to do a ps ordered by RAM usage. Those memory shown on top seems strangely low... – Rui F Ribeiro Jun 16 '18 at 22:17
  • The per process resident memory shown is lower than normal, I'm guessing it is swapped out. The per process virtual size is about normal. The overall used, free, buff/cache etc is what I see when it is normal. – user3421823 Jun 16 '18 at 22:24

Not the answer you're looking for? Browse other questions tagged or ask your own question.