0

My system becomes much, much less responsive, every time I clone a VM image. I'm using virt-manager, and I can see the IO is performed by several qemu-img convert threads.

I tried to gather some information, and it looked like there might have been a lot of swapping (I/O on the swap partition). I have 8GB RAM, and 2GB swap. During and after the clone, free -h showed that 100% of the swap space was used. However, that doesn't tell me how much the system was swapping at the time. Something might have filled the swap before I cloned the VM.

I am using a spinning hard drive. My current OS is Fedora Linux 28.

How can I be prepared when this happens, to gather the relevant information, and see whether there is a lot of swapping or not?

I want some sort of record I can look back at, and collate different information. I.e. if I run a simple top or iotop command, they will overwrite their old output; I don't want that.

0

I have atop installed, with a workaround to make it work correctly when I suspend my system (a laptop) overnight.

The atop logs can be very informative, if you have a prolonged problem with memory consumption. The (default) sample frequency is 10 minutes, so it might miss shorter problems.

  • My problem looks like it lasted 10-20 minutes.
  • Swap usage rose from 1.4G in the previous sample, to 2G (100%).
  • The qemu-img threads themselves did not have a large size in RAM. The qemu-img process only had 25M resident.
  • swout was 175735. That's measured in pages of 4096 bytes, which means about 0.7G was swapped out.

At the same time, cache grew from 0.8G to 2.3G. free memory stayed at 0.1G.

I suspect qemu-img is doing cached IO, the cache is pushing out other memory, and this is what causes the swapping. If I didn't have swap space, I expect there would still be some problem; i.e. loaded program code and other caches would be evicted instead.

It seems like my qemu-img convert is more complex than cp, and is making a lot of "active" pages in the cache.

One could think of the inactive list as a sort of probational status for pages that kernel isn't sure are worth keeping. Pages can get there from the active list as described above, but there's another way to inactive status as well: file-backed pages, when they are faulted in, are placed in the inactive list. It is quite common that a process will only access a file's contents once; requiring a second access before moving file-backed pages to the active list lets the kernel get rid of single-use data relatively quickly.

https://lwn.net/Articles/495543/

If I drop_caches and then cp a 16G file, I seem to have a similar problem, where it triggers quite a lot of swapping. So the kernel doesn't seem to be getting rid of the single-use data as quickly as I hoped it would.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.