0

My system becomes much, much less responsive, every time I clone a VM image. I'm using virt-manager, and I can see the IO is performed by several qemu-img convert threads.

I tried to gather some information, and it looked like there might have been a lot of swapping (I/O on the swap partition). I have 8GB RAM, and 2GB swap. During and after the clone, free -h showed that 100% of the swap space was used. However, that doesn't tell me how much the system was swapping at the time. Something might have filled the swap before I cloned the VM.

I am using a mechanical hard drive. My current OS is Fedora Linux 28.

How can I be prepared when this happens, to gather the relevant information, and see whether there is a lot of swapping or not?

I want some sort of record I can look back at, and collate different information. I.e. if I run a simple top or iotop command they will overwrite their old output, and I don't want that.

Update

I still think it is useful to gather information like this, as I suggested in my original answer. But:

I found the two biggest reasons for my system becoming almost entirely un-responsive. Neither of them were to do with thrashing, (swap or non-swap).

Firstly there was a bug in gnome-shell 3.30.2. It waited on fsync() in the main thread. This is the main thread of the (Wayland) graphics server. While it waits, the display will not be updated.

The second problem was an known issue with ext4. While a file is being written, fsync() on other files could "end up waiting for an indefinite amount of time". So this affects the gnome-shell bug.

Even after gnome-shell was fixed, the long delays in ext4 appeared to affect Firefox. Patches for ext4 have been submitted which seem to address this problem. Hopefully the patches will be included in the Linux kernel release 5.3.

For details, see: Simple file copy (or write) causes ten second+ latency on Linux filesystem

  • I suppose you could try and ask for the I/O to bypass the page cache and see if that makes a difference... – Anon Jun 23 at 19:34
  • @Anon ah, I remember what this was now! I have added an "Update" in the question, to clarify what was really happening. To answer your comment, the linked question I wrote claims that your idea did help. But the big problems I had were probably not the problem you were guessing! I don't really know why bypassing page cache would have helped for the biggest problems I had... Also, now I should not be affected by the combination of those two large problems... but I am not certain whether there are still other equal or otherwise significant problems for this workload. – sourcejedi Jun 24 at 22:06
0

vmstat is the traditional Linux command to track memory, swap and IO. E.g. vmstat 5 will print a line of statistics every 5 seconds.

atop is a newer tool, which is very powerful. Running atop looks similar to top, but it has a lot more information. When you want a log, atop -w <file> will instead write a binary log, which can be read with atop -r <file>. The atop package also includes a service to automatically write logs, using 10 minute intervals (by default).

Update: atop 2.4.0 adds support for Linux Pressure Stall Information. I expect this would help detect stalls due to memory pressure. The memory pressure statistic (shown as ms or mf in atop) could detect both swap and non-swap thrashing. Technically, this means it does not help distinguish between swap and non-swap thrashing :-). But I would have liked to have this information. I did not have much confirmation that thrashing was my problem... and as it turned out in the update, thrashing was actually not the main problem.

Regarding the main problem I had: I think it is harder to gather information on that. There is one generic tracing approach that might help: offcputime --state 2. Although it took some effort to install this tool.

Previous answer

I have atop installed, with a workaround to make it work correctly when I suspend my laptop overnight.

The logs from the atop service can be very informative, if you have a prolonged problem with memory consumption. It might miss shorter problems (due to the default 10 minute logging interval).

  • My problem looks like it lasted 10-20 minutes.
  • Swap usage rose from 1.4G in the previous sample, to 2G (100%).
  • The qemu-img threads themselves did not have a large size in RAM. The qemu-img process only had 25M resident.
  • swout was 175735. That's measured in pages of 4096 bytes, which means about 0.7G was swapped out.

At the same time, cache grew from 0.8G to 2.3G. free memory stayed at 0.1G.

I suspect qemu-img is doing cached IO, the cache is pushing out other memory, and this is what causes the swapping. If I didn't have swap space, I expect there would still be some problem; i.e. loaded program code and other caches would be evicted instead.

If I drop_caches and then cp a 16G file, I can trigger quite a lot of swapping. I think the same problem is being reproduced with cp; I don't think this is limited to some specific detail of qemu-img convert.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.