1

My system becomes much, much less responsive, every time I clone a VM image. I'm using virt-manager, and I can see the IO is performed by several qemu-img convert threads.

I tried to gather some information, and it looked like there might have been a lot of swapping (I/O on the swap partition). I have 8GB RAM, and 2GB swap. During and after the clone, free -h showed that 100% of the swap space was used. However, that doesn't tell me how much the system was swapping at the time. Something might have filled the swap before I cloned the VM.

I am using a mechanical hard drive. My current OS is Fedora Linux 28.

How can I be prepared when this happens, to gather the relevant information, and see whether there is a lot of swapping or not?

I want some sort of record I can look back at, and collate different information. I.e. if I run a simple top or iotop command they will overwrite their old output, and I don't want that.

Update

I still think it is useful to gather information like this, as I suggested in my original answer. But:

I found the two biggest reasons for my system becoming almost entirely un-responsive. Neither of them were to do with thrashing, (swap or non-swap).

Firstly there was a bug in gnome-shell. It waited on fsync() in the main thread. This is the main thread of the (Wayland) graphics server. While it waits, the display will not be updated. The bug was observed in gnome-shell 3.30.2, and is supposed to be fixed in version 3.32.

(It might be possible to diagnose this by comparing a Wayland session of GNOME with an Xorg one. In the Xorg one, the mouse cursor should still be able to move).

The second problem was an known issue with ext4. While a file is being written, fsync() on other files could "end up waiting for an indefinite amount of time". So this affects the gnome-shell bug.

Even when gnome-shell was fixed, the long delays in ext4 appeared to affect Firefox. Fixes for the above issue with ext4 have been merged in Linux kernel release 5.3.[1][2].

The gory details are recorded here: Simple file copy (or write) causes ten second+ latency on Linux filesystem

4
  • I suppose you could try and ask for the I/O to bypass the page cache and see if that makes a difference...
    – Anon
    Jun 23, 2019 at 19:34
  • @Anon ah, I remember what this was now! I have added an "Update" in the question, to clarify what was really happening. To answer your comment, the linked question I wrote claims that your idea did help. But the big problems I had were probably not the problem you were guessing! I don't really know why bypassing page cache would have helped for the biggest problems I had... Also, now I should not be affected by the combination of those two large problems... but I am not certain whether there are still other equal or otherwise significant problems for this workload.
    – sourcejedi
    Jun 24, 2019 at 22:06
  • Oh interesting little catch on Fsync thing. Over time, I started noticing that my Manjaro never reaches a state where the Desktop is frozen under situations of heavy swapping, but my Debian and MX desktop freezes very easily. I wonder if it's something related to that and "waiting for refresh"
    – Winampah
    Feb 16, 2022 at 15:29
  • @Winampah text is now updated to indicate the versions of software which are supposed to fix the issues that I was able to specifically identify.
    – sourcejedi
    Feb 16, 2022 at 18:55

1 Answer 1

1

vmstat is the traditional Linux command to track memory, swap and IO. E.g. vmstat 5 will print a line of statistics every 5 seconds.

atop is a newer tool, which is very powerful. Running atop looks similar to top, but it has a lot more information. When you want a log, atop -w <file> will instead write a binary log, which can be read with atop -r <file>. The atop package also includes a service to automatically write logs, using 10 minute intervals (by default).

Update: atop 2.4.0 adds support for Linux Pressure Stall Information. I expect this would help detect stalls due to memory pressure. The memory pressure statistic (shown as ms or mf in atop) could detect both swap and non-swap thrashing. Technically, this means it does not help distinguish between swap and non-swap thrashing :-). But I would have liked to have this information. I did not have much confirmation that thrashing was my problem... and as it turned out in the update, thrashing was actually not the main problem.

Regarding the main problem I had: I think it is harder to gather information on that. There is one generic tracing approach that might help: offcputime --state 2. Although it took some effort to install this tool.

Previous answer

I have atop installed, with a workaround to make it work correctly when I suspend my laptop overnight.

The logs from the atop service can be very informative, if you have a prolonged problem with memory consumption. It might miss shorter problems (due to the default 10 minute logging interval).

  • My problem looks like it lasted 10-20 minutes.
  • Swap usage rose from 1.4G in the previous sample, to 2G (100%).
  • The qemu-img threads themselves did not have a large size in RAM. The qemu-img process only had 25M resident.
  • swout was 175735. That's measured in pages of 4096 bytes, which means about 0.7G was swapped out.

At the same time, cache grew from 0.8G to 2.3G. free memory stayed at 0.1G.

I suspect qemu-img is doing cached IO, the cache is pushing out other memory, and this is what causes the swapping. If I didn't have swap space, I expect there would still be some problem; i.e. loaded program code and other caches would be evicted instead.

If I drop_caches and then cp a 16G file, I can trigger quite a lot of swapping. I think the same problem is being reproduced with cp; I don't think this is limited to some specific detail of qemu-img convert.

3
  • Yes I have been noticing the Linux CACHE system is not very smart. The system will not distinguish "Available" memory from "Total Available including Swap". So this means you can have situations where you can try copying a file to another location and Linux will put that entire file on Cache simply based on the fact there's Swap space with sufficient free amount, causing unnecessary Swapping and unnecessary extra activity on Disk (specially dangerous for SSD's I suppose? I don't have SSD)
    – Winampah
    Feb 16, 2022 at 15:33
  • Was there ever a solution to this when handling HUGE files such as the ones you're dealing with there?
    – Winampah
    Feb 16, 2022 at 15:34
  • @Winampah This is just too long ago now. I still clone VMs occasionally. I think it works fine. But as well as both of the bug fixes, I'm using an SSD now. Can't remember whether the bugfixes resolved my problem fully, or whether it was still a bit janky and the SSD fixed it. I think the bugfixes are pretty good, but HDDs are also kinda bad. And don't run out of RAM, that's always super bad :-)
    – sourcejedi
    Feb 16, 2022 at 19:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .