My system becomes much, much less responsive, every time I clone a VM image. I'm using virt-manager
, and I can see the IO is performed by several qemu-img convert
threads.
I tried to gather some information, and it looked like there might have been a lot of swapping (I/O on the swap partition). I have 8GB RAM, and 2GB swap. During and after the clone, free -h
showed that 100% of the swap space was used. However, that doesn't tell me how much the system was swapping at the time. Something might have filled the swap before I cloned the VM.
I am using a mechanical hard drive. My current OS is Fedora Linux 28.
How can I be prepared when this happens, to gather the relevant information, and see whether there is a lot of swapping or not?
I want some sort of record I can look back at, and collate different information. I.e. if I run a simple top
or iotop
command they will overwrite their old output, and I don't want that.
Update
I still think it is useful to gather information like this, as I suggested in my original answer. But:
I found the two biggest reasons for my system becoming almost entirely un-responsive. Neither of them were to do with thrashing, (swap or non-swap).
Firstly there was a bug in gnome-shell
. It waited on fsync() in the main thread. This is the main thread of the (Wayland) graphics server. While it waits, the display will not be updated. The bug was observed in gnome-shell 3.30.2, and is supposed to be fixed in version 3.32.
(It might be possible to diagnose this by comparing a Wayland session of GNOME with an Xorg one. In the Xorg one, the mouse cursor should still be able to move).
The second problem was an known issue with ext4. While a file is being written, fsync() on other files could "end up waiting for an indefinite amount of time". So this affects the gnome-shell bug.
Even when gnome-shell was fixed, the long delays in ext4 appeared to affect Firefox. Fixes for the above issue with ext4 have been merged in Linux kernel release 5.3.[1][2].
The gory details are recorded here: Simple file copy (or write) causes ten second+ latency on Linux filesystem