Can I distinguish between swap thrashing and non-swap high IO latency somehow?

Question

My system becomes much, much less responsive, every time I clone a VM image. I'm using virt-manager, and I can see the IO is performed by several qemu-img convert threads.

I tried to gather some information, and it looked like there might have been a lot of swapping (I/O on the swap partition). I have 8GB RAM, and 2GB swap. During and after the clone, free -h showed that 100% of the swap space was used. However, that doesn't tell me how much the system was swapping at the time. Something might have filled the swap before I cloned the VM.

I am using a mechanical hard drive. My current OS is Fedora Linux 28.

How can I be prepared when this happens, to gather the relevant information, and see whether there is a lot of swapping or not?

I want some sort of record I can look back at, and collate different information. I.e. if I run a simple top or iotop command they will overwrite their old output, and I don't want that.

Update

I still think it is useful to gather information like this, as I suggested in my original answer. But:

I found the two biggest reasons for my system becoming almost entirely un-responsive. Neither of them were to do with thrashing, (swap or non-swap).

Firstly there was a bug in gnome-shell. It waited on fsync() in the main thread. This is the main thread of the (Wayland) graphics server. While it waits, the display will not be updated. The bug was observed in gnome-shell 3.30.2, and is supposed to be fixed in version 3.32.

(It might be possible to diagnose this by comparing a Wayland session of GNOME with an Xorg one. In the Xorg one, the mouse cursor should still be able to move).

The second problem was an known issue with ext4. While a file is being written, fsync() on other files could "end up waiting for an indefinite amount of time". So this affects the gnome-shell bug.

Even when gnome-shell was fixed, the long delays in ext4 appeared to affect Firefox. Fixes for the above issue with ext4 have been merged in Linux kernel release 5.3.[1][2].

The gory details are recorded here: Simple file copy (or write) causes ten second+ latency on Linux filesystem

Answer 1

vmstat is the traditional Linux command to track memory, swap and IO. E.g. vmstat 5 will print a line of statistics every 5 seconds.

atop is a newer tool, which is very powerful. Running atop looks similar to top, but it has a lot more information. When you want a log, atop -w <file> will instead write a binary log, which can be read with atop -r <file>. The atop package also includes a service to automatically write logs, using 10 minute intervals (by default).

Update: atop 2.4.0 adds support for Linux Pressure Stall Information. I expect this would help detect stalls due to memory pressure. The memory pressure statistic (shown as ms or mf in atop) could detect both swap and non-swap thrashing. Technically, this means it does not help distinguish between swap and non-swap thrashing :-). But I would have liked to have this information. I did not have much confirmation that thrashing was my problem... and as it turned out in the update, thrashing was actually not the main problem.

Regarding the main problem I had: I think it is harder to gather information on that. There is one generic tracing approach that might help: offcputime --state 2. Although it took some effort to install this tool.

Previous answer

I have atop installed, with a workaround to make it work correctly when I suspend my laptop overnight.

The logs from the atop service can be very informative, if you have a prolonged problem with memory consumption. It might miss shorter problems (due to the default 10 minute logging interval).

• My problem looks like it lasted 10-20 minutes.
• Swap usage rose from 1.4G in the previous sample, to 2G (100%).
• The qemu-img threads themselves did not have a large size in RAM. The qemu-img process only had 25M resident.
• swout was 175735. That's measured in pages of 4096 bytes, which means about 0.7G was swapped out.

At the same time, cache grew from 0.8G to 2.3G. free memory stayed at 0.1G.

I suspect qemu-img is doing cached IO, the cache is pushing out other memory, and this is what causes the swapping. If I didn't have swap space, I expect there would still be some problem; i.e. loaded program code and other caches would be evicted instead.

If I drop_caches and then cp a 16G file, I can trigger quite a lot of swapping. I think the same problem is being reproduced with cp; I don't think this is limited to some specific detail of qemu-img convert.