How to track down the source of I/O spikes?

Question

I am using Linux as a desktop, and keep getting seemingly random I/O Spikes. The machine becomes unusably slow. At first I thought it was just that I don't have enough memory. But looking at the output of free and top there was nothing out of the ordinary. Same thing for CPU load.

If I don't kill the offending process right away, the machine grinds quickly (in about 10min) to a near complete halt and I have to hard-reset.

A coworker told me he had similar issues and he notices I/O spikes. We have the same machines btw (they ware provided by the company).

I have also noticed that those spikes often occur when opening a new tab in Chrome. But it think it has happened on other occasions as well. Like opening a tab in Firefox, or just randomly out of the blue.

I decided to run dstat and have a look at the output, but changed my mind and used the ksysguard, simply because the change is easier to see.

A screenshot of the monitor:

As you can see, there is a spike in disk I/O which coincides with a spike in system load. Strangely, the memory usage goes down at that moment. Could it be related to swap?

There are two spikes. The first one is the one I felt immediately, and coincided with clicking on a link in the Chrome (not even opening a tab, but triggering JavaScript code). I immediately clicked the "close" button on that tab and the machine became responsive again. The second one had no noticeable effect. It is possible that the fact that memory usage went down is because I closed the tab, as it happens slightly after the I/O spike.

The whole spike (the first one) lasted for about 10s.

Any idea what to look out for?

Answer 1

I have found the culprit.

It was indeed due to a faulty swap setup. My fstab listed /dev/mapper/cryptswap as swap space. This was nonexistent. My guess is that as soon as the system needed to swap, it saw swapspace defined, but that device did not exist anywhere.

For testing I simply created empty files as swapspace. Since then, the machine seems to run a lot more stable. I have had not spikes/crashes ever since. But I do see the new swap files being used.