0

I have been testing disk reads, and how Linux reports disk performance. The following command is a sequential read from my mechanical hard drive.

$ sudo dd if=/dev/sda of=/dev/null bs=512k count=10000
10000+0 records in
10000+0 records out
5242880000 bytes (5.2 GB, 4.9 GiB) copied, 35.102 s, 149 MB/s

I noticed that tests like this can sometimes freeze the entire GUI for tens of seconds. Is the diagnosis below correct, yes or no?

(if anyone has a better idea how to make this information into a question and answer, feel to comment :-P)

While frozen, I cannot even move the mouse cursor. I also cannot switch to a text console using Ctrl+Alt+F6.

I can use Alt+SysRQ+R, then Ctrl+Alt+F6 will work. But that breaks the GUI session: when I switch back to the GUI using Ctrl+Alt+F2, if I ever press Ctrl+C, the entire GUI session will be killed immediately, like a text console program would :-). This is expected behaviour.

My GUI is the default GUI for Fedora Workstation 29. It is gnome-shell-3.30.2-1.fc29.x86_64. This runs as a Wayland compositor. I.e. gnome-shell accesses the graphics device itself, not through Xorg ("X11" / "X Window Server"). There might be some differences if I log in using "GNOME on Xorg" instead, but I have not tested it yet.

The following test results apply to a custom kernel build, which is close to both kernel version 5.1 and the Fedora kernel configuration. I have no reason to expect recent changes in this behaviour. However it may change over time, and I have not checked.

1) Running the above read test immediately after boot, did not freeze at all.

2) I went on to read more than the size of my RAM, and that did not freeze either:

$ free -h
              total        used        free      shared  buff/cache   available
Mem:          7.7Gi       847Mi       485Mi       151Mi       6.4Gi       6.4Gi
Swap:            0B          0B          0B

$ sudo dd if=/dev/sda of=/dev/null bs=512k skip=10000 count=20000
20000+0 records in
20000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 71.4632 s, 147 MB/s

3) If I read the part of the disk more than once, which is smaller than my "available" RAM, I can reproduce the freeze. I moved the mouse continuously throughout the test. During the following command, the cursor froze for 19 and then 13 seconds:

$ sudo sh -c 'dd if=/dev/sda of=/dev/null bs=512k count=9000 &&
              dd if=/dev/sda of=/dev/null bs=512k count=9000 &&
              dd if=/dev/sda of=/dev/null bs=512k skip=10000 count=20000'
9000+0 records in
9000+0 records out
4718592000 bytes (4.7 GB, 4.4 GiB) copied, 25.1508 s, 188 MB/s
9000+0 records in
9000+0 records out
4718592000 bytes (4.7 GB, 4.4 GiB) copied, 0.883119 s, 5.3 GB/s
20000+0 records in
20000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 71.2398 s, 147 MB/s

4) If I add iflag=direct to these commands, there is never any freeze.

Diagnosis

Tests 1-3 are "buffered reads". They read the data into the kernel page cache, so that subsequent reads of the same data can be fulfilled much more quickly. E.g. 5.3 GB/s instead of 188 MB/s :-).

These examples are reading from a block device. In the case of a block device, the cache might be dropped as soon as the program finishes and closes the device. The cache was not dropped in my case, because partitions of my /dev/sda were still open, i.e. mounted as a filesystem. For proof, see: "When the program closes the block device file, Linux flushes the associated cache [...]"

In test 1), I had enough free / "available" RAM to cache all the read data.

In test 2), I benefited from the following kernel heuristic:

File-backed pages, when they are faulted in, are placed in the inactive list. It is quite common that a process will only access a file's contents once; requiring a second access before moving file-backed pages to the active list lets the kernel get rid of single-use data relatively quickly.

-- LWN (Linux Weekly News), 2012

In test 3, I accessed the same area of the disk twice. Therefore I inserted 4.4 GiB of cached disk reads into the "active" LRU list.

Linux page replacement does not simply apply "Least Recently Used" to the active list. I suspect my overall experience would probably be much worse if it did. Whatever the algorithm is, during the third dd command, Linux is somehow decides to drop some critical pages that gnome-shell needs. So a number of pages have to be re-read from disk, causing delays.

The delays might be exacerbated by the IO scheduler favouring sequential read throughput. The above results were using the kernel's default IO scheduler, which is mq-deadline.

You can avoid the freeze by adding iflag=direct to the dd command, because this avoids placing data into the kernel page cache. Another way to avoid the freeze is to drop caches before each dd command:

echo 1 | sudo tee /proc/sys/vm/drop_caches

This is usually what you want to do anyway, when you are testing disk performance like this. You usually want to measure the disk performance, without any kernel page cache. Dropping the page cache is not quite as harsh as you might think. At least currently, Linux does not drop pages of code which belong to currently running programs (or any other page which they have mapped into their memory).

(I was running these tests for a different reason though. I was not looking at the reported disk speed. I was looking at how Linux reported disk busy%, and CPU time spent in "iowait").

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.