How to deal with blocking I/O reducing the performance to a crawl?

Question

I have a computer which is experiencing some serious I/O issues.

Software: Kali-rolling, Linux x86-desktop-1 4.12.0-kali2-amd64 #1 SMP Debian 4.12.13-1kali2 (2017-10-03) x86_64 GNU/Linux. But I've had various kernels, stock ones, and custom ones.

Hardware: The relevant hardware is as follows. Ryzen 1800X CPU, 16 GiB RAM, MSI X370 SLI PLUS with the latest BIOS (version 3.6, but I've also tried three others), M.2: Crucial® MX300 525GB M.2 SSD and 256 GB Samsung SSD 840 PRO.

The problem: The problem can perhaps be best described just from a user perspective at first:

I'm copying a lot of small files, such as the Linux kernel source tree. This results in so much slowdown, that even a simple command such as the "ls" binary, or "dmesg" can take 15 seconds or more to run. The whole system freezes in terms of I/O, anything that requires any form of I/O will block until whatever is blocking it allows it to get through the I/O queue.

I've noticed the problem when doing the following:

• Playing a video file over the network, from a CIFS resource. This form of I/O can introduce a second or two delay in the video streaming.
• Playing an MP3 file over the network, while using a web browser can make the audio stutter.
• Copying something locally on the SSD or M.2 device, perhaps especially when it involves many small files.

The problem started since I installed the OS and has always been there.

My reasoning and how I've tried to solve this: The hardware should be more than capable of handling multiple I/O requests at the same time. The idea that this would all be caused by a faulty SSD/M.2 seems unreasonable, as I've tried both an SSD, and a physically separate M.2 device, they both have the same problem. Furthermore, it also seems unreasonable that the stock kali kernel would have this particular I/O issue, especially since I've tried multiple kernels: 4.9, 4.12, 4.13.2, 4.13.10. I've considered that perhaps there are BIOS options that would affect this performance, but I couldn't find anything, regardless, I'm running setup defaults and have tried multiple BIOS versions, including version 3.4, 3.5, and 3.6.

I've checked dmesg for any I/O errors, there are none.

I've also considered that there are multiple I/O schedulers for Linux:

$ cat /sys/block/sda/queue/scheduler 
noop deadline [cfq] 

The scheduler can be changed by issuing e.g.

echo "noop" > /sys/block/sda/queue/scheduler

See this question for more information: https://stackoverflow.com/questions/1009577/selecting-a-linux-i-o-scheduler

However, I've tried noop, deadline, and CFQ, they don't appear to affect the problem at all.

I also wanted to check the performance of the non-volatile devices:

hdparm -t /dev/sda

/dev/sda:
 Timing buffered disk reads: 1112 MB in  3.01 seconds = 369.69 MB/sec
hdparm -t /dev/sda

/dev/sda:
 Timing buffered disk reads: 1122 MB in  3.00 seconds = 373.53 MB/sec

The performance is nothing staggering, I should actually have more than this. But even with this type of performance, these issues should be non-existent. I'm not sure if the issue I'm having is related to this poor I/O performance.

When I was running Gentoo on this same system, I got this on the 840 PRO device (hdparm -t):

510.82 MB/sec
524.05 MB/sec

How should I go about debugging this issue? Is it obvious to anyone what the problem is?

Answer 1

It does not explain what is happening but you could try to work around the problem by using ionice for drive operations and tc (traffic shaping) for network operations.