I have a computer which is experiencing some serious I/O issues.
Software: Kali-rolling, Linux x86-desktop-1 4.12.0-kali2-amd64 #1 SMP Debian 4.12.13-1kali2 (2017-10-03) x86_64 GNU/Linux. But I've had various kernels, stock ones, and custom ones.
Hardware: The relevant hardware is as follows. Ryzen 1800X CPU, 16 GiB RAM, MSI X370 SLI PLUS with the latest BIOS (version 3.6, but I've also tried three others), M.2: Crucial® MX300 525GB M.2 SSD and 256 GB Samsung SSD 840 PRO.
The problem: The problem can perhaps be best described just from a user perspective at first:
I'm copying a lot of small files, such as the Linux kernel source tree. This results in so much slowdown, that even a simple command such as the "ls" binary, or "dmesg" can take 15 seconds or more to run. The whole system freezes in terms of I/O, anything that requires any form of I/O will block until whatever is blocking it allows it to get through the I/O queue.
I've noticed the problem when doing the following:
- Playing a video file over the network, from a CIFS resource. This form of I/O can introduce a second or two delay in the video streaming.
- Playing an MP3 file over the network, while using a web browser can make the audio stutter.
- Copying something locally on the SSD or M.2 device, perhaps especially when it involves many small files.
The problem started since I installed the OS and has always been there.
My reasoning and how I've tried to solve this: The hardware should be more than capable of handling multiple I/O requests at the same time. The idea that this would all be caused by a faulty SSD/M.2 seems unreasonable, as I've tried both an SSD, and a physically separate M.2 device, they both have the same problem. Furthermore, it also seems unreasonable that the stock kali kernel would have this particular I/O issue, especially since I've tried multiple kernels: 4.9, 4.12, 4.13.2, 4.13.10. I've considered that perhaps there are BIOS options that would affect this performance, but I couldn't find anything, regardless, I'm running setup defaults and have tried multiple BIOS versions, including version 3.4, 3.5, and 3.6.
I've checked dmesg for any I/O errors, there are none.
I've also considered that there are multiple I/O schedulers for Linux:
$ cat /sys/block/sda/queue/scheduler
noop deadline [cfq]
The scheduler can be changed by issuing e.g.
echo "noop" > /sys/block/sda/queue/scheduler
See this question for more information: https://stackoverflow.com/questions/1009577/selecting-a-linux-i-o-scheduler
However, I've tried noop, deadline, and CFQ, they don't appear to affect the problem at all.
I also wanted to check the performance of the non-volatile devices:
hdparm -t /dev/sda
/dev/sda:
Timing buffered disk reads: 1112 MB in 3.01 seconds = 369.69 MB/sec
hdparm -t /dev/sda
/dev/sda:
Timing buffered disk reads: 1122 MB in 3.00 seconds = 373.53 MB/sec
The performance is nothing staggering, I should actually have more than this. But even with this type of performance, these issues should be non-existent. I'm not sure if the issue I'm having is related to this poor I/O performance.
When I was running Gentoo on this same system, I got this on the 840 PRO device (hdparm -t
):
510.82 MB/sec
524.05 MB/sec
How should I go about debugging this issue? Is it obvious to anyone what the problem is?
deadline
, which seemed to help a bit, but not much. Then I discovered thatecho 3 | sudo tee /proc/sys/vm/drop_caches
nearly always will unfreeze the blocked processes, sometimes after a bit of delay, and restore performance. I suspect multi-CPU deadlock in the cache structures, but that's just a guess. Don't know if it will help in your case, but it's worth a try.