I'm using Debian testing without any problems for ~6 years (I'm just regularly updating it), but recently it started to show a random behaviour that can be summarized as "Low I/O performance which persists until reboot".
The problem is, suddenly all disk reads and writes slow down to ~5MB/sec which results in continuous read and writes. Since the rate is so low, disks are not mechanically challenged or stressed, but everything slows down until I reboot.
I/O subsystem of the computer consists of one OCZ Vertex 3 SSD and two WD Caviar Black HDDs. SSD holds read-heavy part of the OS and a partition on the HDD holds the rest.
To diagnose the problem I tried the following without success:
topdoesn't show any runaway activity neither in CPU nor I/O usage.
hdparmreturns normal performance ratings of the disks (I only checked
smartctldoesn't show any performance problems in disks. Long tests showed that the disks are as good as new.
System has Z77 Chipset, 16GB of RAM and Intel i7 3770K CPU and the stats show no signs of saturation in RAM, I/O or CPU, but I'm not experienced to debug problems like this (esp. in kernel space). Any help will be appreciated.
- I ran (forced) fsck on every partition as a precaution. All FS are clean.
- Incidentally I found a BIOS upgrade which came out a month ago & applied it.
- No partition is filled more than 50%.
The problem is not surfacing up for two days. Either
fsck or the BIOS update cleaned some clogs in the system. I'm still monitoring the issue and will close the question with a post-mortem answer.
Problem just resurfaced and I did some more digging. Please see the answer.