4

I'm recovering data from a hard disk. The disk is in good working order; a collection of files were deleted by accident. I'm using photorec, with the location to store recovered files on another physical disk. My OS and swap are on separate devices from these.

However, running photorec makes my system essentially unusable. It takes 5-10 seconds just to alt-tab (or otherwise switch) to another window, the mouse is laggy and choppy, and programs are generally unresponsive. I keep having to suspend the output (Ctrl-S, photorec is running inside Konsole) to get anything done. As soon as I resume the output, the system begins slowing down, and within a minute it is again almost unusable.

I have set the IO scheduling class to idle and the nice value to 19 (), and this still keeps happening. I have 32GB (!!!) of RAM, of which no more than a third is being used. I have a quad-core Xeon with HT, and nothing using the CPU (CPU usage is nearly flatlined at idle when photorec is suspended). Both disks show sustained transfer rates of more than 100MB/s, so it shouldn't be a problem with hardware performance. Yet the system is less responsive than my old Windows 98 box under heavy CPU+IO load.

Why on earth does this happen, and how to remedy this?

The OS is Debian 10 (buster), running the stock 4.19 kernel.

The system is perfectly usable even if I have 10 threads/processes running flat out (encoding video or compiling large projects is no problem), what I'm experiencing is not ordinary slowdown due to load.

6

Let's look at some of the things in your question:

As soon as I resume the output, the system begins slowing down, and within a minute it is again almost unusable.

This gives a hint that something is backing up over a period of time and eventually can't sustain its throughput. Since you said that this is on a different disk than everything else, it should be something system-global.

I have 32GB (!!!) of RAM, of which no more than a third is being used.

By "used", I assume you mean "not RSS". Unfortunately it's not quite that simple -- while page caches are more likely to be easily freeable than RSS, that doesn't mean that they are free -- as an example, they could be dirty and require flushing back to disk first -- and this question is likely an illustration. :-)

This is usually a question of a.) the I/O scheduler in use, and b.) the type of I/O being performed. In your case it's probably a significant amount of page cache writebacks, which generally can't trivially be throttled by the kernel once they are started. Even though these writes may be on a different disk, you still have a single source of shared state in the form of the page cache.

I/O scheduling classes, in the sense of ionice, only have any effect on the CFQ I/O scheduler and are a noop on others. However, CFQ has a significant number of tradeoffs that lean towards "fairness" instead of "latency", which can result in situations like this.

CFQ is based on a per-TID model where each thread gets its own queue. These queues are then processed round robin by the kernel, which pops some amount of items off of each queue, actions them, and goes on about its merry way -- this guaranteed actioning of each process queue is the "fair" part of CFQ. Fairness does not necessarily equate to performance, however, since this means each process generally receives the same prioritisation (excepting adjustments like ionice).

Deadline, by comparison, is based on imposing a latency timeout on each I/O request, as the name suggests. Instead of being focused on fairness at the TID level, it is focused on a number of other issues -- mostly avoiding request starvation (through variable expiration per operation type) and through treating the system as a unit rather than each process as a unit of operation for "fairness".

I strongly suggest trying the following things:

  1. Set the I/O scheduler to mq-deadline. Deadline in general does a better job making sure reads don't become starved than CFQ, which can avoid these kinds of multi-second stalls when something eventually accesses the disk in question. In the case of desktop usage, you're mostly doing reads when you expect things to be responsive, so this makes sense.
  2. Consider using cgroup v2's io.latency, which I go over a bit in this talk. This is systemwide, not per-device, and allows you to set much finer-grained controls on I/O protection and prioritisation than you can do with CFQ's ionice. You can then run your desktop in a cgroup which demands low latency I/O, and use systemd-run or the like to run your data recovery without such protections in another cgroup. This also allows us to somewhat dial back these "unstoppable" writes, like page cache writebacks, by getting ahead of them before they are in-flight.
  3. There are two types of memory reclaim in the kernel: direct reclaim, and kswapd reclaim. kswapd reclaim is where we try to avoid the system memory usage (including caches!) getting to 100%. This avoids us going into the next stage of reclaim, which is called direct reclaim. Direct reclaim is what happens when an application requests memory and there simply isn't enough left to fulfil its request. This then results in actually suspending the affected application, which can cause the kind of stalls you're describing. If you see a lot of direct reclaim during these periods (grep allocstall /proc/vmstat shows these), it might be worth testing if making the boundary for kswapd reclaim lower improves things. You can do this using the vm.watermark_scale_factor sysctl -- see here for the documentation on how to use it.
  • Turns out that I already had mq-deadline enabled (probably explains why ionice did nothing for me) - apparently that's the default on Debian now (in fact, mq-deadline and noop appear to be the only options). Allocstall stats at the moment (photorec having finished overnight) are 554 for normal and 999 for movable, 0 for both dma and dma32 - which I gather are not particularly high. That leaves me to play with (2) - I'll definitely try this just out of curiosity if nothing else. – jaymmer - Reinstate Monica Nov 16 at 3:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.