Let's look at some of the things in your question:
As soon as I resume the output, the system begins slowing down, and within a minute it is again almost unusable.
This gives a hint that something is backing up over a period of time and eventually can't sustain its throughput. Since you said that this is on a different disk than everything else, it should be something system-global.
I have 32GB (!!!) of RAM, of which no more than a third is being used.
By "used", I assume you mean "not RSS". Unfortunately it's not quite that simple -- while page caches are more likely to be easily freeable than RSS, that doesn't mean that they are free -- as an example, they could be dirty and require flushing back to disk first -- and this question is likely an illustration. :-)
This is usually a question of a.) the I/O scheduler in use, and b.) the type of I/O being performed. In your case it's probably a significant amount of page cache writebacks, which generally can't trivially be throttled by the kernel once they are started. Even though these writes may be on a different disk, you still have a single source of shared state in the form of the page cache.
I/O scheduling classes, in the sense of ionice, only have any effect on the CFQ I/O scheduler and are a noop on others. However, CFQ has a significant number of tradeoffs that lean towards "fairness" instead of "latency", which can result in situations like this.
CFQ is based on a per-TID model where each thread gets its own queue. These queues are then processed round robin by the kernel, which pops some amount of items off of each queue, actions them, and goes on about its merry way -- this guaranteed actioning of each process queue is the "fair" part of CFQ. Fairness does not necessarily equate to performance, however, since this means each process generally receives the same prioritisation (excepting adjustments like ionice).
Deadline, by comparison, is based on imposing a latency timeout on each I/O request, as the name suggests. Instead of being focused on fairness at the TID level, it is focused on a number of other issues -- mostly avoiding request starvation (through variable expiration per operation type) and through treating the system as a unit rather than each process as a unit of operation for "fairness".
I strongly suggest trying the following things:
- Set the I/O scheduler to mq-deadline. Deadline in general does a better job making sure reads don't become starved than CFQ, which can avoid these kinds of multi-second stalls when something eventually accesses the disk in question. In the case of desktop usage, you're mostly doing reads when you expect things to be responsive, so this makes sense.
- Consider using cgroup v2's
io.latency, which I go over a bit in this talk. This is systemwide, not per-device, and allows you to set much finer-grained controls on I/O protection and prioritisation than you can do with CFQ's ionice. You can then run your desktop in a cgroup which demands low latency I/O, and use systemd-run or the like to run your data recovery without such protections in another cgroup. This also allows us to somewhat dial back these "unstoppable" writes, like page cache writebacks, by getting ahead of them before they are in-flight.
- There are two types of memory reclaim in the kernel: direct reclaim, and kswapd reclaim. kswapd reclaim is where we try to avoid the system memory usage (including caches!) getting to 100%. This avoids us going into the next stage of reclaim, which is called direct reclaim. Direct reclaim is what happens when an application requests memory and there simply isn't enough left to fulfil its request. This then results in actually suspending the affected application, which can cause the kind of stalls you're describing. If you see a lot of direct reclaim during these periods (
grep allocstall /proc/vmstat shows these), it might be worth testing if making the boundary for kswapd reclaim lower improves things. You can do this using the vm.watermark_scale_factor sysctl -- see here for the documentation on how to use it.
