Linux GUI becomes very unresponsive when doing heavy disk I/O - what to tune?

Question

I have a device /dev/mydisk that is based on a stack of functionality: a LUKS-encrypted, software RAID-1.

From time to time, I do a backup of /dev/mydisk contents to an external USB disk, which is itself encrypted using LUKS. A couple of 100 GiB need to be transferred. This operation is not a simple dd but a recursive cp (I still need to change to use rsync)

A while after the backup starts, the interactivity of the whole system declines tremendously. The KDE interface is choked to death apparently waiting for memory requests to be granted. A wait time of 2 minutes for the prompt is not unusual. Waiting for network I/O likewise demands a lot patience. This is similar behaviour to what happens when baloo kicks in and decides to unzip every zip and index every file content for purposes unknown: The system becomes swamp canoe.

It seems that the kernel gives all the RAM to the copying processes and is loath to hand it back to give interactive processes a chance. RAM is not shabby: 23 GiB. There is also 11 GiB of swap space, just in case, but it's just occupied by a few MiB at any time.

Is it possible to make sure interactive processes get their RAM in preference to the copying processes? If so, how?

Version information:

• This is a Fedora 29 (4.19.15-300.fc29.x86_64) system but I know that I had this issue in earlier Fedora systems, too.
• The KDE version is based on "KDE Frameworks: 5.53.0".

Update

Thanks to everyone for the answers so far!

Once one knows what to search for, one finds some things.

What I have hauled in:

• 2018-10: U&LSE entry apparently exactly about my problem: System lags when doing large R/W operations on external disks. As the questioner uses dd, the remedy is to use the flag oflag=direct to bypass the page cache.
• 2018-11: U&LSE relatively general question about slowdown-on-write Why were “USB-stick stall” problems reported in 2013? Why wasn't this problem solved by the existing “No-I/O dirty throttling” code?. This is rather confusing and and we have to wrestle rumours and phenomena.
  • 2013-11: Jonathan Corbet at LWM.net: The pernicious USB-stick stall problem. This is the article the "reported the problem in 2013". However, an answer to the question of 2018-11 says that this article is wrong and based on incorrect premises.
• 2011-08: U&LSE entry about how to forcefully clear the page cache, which may bring responsiveness back: Setting /proc/sys/vm/drop_caches to clear cache
• 2016-01: U&LSE entry about how to restrict the size of the buffer cache: Restrict size of buffer cache in Linux
• Discussions about I/O schedulers and writeback throttling.
  • 2018-10: U&LSE question on this: Is “writeback throttling” a solution to the “USB-stick stall problem”?
  • 2016-04: Jonathan Corbet at LWM.net: Toward less-annoying background writeback.
• I'm also thinking about: 2017-05: Improving Linux System Performance with I/O Scheduler Tuning, 2009-06: Selecting a Linux I/O Scheduler

Why aren't there expert systems handling the I/O tuning by now..?

Answer 1

I'd nice -n 19 the backup process (it gives low priority to CPU), and maybe also ionice -c 3 (I/O on idle).

rsync will also be a major improvement (it won't copy the 100Gb each time). For instance, my backup scripts look like this:

SOURCE=/whatever/precious/directory
DESTINATION=/media/some_usb_drive/backup
nice -n 19 rsync --verbose --archive --compress --delete --force --recursive --links --safe-links --rsh ssh --exclude-from=$EXCLUDEFILE $SOURCE $DESTINATION
# or
nice -n 19 ionice -c 3 rsync --verbose --archive --compress --delete --force --recursive --links --safe-links --rsh ssh --exclude-from=$EXCLUDEFILE $SOURCE $DESTINATION

(exclude-from is used to avoid the .cache directories, .o files, etc.)

Answer 2

nocache

I searched, and although the documentation does not mention it, nocache should work correctly for writes. Although it will run slower when copying small files, because it requires an fdatasync() call on each file.

(The impact of large numbers of fdatasync() / fsync() calls can be reduced, using Linux-specific features. See note "[1]" about how dpkg works, in this related answer about IO and cache effects. However this would require changing nocache to defer close(), and this could have unwanted side effects in some situations :-(.)

---

An alternative idea is to run your copy process in a cgroup, possibly using systemd-run, and setting a limit on memory consumption. The cgroup memory controller controls cache as well as process memory. I can't find a great example for the systemd-run command (maybe someone will provide one :-).