3

I have a device /dev/mydisk that is based on a stack of functionality: a LUKS-encrypted, software RAID-1.

enter image description here

From time to time, I do a backup of /dev/mydisk contents to an external USB disk, which is itself encrypted using LUKS. A couple of 100 GiB need to be transferred. This operation is not a simple dd but a recursive cp (I still need to change to use rsync)

A while after the backup starts, the interactivity of the whole system declines tremendously. The KDE interface is choked to death apparently waiting for memory requests to be granted. A wait time of 2 minutes for the prompt is not unusual. Waiting for network I/O likewise demands a lot patience. This is similar behaviour to what happens when baloo kicks in and decides to unzip every zip and index every file content for purposes unknown: The system becomes swamp canoe.

It seems that the kernel gives all the RAM to the copying processes and is loath to hand it back to give interactive processes a chance. RAM is not shabby: 23 GiB. There is also 11 GiB of swap space, just in case, but it's just occupied by a few MiB at any time.

Is it possible to make sure interactive processes get their RAM in preference to the copying processes? If so, how?

Version information:

  • This is a Fedora 29 (4.19.15-300.fc29.x86_64) system but I know that I had this issue in earlier Fedora systems, too.
  • The KDE version is based on "KDE Frameworks: 5.53.0".

Update

Thanks to everyone for the answers so far!

Once one knows what to search for, one finds some things.

What I have hauled in:

Why aren't there expert systems handling the I/O tuning by now..?

  • For reference, please specify the version either of KDE or the OS you are using. gnome-shell currently has an issue where it calls fsync() on the main thread, which can hang the entire GUI for tens of seconds. Obviously it would be nice if fsync() didn't do this, but gnome-shell should not be doing it in the first place, and it may be fixed in some later versions (and some parts of the code are already deliberately avoiding it). gitlab.gnome.org/GNOME/gnome-shell/issues/815 . So IMO it would be useful to note the version of KDE you are using here. – sourcejedi Jan 21 at 10:52
  • Thanks @sourcejedi Added info. KDE has the "stop the world" phenomenon also even where the system is basically idle but it needs to do some internal configuration. But that is not likely related to the described slowdown. – David Tonhofer Jan 21 at 11:03
  • 1
    I noticed similar symptoms on my system, and dropping caches with echo 3 > /proc/sys/vm/drop_caches restores interactivity (and improves disk transfer rate). Your problem may or may not be related. I haven't yet found out the reason for this behaviour. – dirkt Jan 21 at 13:38
  • @dirkt 3 drops both buffer cache and reclaimable dentries and inodes. If you have a lot of dentries and inodes cached, dropping them will increase the limit for dirty buffer cache. unix.stackexchange.com/a/480999/29483 That's one possible mechanism. – sourcejedi Jan 21 at 15:28
  • In case you're puzzled why the link talks about restricting the "buffer cache" - it is actually restricting the page cache. buffer and page cache were two different mechanisms in older UNIX-like OS's, however they are now unified. It is implemented as a page cache, but it is still very common to refer to it as a buffer cache. unix.stackexchange.com/a/440563/29483 – sourcejedi Feb 24 at 9:49
1

I'd nice -n 19 the backup process (it gives low priority to CPU), and maybe also ionice -c 3 (I/O on idle).

rsync will also be a major improvement (it won't copy the 100Gb each time). For instance, my backup scripts look like this:

SOURCE=/whatever/precious/directory
DESTINATION=/media/some_usb_drive/backup
nice -n 19 rsync --verbose --archive --compress --delete --force --recursive --links --safe-links --rsh ssh --exclude-from=$EXCLUDEFILE $SOURCE $DESTINATION
# or
nice -n 19 ionice -c 3 rsync --verbose --archive --compress --delete --force --recursive --links --safe-links --rsh ssh --exclude-from=$EXCLUDEFILE $SOURCE $DESTINATION

(exclude-from is used to avoid the .cache directories, .o files, etc.)

1

nocache

I searched, and although the documentation does not mention it, nocache should work correctly for writes. Although it will run slower when copying small files, because it requires an fdatasync() call on each file.

(The impact of large numbers of fdatasync() / fsync() calls can be reduced, using Linux-specific features. See note "[1]" about how dpkg works, in this related answer about IO and cache effects. However this would require changing nocache to defer close(), and this could have unwanted side effects in some situations :-(.)


An alternative idea is to run your copy process in a cgroup, possibly using systemd-run, and setting a limit on memory consumption. The cgroup memory controller controls cache as well as process memory. I can't find a great example for the systemd-run command (maybe someone will provide one :-).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.