Somehow I happened to swap out 14 GB of memory. After having killed the culprit, I have tons of free memory again, so I thought I could bring in the important data again. So with 5 GB out of 32 GB used and 14 GB of swap space used, I ran
swapoff -a.... and 4 hours later about half of the work was finished.
This means less that 1 MB/s, while I can easily copy 200 MB/s. My swap is encrypted but so are all normal partitions and with aes-ni it leads to no noticeable CPU load (and filling the swap space took only a few minutes). I see that there's no special reason to optimize
swapoff, however I wonder how it could get that slow?
Just adding some more data: My main memory is 32 GB and I have 32 GB swap space on each of 4 harddisks (surely an overkill, but who cares?). The whole swap space can be (decrypted and) read in less than 5 minutes:
time -p sudo sh -c 'for i in /dev/mapper/cryptswap?; do md5sum $i & done; wait' 014a2b7ef300e11094134785e1d882af /dev/mapper/cryptswap1 a6d8ef09203c1d8d459109ff93b6627c /dev/mapper/cryptswap4 05aff81f8d276ddf07cf26619726a405 /dev/mapper/cryptswap3 e7f606449327b9a016e88d46049c0c9a /dev/mapper/cryptswap2 real 264.27
Reading a part of a partition can't be slower than reading it all. Yet reading about 1/10th of it takes about 100 times longer.
I observed that during
swapoff both the CPU was mostly idle (maybe 10% of one core) and so were the disks ("measured" by the LEDs). I also saw that the swap spaces were turned off one after the other.