How can swapoff be that slow?

Question

Somehow I happened to swap out 14 GB of memory. After having killed the culprit, I have tons of free memory again, so I thought I could bring in the important data again. So with 5 GB out of 32 GB used and 14 GB of swap space used, I ran swapoff -a.... and 4 hours later about half of the work was finished.

This means less that 1 MB/s, while I can easily copy 200 MB/s. My swap is encrypted but so are all normal partitions and with aes-ni it leads to no noticeable CPU load (and filling the swap space took only a few minutes). I see that there's no special reason to optimize swapoff, however I wonder how it could get that slow?

---

Just adding some more data: My main memory is 32 GB and I have 32 GB swap space on each of 4 harddisks (surely an overkill, but who cares?). The whole swap space can be (decrypted and) read in less than 5 minutes:

time -p sudo sh -c 'for i in /dev/mapper/cryptswap?; do md5sum $i & done; wait'
014a2b7ef300e11094134785e1d882af  /dev/mapper/cryptswap1
a6d8ef09203c1d8d459109ff93b6627c  /dev/mapper/cryptswap4
05aff81f8d276ddf07cf26619726a405  /dev/mapper/cryptswap3
e7f606449327b9a016e88d46049c0c9a  /dev/mapper/cryptswap2
real 264.27

Reading a part of a partition can't be slower than reading it all. Yet reading about 1/10th of it takes about 100 times longer.

I observed that during swapoff both the CPU was mostly idle (maybe 10% of one core) and so were the disks ("measured" by the LEDs). I also saw that the swap spaces were turned off one after the other.

Answer 1

First, let's look at what you can expect from your hard drive. Your hard drive can do 200 MB/s sequentially. When you factor seek times in, it can be much slower. To pick an arbitrary example, take a look at the specs for one of Seagate's modern 3TB disks, the ST3000DM001:

• Max sustained data rate: 210 MB/s

• Seek average read: <8.5 ms

• Bytes per sector: 4,096

If you never need to seek, and if your swap is near the edge of the disk, you can expect to see up to the max rate = 210 MB/s

But if your swap data is entirely fragmented, in the worst case scenario, you'd need to seek around for every sector you read. That means that you only get to read 4 KB every 8.5 ms, or 4 KB / 0.0085 = 470 KB/s

So right off the bat, it's not inconceivable that you are in fact running up against hard drive speeds.

---

That said, it does seem silly that swapoff would run so slowly and have to read pages out of order, especially if they were written quickly (which implies in-order). But that may just be how the kernel works. Ubuntu bug report #486666 discusses the same problem:

The swap is being removed at speed of 0.5 MB/s, while the
hard drive speed is 60 MB/s;
No other programs are using harddrive a lot, system is not under
high load etc.

Ubuntu 9.10 on quad core.

Swap partition is encrypted.
Top (atop) shows near 100% hard drive usage
  DSK | sdc | busy 88% | read 56 | write 0 | avio 9 ms |
but the device transfer is low (kdesysguard)
  0.4 MiB/s on /dev/sdc reads, and 0 on writes

One of the replies was:

It takes a long time to sort out because it has to rearrange and flush the
memory, as well as go through multiple decrypt cycles, etc. This is quite
normal

The bug report was closed unresolved.

Mel Gorman's book "Understanding the Linux Virtual Memory Manager" is a bit out of date, but agrees that this is a slow operation:

The function responsible for deactivating an area is, predictably enough, called sys_swapoff(). This function is mainly concerned with updating the swap_info_struct. The major task of paging in each paged-out page is the responsibility of try_to_unuse() which is extremely expensive.

There's a bit more discussion from 2007 on the linux-kernel mailing list with the subject "speeding up swapoff" -- although the speeds they're discussing there are a bit higher than what you are seeing.

---

It's an interesting question that probably gets generally ignored, since swapoff is rarely used. I think that if you really wanted to track it down, the first step would be trying to watch your disk usage patterns more carefully (maybe with atop, iostat, or even more powerful tools like perf or systemtap). Things to look for might be excessive seeking, small I/O operations, constant rewriting and movement of data, etc.

Answer 2

I've been experiencing the same problem with my laptop which has a SSD so seeks times shouldn't be a problem.

I found an alternative explanation. Here is an excerpt

The way it works now, swapoff looks at each swapped out memory page in the swap partition, and tries to find all the programs that use it. If it can’t find them right away, it will look at the page tables of every program that’s running to find them. In the worst case, it will check all the page tables for every swapped out page in the partition. That’s right–the same page tables get checked over and over again.

So it is a kernel problem rather than anything else.

Answer 3

You need quite complicated workaround to fix the inefficient swapoff mechanism.

The one-liner

perl -we 'for(`ps -e -o pid,args`) { if(m/^ *(\d+) *(.{0,40})/) { $pid=$1; $desc=$2; if(open F, "/proc/$pid/smaps") { while(<F>) { if(m/^([0-9a-f]+)-([0-9a-f]+) /si){ $start_adr=$1; $end_adr=$2; }  elsif(m/^Swap:\s*(\d\d+) *kB/s){ print "SSIZE=$1_kB\t gdb --batch --pid $pid -ex \"dump memory /dev/null 0x$start_adr 0x$end_adr\"\t2>&1 >/dev/null |grep -v debug\t### $desc \n" }}}}}' | sort -Vr | head

This runs maybe 2 seconds and won't actually do anything, just list the top 10 memory segments (actually it prints more one-liners; yes I do love one-liners; just examine the commands, accept the risk, copy and paste into your shell; these will actually read from swap).

...Paste the generated one-liners...
swapoff /your/swap    # much faster now

Any dangers?

The main one-liner is safe (for me), except it reads a lot of /proc.

The sub-commands prepared for your manual examination are not safe. Each command will hang one process for the duration of reading a memory segment from swap. So it's unsafe with processes that don't tolerate any pauses. The transfer speeds I saw were on the order of 1 gigabyte per minute.

Another danger is putting too much memory pressure on the system, so check with the usual free -m

But what does it do?

for(`ps -e -o pid,args`) {

  if(m/^ *(\d+) *(.{0,40})/) { 
    $pid=$1; 
    $desc=$2; 

    if(open F, "/proc/$pid/smaps") { 

      while(<F>) { 

        if(m/^([0-9a-f]+)-([0-9a-f]+) /si){ 
          $start_adr=$1; 
          $end_adr=$2; 
        } elsif( m/^Swap:\s*(\d\d+) *kB/s ){
          print "SSIZE=$1_kB\t gdb --batch --pid $pid -ex \"dump memory /dev/null 0x$start_adr 0x$end_adr\"\t2>&1 >/dev/null |grep -v debug\t### $desc \n" 
        }
      }
    }
  }
}

The output of this perl script is a series of gdb commands dump memory (range) which recall swapped pages to memory and incidentally make further swapoff blazingly fast (I saw hundreds MB/s). The core idea of using gdb and smaps comes from this answer by jlong.

The output starts with the size, so it's easy enough to pass it trough | sort -Vr | head to get top 10 largest segments by size (SSIZE). The -V stands for version-number-suitable sorting, but it works for my purpose. I couldn't figure how to make numerical sort work.

Answer 4

During swapoff, if an in-use swap slot is detected, then the kernel first swaps in the page. The function unuse_process() then tries to find all page table entries that correspond to the just swapped-in page and makes necessary update to the page tables. The search is exhaustive and very time-consuming: it visits every memory descriptor (of the entire system) and examines its page table entries one by one.

Please refer to page 724 of "Understanding the Linux Kernel 3rd version".