71

Somehow I happened to swap out 14 GB of memory. After having killed the culprit, I have tons of free memory again, so I thought I could bring in the important data again. So with 5 GB out of 32 GB used and 14 GB of swap space used, I ran swapoff -a.... and 4 hours later about half of the work was finished.

This means less that 1 MB/s, while I can easily copy 200 MB/s. My swap is encrypted but so are all normal partitions and with aes-ni it leads to no noticeable CPU load (and filling the swap space took only a few minutes). I see that there's no special reason to optimize swapoff, however I wonder how it could get that slow?


Just adding some more data: My main memory is 32 GB and I have 32 GB swap space on each of 4 harddisks (surely an overkill, but who cares?). The whole swap space can be (decrypted and) read in less than 5 minutes:

time -p sudo sh -c 'for i in /dev/mapper/cryptswap?; do md5sum $i & done; wait'
014a2b7ef300e11094134785e1d882af  /dev/mapper/cryptswap1
a6d8ef09203c1d8d459109ff93b6627c  /dev/mapper/cryptswap4
05aff81f8d276ddf07cf26619726a405  /dev/mapper/cryptswap3
e7f606449327b9a016e88d46049c0c9a  /dev/mapper/cryptswap2
real 264.27

Reading a part of a partition can't be slower than reading it all. Yet reading about 1/10th of it takes about 100 times longer.

I observed that during swapoff both the CPU was mostly idle (maybe 10% of one core) and so were the disks ("measured" by the LEDs). I also saw that the swap spaces were turned off one after the other.

  • 1
    I wonder, does the same phenomenon occur when system loads swapped pages back into RAM by itself? For example, if I a system was suspended to disk and then starting, everything was swapped out and it's being loaded back to RAM. Seems to be very slow to me too. – Petr Pudlák Aug 21 '12 at 6:11
  • Are all swap-devices activated with the same priority? – Nils Aug 29 '12 at 21:45
  • @Petr Pudlák: Suspend to disk is a bit different, it simply writes the RAM content into a free space in the swap area, and this (and un-suspend) is probably much faster. I can't try as it doesn't work with encrypted swap. – maaartinus Aug 30 '12 at 5:56
  • @Nils: Yes, the priority is the same and so are the the disk and their partitioning. – maaartinus Aug 30 '12 at 5:57
  • That makes it more strange. In this case the swap is striped across all disks - this should be very fast. Did iostat -d 5 show low IO on the disks during swapoff, too? – Nils Aug 30 '12 at 20:59
49

First, let's look at what you can expect from your hard drive. Your hard drive can do 200 MB/s sequentially. When you factor seek times in, it can be much slower. To pick an arbitrary example, take a look at the specs for one of Seagate's modern 3TB disks, the ST3000DM001:

  • Max sustained data rate: 210 MB/s

  • Seek average read: <8.5 ms

  • Bytes per sector: 4,096

If you never need to seek, and if your swap is near the edge of the disk, you can expect to see up to the max rate = 210 MB/s

But if your swap data is entirely fragmented, in the worst case scenario, you'd need to seek around for every sector you read. That means that you only get to read 4 KB every 8.5 ms, or 4 KB / 0.0085 = 470 KB/s

So right off the bat, it's not inconceivable that you are in fact running up against hard drive speeds.


That said, it does seem silly that swapoff would run so slowly and have to read pages out of order, especially if they were written quickly (which implies in-order). But that may just be how the kernel works. Ubuntu bug report #486666 discusses the same problem:

The swap is being removed at speed of 0.5 MB/s, while the
hard drive speed is 60 MB/s;
No other programs are using harddrive a lot, system is not under
high load etc.

Ubuntu 9.10 on quad core.

Swap partition is encrypted.
Top (atop) shows near 100% hard drive usage
  DSK | sdc | busy 88% | read 56 | write 0 | avio 9 ms |
but the device transfer is low (kdesysguard)
  0.4 MiB/s on /dev/sdc reads, and 0 on writes

One of the replies was:

It takes a long time to sort out because it has to rearrange and flush the
memory, as well as go through multiple decrypt cycles, etc. This is quite
normal

The bug report was closed unresolved.

Mel Gorman's book "Understanding the Linux Virtual Memory Manager" is a bit out of date, but agrees that this is a slow operation:

The function responsible for deactivating an area is, predictably enough, called sys_swapoff(). This function is mainly concerned with updating the swap_info_struct. The major task of paging in each paged-out page is the responsibility of try_to_unuse() which is extremely expensive.

There's a bit more discussion from 2007 on the linux-kernel mailing list with the subject "speeding up swapoff" -- although the speeds they're discussing there are a bit higher than what you are seeing.


It's an interesting question that probably gets generally ignored, since swapoff is rarely used. I think that if you really wanted to track it down, the first step would be trying to watch your disk usage patterns more carefully (maybe with atop, iostat, or even more powerful tools like perf or systemtap). Things to look for might be excessive seeking, small I/O operations, constant rewriting and movement of data, etc.

  • 4
    Excellent explanation. It should be noted that it's possible to circumvent most of the fragmentation and free up a majority of swap quickly by core-dumping the large sections of swapped memory: unix.stackexchange.com/questions/254202/… – Brandon DuPree Jan 14 '16 at 5:21
  • It's not just fragmentation/seek time. My swap is on SSD and random reads are very fast, yet the swapoff command is way slower than it should and my SSD load sits at around 1% util. I'm suspecting there's list-walking involved somewhere in the kernel or in swapoff (which uses ~90-100% CPU). Of course if all work is done sequentially and the disk seeks are slow too it can add up significantly. – Thomas Guyot-Sionnest Apr 24 at 17:12
30

I've been experiencing the same problem with my laptop which has a SSD so seeks times shouldn't be a problem.

I found an alternative explanation. Here is an excerpt

The way it works now, swapoff looks at each swapped out memory page in the swap partition, and tries to find all the programs that use it. If it can’t find them right away, it will look at the page tables of every program that’s running to find them. In the worst case, it will check all the page tables for every swapped out page in the partition. That’s right–the same page tables get checked over and over again.

So it is a kernel problem rather than anything else.

  • No, it's not kernel problem IMHO. It's how swapoff is implemented. When swapped out process exits it doesn't take so long. – Marki555 Jan 22 '15 at 14:05
  • 14
    It is a problem with the implementation of swapoff which is in the kernel - hence a kernel problem! You can see if you strace swapoff that pretty much all it does is call the swapoff system call. – Nick Craig-Wood Jan 22 '15 at 15:51
  • 1
    I do have a server with 48GB RAM (32cores), had 6 GB free bug swap was used 0.7GB. swappiness=10, tried to make it 0 and also tried the swapoff in order to see what happens. swapoff takes ages, probably 30minutes, releasing the swap extreamly slow. I do have SSD under almost no load and CPU is similar, expect the swapoff process which takes one cpu 100%. – sorin Jan 23 '15 at 12:25
  • 1
    It's problem of how swapoff is implemented (in kernel). There were discussions about much better approach few years ago in kernel-dev, but they says it's a corner case and don't want the effort to change it. – Marki555 Jan 26 '15 at 23:17
  • 6
    On server with 1 TB RAM (yes, TB) and 2 GB swap (silly SAP requirement) swapoff took 12 hours to free 5% of that 2 GB (with 1 cpu core at 100%). – Marki555 Jan 26 '15 at 23:19
19

You need quite complicated workaround to fix the inefficient swapoff mechanism.

The one-liner

perl -we 'for(`ps -e -o pid,args`) { if(m/^ *(\d+) *(.{0,40})/) { $pid=$1; $desc=$2; if(open F, "/proc/$pid/smaps") { while(<F>) { if(m/^([0-9a-f]+)-([0-9a-f]+) /si){ $start_adr=$1; $end_adr=$2; }  elsif(m/^Swap:\s*(\d\d+) *kB/s){ print "SSIZE=$1_kB\t gdb --batch --pid $pid -ex \"dump memory /dev/null 0x$start_adr 0x$end_adr\"\t2>&1 >/dev/null |grep -v debug\t### $desc \n" }}}}}' | sort -Vr | head

This runs maybe 2 seconds and won't actually do anything, just list the top 10 memory segments (actually it prints more one-liners; yes I do love one-liners; just examine the commands, accept the risk, copy and paste into your shell; these will actually read from swap).

...Paste the generated one-liners...
swapoff /your/swap    # much faster now

Any dangers?

The main one-liner is safe (for me), except it reads a lot of /proc.

The sub-commands prepared for your manual examination are not safe. Each command will hang one process for the duration of reading a memory segment from swap. So it's unsafe with processes that don't tolerate any pauses. The transfer speeds I saw were on the order of 1 gigabyte per minute.

Another danger is putting too much memory pressure on the system, so check with the usual free -m

But what does it do?

for(`ps -e -o pid,args`) {

  if(m/^ *(\d+) *(.{0,40})/) { 
    $pid=$1; 
    $desc=$2; 

    if(open F, "/proc/$pid/smaps") { 

      while(<F>) { 

        if(m/^([0-9a-f]+)-([0-9a-f]+) /si){ 
          $start_adr=$1; 
          $end_adr=$2; 
        } elsif( m/^Swap:\s*(\d\d+) *kB/s ){
          print "SSIZE=$1_kB\t gdb --batch --pid $pid -ex \"dump memory /dev/null 0x$start_adr 0x$end_adr\"\t2>&1 >/dev/null |grep -v debug\t### $desc \n" 
        }
      }
    }
  }
}

The output of this perl script is a series of gdb commands dump memory (range) which recall swapped pages to memory and incidentally make further swapoff blazingly fast (I saw hundreds MB/s). The core idea of using gdb and smaps comes from this answer by jlong.

The output starts with the size, so it's easy enough to pass it trough | sort -Vr | head to get top 10 largest segments by size (SSIZE). The -V stands for version-number-suitable sorting, but it works for my purpose. I couldn't figure how to make numerical sort work.

  • You would use numerical sort here with sort -t = -k 2n – Stéphane Chazelas Nov 24 '16 at 17:02
  • 9
    There doesn't seem to be any need to use gdb to peek the process memory (at least on recent kernels). One can just open /proc/$pid/mem, seek and read directly. Here's PoC largely based on your snippet: gist.github.com/WGH-/91260f6d65db88be2c847053c49be5ae This way process is not stopped, AFAIK there shouldn't be any dangers caused by this. – WGH Oct 20 '17 at 0:13
9

During swapoff, if an in-use swap slot is detected, then the kernel first swaps in the page. The function unuse_process() then tries to find all page table entries that correspond to the just swapped-in page and makes necessary update to the page tables. The search is exhaustive and very time-consuming: it visits every memory descriptor (of the entire system) and examines its page table entries one by one.

Please refer to page 724 of "Understanding the Linux Kernel 3rd version".

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.