Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Somehow I happened to swap out 14 GB of memory. After having killed the culprit, I have tons of free memory again, so I thought I could bring in the important data again. So with 5 GB out of 32 GB used and 14 GB of swap space used, I ran swapoff -a.... and 4 hours later about half of the work was finished.

This means less that 1 MB/s, while I can easily copy 200 MB/s. My swap is encrypted but so are all normal partitions and with aes-ni it leads to no noticeable CPU load (and filling the swap space took only a few minutes). I see that there's no special reason to optimize swapoff, however I wonder how it could get that slow?


Just adding some more data: My main memory is 32 GB and I have 32 GB swap space on each of 4 harddisks (surely an overkill, but who cares?). The whole swap space can be (decrypted and) read in less than 5 minutes:

time -p sudo sh -c 'for i in /dev/mapper/cryptswap?; do md5sum $i & done; wait'
014a2b7ef300e11094134785e1d882af  /dev/mapper/cryptswap1
a6d8ef09203c1d8d459109ff93b6627c  /dev/mapper/cryptswap4
05aff81f8d276ddf07cf26619726a405  /dev/mapper/cryptswap3
e7f606449327b9a016e88d46049c0c9a  /dev/mapper/cryptswap2
real 264.27

Reading a part of a partition can't be slower than reading it all. Yet reading about 1/10th of it takes about 100 times longer.

I observed that during swapoff both the CPU was mostly idle (maybe 10% of one core) and so were the disks ("measured" by the LEDs). I also saw that the swap spaces were turned off one after the other.

share|improve this question
    
I wonder, does the same phenomenon occur when system loads swapped pages back into RAM by itself? For example, if I a system was suspended to disk and then starting, everything was swapped out and it's being loaded back to RAM. Seems to be very slow to me too. –  Petr Pudlák Aug 21 '12 at 6:11
    
Are all swap-devices activated with the same priority? –  Nils Aug 29 '12 at 21:45
    
@Petr Pudlák: Suspend to disk is a bit different, it simply writes the RAM content into a free space in the swap area, and this (and un-suspend) is probably much faster. I can't try as it doesn't work with encrypted swap. –  maaartinus Aug 30 '12 at 5:56
    
@Nils: Yes, the priority is the same and so are the the disk and their partitioning. –  maaartinus Aug 30 '12 at 5:57
    
That makes it more strange. In this case the swap is striped across all disks - this should be very fast. Did iostat -d 5 show low IO on the disks during swapoff, too? –  Nils Aug 30 '12 at 20:59
add comment

2 Answers 2

First, let's look at what you can expect from your hard drive. Your hard drive can do 200 MB/s sequentially. When you factor seek times in, it can be much slower. To pick an arbitrary example, take a look at the specs for one of Seagate's modern 3TB disks, the ST3000DM001:

  • Max sustained data rate: 210 MB/s

  • Seek average read: <8.5 ms

  • Bytes per sector: 4,096

If you never need to seek, and if your swap is near the edge of the disk, you can expect to see up to the max rate = 210 MB/s

But if your swap data is entirely fragmented, in the worst case scenario, you'd need to seek around for every sector you read. That means that you only get to read 4 KB every 8.5 ms, or 4 KB / 0.0085 = 470 KB/s

So right off the bat, it's not inconceivable that you are in fact running up against hard drive speeds.


That said, it does seem silly that swapoff would run so slowly and have to read pages out of order, especially if they were written quickly (which implies in-order). But that may just be how the kernel works. Ubuntu bug report #486666 discusses the same problem:

The swap is being removed at speed of 0.5 MB/s, while the
hard drive speed is 60 MB/s;
No other programs are using harddrive a lot, system is not under
high load etc.

Ubuntu 9.10 on quad core.

Swap partition is encrypted.
Top (atop) shows near 100% hard drive usage
  DSK | sdc | busy 88% | read 56 | write 0 | avio 9 ms |
but the device transfer is low (kdesysguard)
  0.4 MiB/s on /dev/sdc reads, and 0 on writes

One of the replies was:

It takes a long time to sort out because it has to rearrange and flush the
memory, as well as go through multiple decrypt cycles, etc. This is quite
normal

The bug report was closed unresolved.

Mel Gorman's book "Understanding the Linux Virtual Memory Manager" is a bit out of date, but agrees that this is a slow operation:

The function responsible for deactivating an area is, predictably enough, called sys_swapoff(). This function is mainly concerned with updating the swap_info_struct. The major task of paging in each paged-out page is the responsibility of try_to_unuse() which is extremely expensive.

There's a bit more discussion from 2007 on the linux-kernel mailing list with the subject "speeding up swapoff" -- although the speeds they're discussing there are a bit higher than what you are seeing.


It's an interesting question that probably gets generally ignored, since swapoff is rarely used. I think that if you really wanted to track it down, the first step would be trying to watch your disk usage patterns more carefully (maybe with atop, iostat, or even more powerful tools like perf or systemtap). Things to look for might be excessive seeking, small I/O operations, constant rewriting and movement of data, etc.

share|improve this answer
add comment

I am seeing extremely slow performance with swapoff. In my case the CPU is pegged at 100% and I see almost zero disk activity.

iostat averages reads of 400-500K/second and writes of 8-120K/second. The hard drive light barely blinks. The hard drive is a reasonably fast 7200RPM hard drive.

In my case I enabled a new swap partition on another LVM on a fast unused hard drive before running swapoff. About 7GB out of 16GB of swap space is in use on the old swap partition which is also the root filesystem with all the home data also there (single partition).

Kernel version is 3.4.11.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.