Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have used GNU/Linux on systems from 4 MB RAM to 512 GB RAM. When they start swapping, most of the time you can still log in and kill off the offending process - you just have to be 100-1000 times more patient.

On my new 32 GB system that has changed: It blocks when it starts swapping. Sometimes with full disk activity but other times with no disk activity.

To examine what might be the issue I have written this program. The idea is:

1 grab 3% of the memory free right now
2 if that caused swap to increase: stop
3 keep the chunk used for 30 seconds by forking off
4 goto 1

-

#!/usr/bin/perl

sub freekb {
    my $free = `free|grep buffers/cache`;
    my @a=split / +/,$free;
    return $a[3];
}

sub swapkb {
    my $swap = `free|grep Swap:`;
    my @a=split / +/,$swap;
    return $a[2];
}


my $swap = swapkb();
my $lastswap = $swap;
my $free;
while($lastswap >= $swap) {
    print "$swap $free";
    $lastswap = $swap;
    $swap = swapkb();
    $free = freekb();
    my $used_mem = "x"x(1024 * $free * 0.03);
    if(not fork()) {
    sleep 30;
    exit();
    }
}
print "Swap increased $swap $lastswap\n";

Running the program forever ought to keep the system at the limit of swapping, but only grabbing a minimal amount of swap and do that very slowly (i.e. a few MB at a time at most).

If I run:

forever free | stdbuf -o0 timestamp > freelog

I ought to see swap slowly rising every second. (forever and timestamp from https://github.com/ole-tange/tangetools).

But that is not the behaviour I see: I see swap increasing in jumps and that the system is completely blocked during these jumps. Here the system is blocked for 30 seconds with the swap usage increases with 1 GB:

secs
169.527 Swap:     18440184     154184   18286000
170.531 Swap:     18440184     154184   18286000
200.630 Swap:     18440184    1134240   17305944
210.259 Swap:     18440184    1076228   17363956

Blocked: 21 secs. Swap increase 2400 MB:

307.773 Swap:     18440184     581324   17858860
308.799 Swap:     18440184     597676   17842508
330.103 Swap:     18440184    2503020   15937164
331.106 Swap:     18440184    2502936   15937248

Blocked: 20 secs. Swap increase 2200 MB:

751.283 Swap:     18440184     885288   17554896
752.286 Swap:     18440184     911676   17528508
772.331 Swap:     18440184    3193532   15246652
773.333 Swap:     18440184    1404540   17035644

Blocked: 37 secs. Swap increase 2400 MB:

904.068 Swap:     18440184     613108   17827076
905.072 Swap:     18440184     610368   17829816
942.424 Swap:     18440184    3014668   15425516
942.610 Swap:     18440184    2073580   16366604

This is bad enough, but what is even worse is that the system sometimes stops responding at all - even if I wait for hours. I have the feeling it is related to the swapping issue, but I cannot tell for sure.

My first idea was to tweak /proc/sys/vm/swappiness from 60 to 0 or 100, just to see if that had any effect at all. 0 did not have an effect, but 100 did cause the problem to arise less often.

How can I prevent the system from blocking for such a long time?

Why does it decide to swapout 1-3 GB when less than 10 MB would suffice?

System info:

$ uname -a
Linux aspire 3.8.0-32-generic #47-Ubuntu SMP Tue Oct 1 22:35:23 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

Edit:

I tested if the problem is due to 32 GB RAM by removing 24 GB and trying with only 8 GB - I see the same behaviour.

I can also reproduce the swapping behaviour (though not the freezing) by installing GNU/Linux Mint 15 in VirtualBox.

I cannot reproduce the problem on my 8 GB laptop: The script above runs beautifully for hours and hours - swapping out a few megabytes, but never a full gigabyte. So I compared all the variables in /proc/sys/vm/* on both systems: They are exactly the same. This leads me to believe the problem is elsewhere. The laptop runs a different kernel:

Linux hk 3.2.0-55-generic #85-Ubuntu SMP Wed Oct 2 12:29:27 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

Maybe something in the VM system changed from 3.2.0 to 3.8.0?

share|improve this question
    
What's the load and disk IO? –  josten Nov 12 '13 at 22:43
    
0. The system does nothing else. –  Ole Tange Nov 13 '13 at 12:18
add comment

migrated from serverfault.com Nov 13 '13 at 17:27

This question came from our site for professional system and network administrators.

1 Answer

Can't say I've used this myself, but you could try sysctl -w vm.swappiness=100, maybe also tinkering with the "dirty" settings, they look pretty good for this.

share|improve this answer
    
The sysctl is the same as changing /proc/sys/vm/swappiness which has already been tried (see question). –  Ole Tange Nov 14 '13 at 2:30
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.