23

I'm trying to diagnose some random segfaults on a headless server and one thing that seems curious is that they only seem to happen under memory pressure and my swap size will not go above 0.

How can I force my machine to swap to make sure that it is working properly?

orca ~ # free
             total       used       free     shared    buffers     cached
Mem:       1551140    1472392      78748          0     333920    1046368
-/+ buffers/cache:      92104    1459036
Swap:      1060280          0    1060280

orca ~ # swapon -s
Filename                                Type            Size    Used    Priority
/dev/sdb2                               partition       1060280 0       -1
3
  • Are there any messages in kern.log at the time of the segfaults? A message about oom-killer would indicate that your system doesn't have enough virtual memory, which could mean that swap isn't being used. Is this a virtualized server (and what kind)? Aug 29, 2010 at 20:24
  • There are no oom-killer entries in the log just stuff like segfault at 54 ip b7619ba8 sp bf9c3380 error 4 I'm thinking it's a hardware problem which is going to be a pain to track down. This is a physical server with dual Athlon MP 2000+ processors and 1.5GB of RAM. It runs fairly stably but segfaults during compiles.
    – joshperry
    Aug 30, 2010 at 0:45
  • 1
    Well, turned out that the case fans were not plugged in which was causing issues when the server would start doing anything processor intensive causing it to overheat.
    – joshperry
    Aug 30, 2010 at 1:35

2 Answers 2

33

Is this Linux? If so, you could try the following:

# sysctl vm.swappiness=100

(You might want to use sysctl vm.swappiness first to see the default value, on my system it was 10)

And then either use a program(s) that uses lots of RAM or write a small application that just eats up RAM. The following will do that (source: Experiments and fun with the Linux disk cache):

#include <stdlib.h>
#include <stdio.h>
#include <string.h>

#include <unistd.h>


int main(int argc, char** argv) {
    int max = -1;
    int mb = 0;
    int multiplier = 1; // allocate 1 MB every time unit. Increase this to e.g.100 to allocate 100 MB every time unit.
    char* buffer;

    if(argc > 1)
        max = atoi(argv[1]);

    while((buffer=malloc(multiplier * 1024*1024)) != NULL && mb != max) {
        memset(buffer, 1, multiplier * 1024*1024);
        mb++;
        printf("Allocated %d MB\n", multiplier * mb);
        sleep(1); // time unit: 1 second
    }      
    return 0;
}

Coded the memset line to initialise blocks with 1s rather than 0s, because the Linux virtual memory manager may be smart enough not to actually allocate any RAM otherwise.  I added the sleep(1) in order to give you more time to watch the processes as it gobbles up ram and swap. The OOM killer should kill this once you are out of RAM and SWAP to give to the program. You can compile it with

gcc filename.c -o memeater

where filename.c is the file you save the above program in. Then you can run it with ./memeater.

I wouldn't do this on a production machine.

3
  • Thanks, that worked well to gobble up memory and start swapping. I guess my segfaults are caused by something else... probably hardware :/
    – joshperry
    Aug 30, 2010 at 0:40
  • For linux, it might be useful to start by writing to /proc/self/oom_score_adj to ensure that it is the most likely OOM-killer victim... Oct 15, 2017 at 20:33
  • 2
    need to include <unistd.h> for sleep, otherwise it throws a warning warning: implicit declaration of function ‘sleep’; Jan 26, 2018 at 18:22
0

In order to run the tests in this post, you’ll need the following:

For the first test, you’ll want to make sure that reading and writing to the swap partition works under normal circumstances. You can do so by running these commands. Don’t forget to change amount_of_swap to the actual amount of swap that you have. You may also need to increase timeout if your swap is particularly slow or particularly large.

$ amount_of_swap=2G
$ timeout=60
$ systemd-run --property="MemoryHigh=128M" -- \
    stress-ng \
        --timeout "$timeout" \
        --vm 1 \
        --vm-hang 0 \
        --vm-method zero-one \
        --vm-bytes "$amount_of_swap"
Running as unit: run-u7.service
$ # Wait for it to start using swap, then run:
$ free
               total        used        free      shared  buff/cache   available
Mem:          479432      345384       19136        3284      114912      117948
Swap:        2097148     1975096      122052
$ # Make sure that stress-ng exited successfully:
$ unit_name=run-u7.service  # This might be different on your system. See systemd-run’s output.
$ journalctl --boot --unit="$unit_name"
Started /nix/store/fmsawx6292lg2mc96hj5gmql1mk973dz-stress-ng-0.17.01/bin/stress-ng --timeout 60 --vm 1 --vm-hang 0 --vm-method zero-one --vm-bytes 2G.
invoked with '/nix/store/fmsawx6292lg2mc96hj5gmql1mk973dz-stress-ng-0.17.01/bin/stress-ng --timeout 60 --vm 1 --vm-hang 0 --vm-method zero-one --vm-bytes 2G' by user 0 'root'
stress-ng: info:  [2237] setting to a 1 min, 0 secs run per stressor
stress-ng: info:  [2237] dispatching hogs: 1 vm
system: 'jasonyundt' Linux 6.8.1 #1-NixOS SMP PREEMPT_DYNAMIC Fri Mar 15 18:19:29 UTC 2024 x86_64
memory (MB): total 468.20, free 127.03, shared 3.21, buffer 1.59, swap 2048.00, free swap 2046.73
stress-ng: info:  [2237] skipped: 0
stress-ng: info:  [2237] passed: 1: vm (1)
stress-ng: info:  [2237] failed: 0
stress-ng: info:  [2237] metrics untrustworthy: 0
stress-ng: info:  [2237] successful run completed in 1 min, 3.96 secs
run-u7.service: Deactivated successfully.
run-u7.service: Consumed 28.368s CPU time, no IP traffic.

The free command’s output will show you whether or not the swap actually got used.


Most of the time that previous test will be enough. Unfortunately, it’s possible to create swap that can’t be used by the kernel when the kernel is close to running out of memory. Specifically, if there’s less than min_free_kbytes of free memory left, then the kernel will go into a minimum-memory emergency mode where only PF_MEMALLOC allocations are allowed. If writing to your swap device or swap file requires non-PF_MEMALLOC memory allocations, then your system will break if you use too much RAM.

Here’s how you can test whether or not hitting the min_free_kbytes limit will break your system:

#!/usr/bin/env bash
# Again, remember to potentially adjust amount_of_ram and timeout.
amount_of_ram=1G
timeout=60

original_min_free_kbytes="$(sysctl -n vm.min_free_kbytes)"
sudo -v
{
    sleep "$(( timeout / 2 ))"
    free
    sudo sysctl vm.min_free_kbytes="$(( 128 * 1024 ))"
} &
stress-ng \
    --timeout "$timeout" \
    --vm 1 \
    --vm-hang 0 \
    --vm-method zero-one \
    --vm-bytes "$amount_of_ram" &
wait

sudo sysctl vm.min_free_kbytes="$original_min_free_kbytes"

If your system is fine, then that script will exit successfully. If your system needs non-PF_MEMALLOC memory allocations in order to swap, then this will happen:

[ 1106.923468] INFO: task systemd:1 blocked for more than 122 seconds.
[ 1106.924018]       Tainted: G        W          6.8.1 #1-NixOS
[ 1106.924512] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this 
message.
[ 1106.925344] INFO: task kthreadd:2 blocked for more than 122 seconds.
[ 1106.925876]       Tainted: G        W          6.8.1 #1-NixOS
[ 1106.926356] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this 
message.
[ 1106.927188] INFO: task kworker/u2:0:11 blocked for more than 122 seconds.
[ 1106.927757]       Tainted: G        W          6.8.1 #1-NixOS
[ 1106.928234] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this 
message.
[ 1106.929447] INFO: task kworker/u2:1:23 blocked for more than 122 seconds.
[ 1106.930018]       Tainted: G        W          6.8.1 #1-NixOS
[ 1106.930506] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this 
message.
[ 1106.931598] INFO: task kswapd0:37 blocked for more than 122 seconds.
[ 1106.932129]       Tainted: G        W          6.8.1 #1-NixOS
[ 1106.932619] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this 
message.
[ 1106.933396] INFO: task kworker/0:3:139 blocked for more than 122 seconds.
[ 1106.933968]       Tainted: G        W          6.8.1 #1-NixOS
[ 1106.934452] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this 
message.
[ 1106.935430] INFO: task systemd-udevd:425 blocked for more than 122 seconds.
[ 1106.936051]       Tainted: G        W          6.8.1 #1-NixOS
[ 1106.936611] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this 
message.
[ 1106.937482] INFO: task systemd-oomd:578 blocked for more than 122 seconds.
[ 1106.938077]       Tainted: G        W          6.8.1 #1-NixOS
[ 1106.938582] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this 
message.
[ 1106.939438] INFO: task systemd-timesyn:605 blocked for more than 122 seconds.
[ 1106.940063]       Tainted: G        W          6.8.1 #1-NixOS
[ 1106.940572] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this 
message.
[ 1106.941436] INFO: task kworker/0:5:642 blocked for more than 122 seconds.
[ 1106.942028]       Tainted: G        W          6.8.1 #1-NixOS
[ 1106.942539] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this 
message.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .