5

I have a server with Intel(R) Atom(TM) CPU D525 and 1 GB memory. I noticed the server would shut down and restart automatically about every 7 days.

I checked the memory usage and found that when the memory usage reached 90%, the kernel reboots. When I checked the kernel log in /var/log/messages file, I didn't find anything about the kernel shutting down, just a message about the kernel start. I checked the file /proc/sys/vm/min_free_kbytes, the value is "3765".


I guess when the available memory is very low, but doesn't get to the number the system starts reclaiming memory. Then the kernel can't do anything, so it then reboots.

Can you give me some insight?

  • Very poorly worded question. – poige Aug 22 '13 at 5:02
  • means the number the system starts reclaiming memory with "...,but don't get to the value kernel recover,..." – jofox Aug 22 '13 at 5:25
  • $ man swapon - swapon, swapoff - enable/disable devices and files for paging and swapping – innocent-world Aug 22 '13 at 11:57
  • It is perfectly normal for buffers/cache to consume nearly all of memory. Your problem is unrelated to memory. – psusi Sep 5 '13 at 14:43
17

On some demand-paged virtual memory systems, the operating system refuses to allocate anonymous pages (i.e. pages containing data without a filesystem source such as runtime data, program stack etc.) unless there is sufficient swap space to swap out the pages in order to free up physical memory. This strict accounting has the advantage that each process is guaranteed access to as much virtual memory they allocate, but is also means that the amount of virtual memory available is essentially limited by the size of the swap space.

In practice, programs tend to allocate more memory than they use. For instance, the Java Virtual Machine allocates a lot of virtual memory on startup, but does not use it immediately. Memory accounting in the Linux kernel attempts to compensate for this by tracking the amount of memory actually in use by processes, and overcommits the amount of virtual memory. In other words the amount of virtual memory allocated by the kernel can exceed the amount of physical memory and swap space combined on the system. While this leads to better utilization of physical memory and swap space, the downside is that when the amount of of memory in use exceeds the amount of physical memory and swap space available, the kernel must somehow free memory resources in order to meet the memory allocation commitment.

The kernel mechanism that is used to reclaim memory to fill the overcommitment is called the out-of-memory-killer (OOM-killer). Typically the mechanism will start killing off memory-hogging "rogue" processes to free up memory for other processes. However, if the vm.panic_on_oom sysctl setting is non-zero, the kernel will panic instead when the system runs out of memory.

The possible values for the vm.panic_on_oom setting are as follows:

  • 0 (default) When an out-of-memory situation arises, the OOM-killer will kill a rogue process.

  • 1 The kernel normally panics, but if process that has reached its memory allocation limit set with mbind(MPOL_BIND) or cpuset, the process is killed instead.

  • 2 The kernel always panics in an out-of-memory situation.

The heuristic used by the OOM-killer can be modified through the vm.oom_kill_allocating_task sysctl setting. The possible values are as follows:

  • 0 (default) The OOM-killer will scan through the task list and select a task rogue task utilizing a lot of memory to kill.

  • 1 (non-zero) The OOM-killer will kill the task that triggered the out-of-memory condition.

The kernel memory accounting algorithm can be tuned with the vm.overcommit_memory sysctl settings. The possible values are as follows:

  • 0 (default) Heuristic overcommit with weak checks.

  • 1 Always overcommit, no checks.

  • 2 Strict accounting, in this mode the virtual address space limit is determined by the value of vm.overcommit_ratio settings according to the following formula:

    virtual memory = (swap + physical memory * (overcommit_ratio / 100))
    

When strict memory accounting is in use, the kernel will no longer allocate anonymous pages unless it has enough free physical memory or swap space to store the pages. This means it is essential that the system is configured with enough swap space.

The sysctl settings can be checked or modified at runtime with the sysctl command. To make changes permanent the settings can be written to /etc/sysctl.conf. The above settings are also available via the /proc/sys/vm interface. The corresponding files are:

  • /proc/sys/vm/panic_on_oom

  • /proc/sys/vm/oom_kill_allocating_task

  • /proc/sys/vm/overcommit_memory

  • /proc/sys/vm/overcommit_ratio

  • Thank you very much! But the sever just shut down and started, I find nothing about oom in /var/log/messages file. I find the buffer size is growing and growing, never go down! – jofox Aug 23 '13 at 2:16
  • 1
    @jofox Have you considered the possibility of faulty memory? RAM issues can be hard to diagnose, but a memory diagnostics tool such as Memtest86 can prove useful. – Thomas Nyman Aug 23 '13 at 4:37
  • I have tested. The physical memory is ok. – jofox Aug 23 '13 at 5:14
1
  • I think that you should find out the process consumed lot of memory.Maybe the program has a memory leak bug.You should fix it.
  • If there is no enough memory,Linux will shutdown some process consumed lot of memory,or reboot itself.Because kernel must use some memory.
  • The different behavior is determined by config.But I think that you should fix the memory leak bug.
  • "The different behavior is determined by config", what config do you mean? – jofox Aug 22 '13 at 5:20
  • I do not remember. – Edward Shen Aug 22 '13 at 6:06
  • I'm sure that there is not the memory leak bug. Why does Linux reboot itself but not reclaim memory when there is no enouth memory? – jofox Aug 22 '13 at 6:08
0

I'm guessing that some (kernel?)watchdog detects a system hang and that's what's causing the reboots. Kernel .config options like CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC can cause a reboot after a certain timeout of detecting the system hung.

The real question is: why does the system hang? which I believe is the same as this question: OOM killer doesn't work properly, leads to a frozen OS

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.