20

I know about memory overcommitment and I profoundly dislike it and usually disable it. I am not thinking of setuid-based system processes (like those running sudo or postfix) but of an ordinary Linux process started on some command line by some user not having admin privileges.

A well written program could malloc (or mmap which is often used by malloc) more memory than available and crash when using it. Without memory overcommitment, that malloc or mmap would fail and the well written program would catch that failure. The poorly written program (using malloc without checks against failure) would crash when using the result of a failed malloc.

Of course virtual address space (which gets extended by mmap so by malloc) is not the same as RAM (RAM is a resource managed by the kernel, see this; processes have their virtual address space initialized by execve(2) and extended by mmap & sbrk so don't consume directly RAM, only virtual memory).

Notice that optimizing RAM usage could be done with madvise(2) (which could give a hint, using MADV_DONTNEED to the kernel to swap some pages onto the disk), when really needed. Programs wanting some overcommitment could use mmap(2) with MAP_NORESERVE. My understanding of memory overcommitment is as if every memory mapping (by execve or mmap) is using implicitly MAP_NORESERVE

My perception of it is that it is simply useful for very buggy programs. But IMHO a real developer should always check failure of malloc, mmap and related virtual address space changing functions (e.g. like here). And most free software programs whose source code I have studied have such check, perhaps as some xmalloc function....

Are there real life programs, e.g. packaged in a typical Linux distributions, which actually need and are using memory overcommitment in a sane and useful way? I know none of them!

What are the disadvantages of disabling memory overcommitment? Many older Unixes (e.g. SunOS4, SunOS5 from the previous century) did not have it, and IMHO their malloc (and perhaps even the general full-system performance, malloc-wise) was not much worse (and improvements since then are unrelated to memory overcommitment).

I believe that memory overcommitment is a misfeature for lazy programmers.

The user of that program could setup some resource limit for setrlimit(2) called with RLIMIT_AS by the parent process (e.g. ulimit builtin of /bin/bash; or limit builtin of zsh, or any modern equivalent for e.g. at, crontab, batch, ...), or a grand-parent process (up to eventually /sbin/init of pid 1 or its modern systemd variant).

7
  • 2
    Certain packages for R have trouble on OpenBSD because they want large amounts of virtual memory and OpenBSD says nope. Those same packages are fine on Linux and do not cause the crazy drunk oom killer to start blasting away at the process table.
    – thrig
    May 2, 2018 at 19:15
  • 2
    @thrig that could be an answer, notably if you where more specific. What R packages? Can they still work in practice? Why do they need memory overcommit? May 2, 2018 at 19:17
  • this was 5+ years ago and I could not find the offending package (or something has been fixed, meanwhile...) in a brief search
    – thrig
    May 2, 2018 at 22:48
  • A couple of comments on the edits you have made to the question: without memory overcommitment, a call to malloc would fail, even if there is plenty of memory still available, I think it would be unrealistic to require all programs to use madvise. The kernel usually gets the hint automatically anyway, by keeping a count of which pages have been recenlty used and which have not. The MAP_NORESERVE flag to the mmap system call only means no swap space is reserved for the mapping, it does not disable demand paging. May 3, 2018 at 7:33
  • what kind of "plenty of memory" do you refer to? virtual memory or RAM? IMHO RAM is managed by the kernel, and application code don't care about it. It uses virtual address space. May 3, 2018 at 7:34

5 Answers 5

27

The reason for overcommitting is to avoid underutilization of physical RAM. There is a difference between how much virtual memory a process has allocated and how much of this virtual memory has been actually mapped to physical page frames. In fact, right after a process is started, it reserves very little RAM. This is due to demand paging: the process has a virtual memory layout, but the mapping from the virtual memory address to a physical page frame isn't established until the memory is read or written.

A program typically never uses its whole virtual memory space, and the memory areas touched varies during the run of the program. For example, mappings to page frames containing initialization code that is executed only at the start of the run can be discarded and the page frames can be used for other mappings.

The same applies to data: when a program calls malloc, it reserves a sufficiently large contiguous virtual address space for storing data. However, mappings to physical page frames are not established until the pages are actually used, if ever. Or consider the program stack: every process gets a fairly big contiguous virtual memory area set aside for the stack (typically 8 MB). A process typically uses only a fraction of this stack space; small and well-behaving programs use even less.

A Linux computer typically has a lot of heterogeneous processes running in different stages of their lifetimes. Statistically, at any point in time, they do not collectively need a mapping for every virtual page they have been assigned (or will be assigned later in the program run).

A strictly non-overcommitting scheme would create a static mapping from virtual address pages to physical RAM page frames at the moment the virtual pages are allocated. This would result in a system that can run far fewer programs concurrently, because a lot of RAM page frames would be reserved for nothing.

I don't deny that overcommitting memory has its dangers, and can lead to out-of-memory situations that are messy to deal with. It's all about finding the right compromise.

6
  • Avoiding underutilization of RAM might be done with madvise(2) using MADV_DONTNEED (and perhaps should be done so by applications), so I am still not very convinced. But your explanation is interesting, so thanks! May 3, 2018 at 5:10
  • And the usual scheme would be to reserve virtual memory (by allocating it on swap area), not RAM May 3, 2018 at 5:23
  • 2
    Processes have a virtual address space (in virtual memory). RAM is managed by the kernel, whole system wise. So be careful in expliciting virtual address space, virtual memory, RAM May 3, 2018 at 8:13
  • 1
    Even without overcommitment it may be possible to use some of that reserved RAM for cache.
    – gmatht
    May 31, 2020 at 5:05
  • 1
    A strictly non-overcommitting scheme would create a static mapping from virtual address pages to physical RAM page frames at the moment the virtual pages are allocated. This is only valid when no on disk swap area is set. Solaris is a strictly non over-committing OS but doesn't waste RAM at all if the swap area is large enough to back all reservations.
    – jlliagre
    Jun 1, 2020 at 16:37
7

You say this as if laziness is not considered a virtue in programming :).

Large quantities of software are optimized for simplicity and maintainability, with surviving low-memory conditions as a very low priority. It is common to treat allocation failure as fatal. Exiting the process which exhausts memory avoids a situation where there is no free memory, and the system cannot make progress without either allocating more memory, or complexity in the form of comprehensive pre-allocation.

Notice how fine the difference is between checking allocations and dying, or not checking and crashing. It would not be fair to blame overcommit on programmers simply not bothering to check whether malloc() succeeded or failed.

There is a only a small amount of software which you can trust to continue "correctly" in the face of failed allocations. The kernel should generally be expected to survive. sqlite has a notoriously robust test which includes out of memory testing, specifically because it is intended to support various small embedded systems.

As a failure path not used in normal operation, handling low memory conditions correctly imposes a significant extra burden in maintenance and testing. If that effort does not yield a commensurate benefit, it can more profitably be spent elsewhere.

Where this breaks down, it is also common to have special cases for large allocations, to handle the most common causes of failure.

Allowing a certain amount of overcommit is probably best viewed in this context. It is part of the current default compromise on Linux.

Note the idea that one should disable kernel-level overcommit and instead provide more swap than you ever want to use, also has its haters. The gap in speed between RAM and rotating hard drives has grown over time, such that when the system actually uses the swap space you have allowed it, it can more often be described as "grinding to a halt".

2
  • Laziness could justify to not bother free-ing most heap mallocated memory and leave the operating system kernel to properly clean the process after its termination Aug 28, 2019 at 10:32
  • 2
    @BasileStarynkevitch absolutely true, and for programs that only run for a brief period and then exit, doing that is acceptable (although not preferable). For programs that are expected to run indefinitely, OTOH, that won't work for obvious reasons. May 29, 2020 at 0:01
5

I agree with and upvoted Johan Myréen answer but here are more explanations that might help you understanding the issue.

You seem to confuse the swap area, i.e. on disk space intended to store less used RAM and virtual memory. The latter is made of a combination of RAM areas and on disk areas.

Processes are reserving and using virtual memory. They have no idea about where it is stored. When they need to access some data which isn't in RAM, processes (or threads) are suspended until the kernel do the job for the data page to be available.

When there is RAM demand, the kernel free some RAM by storing less used process pages on the disk swap area.

When a process reserves memory (i.e. malloc and the likes), non overcommiting OSes mark unused portions of the virtual memory as unavailable. That means that when the process having made the allocation will actually need to access the reserved pages, they will be guaranteed to be present.

The drawback is given the fact that memory is not usable by any other process is preventing these processes to have their pages paginated out, so RAM is wasted if these processes are inactive. Worst, if the sum of reservations is larger than the size of the swap area, pages of RAM will also be reserved to match the reservation despite not containing any data. This is quite a bad scenario because you'll have RAM which is both unusable and unused. Finally, the worst case scenario is for a huge reservation not to be able to be accepted because there is no more virtual memory (swap + ram) available for it. The process doing the reservation will usually crash.

On the other hand, overcommiting OSes like Linux bet there will be no virtual memory shortage at any given time. They accept most memory reservations (but not unrealistic ones, this can be more or less tuned) and in general, this allows a better utilization of the RAM and swap resources.

This is similar to airline companies overbooking seats. This improve occupancy rate but some passengers might be unhappy. Hopefully, airlines just book them to another flight and possibly compensate them while Linux just throws the heavier passengers out of the flying plane...

To summarize, Linux reserves memory a lazy, "best effort" way while several other OSes do guarantee reservations.

Real cases scenario where overcommitting makes a lot of sense is when a program which uses a lot of virtual memory does a fork followed by an exec.

Let's say you have 4 GB of RAM from which 3 GB are available for virtual memory and 4 GB of swap. There is a process reserving 4 GB but only using 1 GB out of it. There is no pagination so the system performs well. On a non overcommiting OS, that process cannot fork because just after the fork, 4 GB more of virtual memory need to be reserved, and there is only 3 GB left.

On Linux, this fork (or clone) system call will complete successfully (but cheating under the cover) and after the following exec (if any), these reserved but unused 4 GB will be freed without any harm.

6
  • Would it be practical to design a Unix system so that the only allocations without pre-committed storage would be blocks tentatively duplicated by fork, and add a system call that would either commit all such blocks or report that it is unable to do so (allowing a program a chance to attempt a graceful shut down if possible).
    – supercat
    Jun 1, 2020 at 15:43
  • @supercat That wouldn't worth the effort. Adding a system call for such a corner case has zero chance to be accepted by Unix/Linux kernel maintainers.
    – jlliagre
    Jun 1, 2020 at 16:32
  • The main goal would be ensuring that the only program that could fail due to overcommit would be a spawned process that makes use of the parent process' address space. I would think that a call to implicitly touch all the copy-on-write pages wouldn't be particularly difficult to implement, but I don't know how things work internally.
    – supercat
    Jun 1, 2020 at 16:34
  • Implementing this would have an impact on performance. The fact pages belong to the parent space doesn't imply they are mapped in RAM. Touching unmapped pages would lead to useless I/Os.
    – jlliagre
    Jun 1, 2020 at 17:08
  • There may be some better way of ensuring that pages have storage committed to them, but the main point to provide a means by which applications that need some guarantees as to when failures could or could not occur would be able to achieve the required semantics without having to block all forms of over-commitment.
    – supercat
    Jun 1, 2020 at 17:16
2

Sparse arrays

man proc for /proc/sys/vm/overcommit_memory actually cites an application:

In mode 1, the kernel pretends there is always enough memory, until memory actually runs out. One use case for this mode is scientific computing applications that employ large sparse arrays.

I'm not sure if this is actually used in practice though.

More info on /proc/sys/vm/overcommit_memory at: https://stackoverflow.com/questions/2798330/maximum-memory-which-malloc-can-allocate/57687432#57687432

1

Killing processes can be desirable

Other answers discuss how over-commitment is more efficient. Additionally, sometimes the killing processes is the right thing to do.

First consider the code while(1){malloc(1)}. With overcommit this will eventually be killed by the OOM killer. Without overcommit it will gobble up all memory available bringing the system to its knees.

One solution without overcommit would be to limit amount of memory using cgroups. However, that leaves the challenge of picking sensible defaults. Sometimes there aren't any sensible defaults. Some scientific tasks are as hard as solving the halting problem, so you don't know how much memory and time they need until they finish. You maximise their chance of success by letting scientific users allocate every byte of available memory safe in the knowledge that the OOM killer will take them down if a more vital process needs their memory.

1
  • Somehow true, but the process (or its indirect parents, including /sbin/init of pid 1) could call setrlimit(2). This might be as simple as some ulimit or limit command of your unix shell. If you run a scientific code for months of elapsed time without any limits and without checkpoints, you deserve what you risk. May 31, 2020 at 5:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .