Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
Related to the SuperUser question at https://superuser.com/questions/200387/linux-overcommit-memory my question is what was the reason why they made to allow overcommit the default?
Since 2.5.30 the values are: 0 (default): as before: guess about how much overcommitment is reasonable,
Big part of the need to overcommit memory in Linux (and Unix systems overall) come from the need to implement the fork() system call, which duplicates the calling process' address space.
Most often, this system call is followed by exec(), the combination of which results in spawning a separate program as a child of the current process, in which case most of the duplicated address space will end up not being used.
In order to make that efficient, Linux uses copy-on-write to avoid duplicating the memory of the application calling fork(), in which case it can avoid having to copy all the pages, just to discard them shortly after exec() is called.
But at the time fork() is called, there's no way to tell whether an exec() is coming. It's quite possible that this is being used to spawn worker children and that reusing the address space of the parent is what's desired. (This technique was quite popular in daemons using pre-forked workers to handle connections.) In which case, most or at least some of the memory requirements will exist for a forked child (perhaps not 100% of the parent's memory, but one could assume most of it.)
But always reserving memory for that case is troublesome for the fork() + exec() case, especially if the parent is a long-running process that reserves multiple gigabytes of memory and forks many children. If it wasn't for overcommit, you'd have to reserve an amount totalling the many gigabytes used by the parent, and that for each forked child. But none (or almost none) of that would be really used, since exec() would flush that reservation right away. The end result is that such a workload would either require a huge amount of swap space (to cope with the reservations, most of it would be unused, but would need to be there for worst case) or something like overcommit.
While fork() is a great illustration of this example, other APIs in Linux/Unix also lead to the need to overcommit. For instance, when malloc() is called (or more precisely the syscalls implementing it), no memory is actually allocated until it is "touched" by the process, so it's perfectly valid to allocate a very large block of gigabytes and use that sparsely so that only a few megabytes are actually used. The fact that these APIs work that way means programs exploit those properties, meaning they would most likely break in absence of overcommit (unless you really have a lot of memory or swap to waste by backing these reservations.)
An interesting discussion on the issue with fork() can be found on this LWN posting about an article from Microsoft Research. The article itself is interesting, of course. But you can see how the comments go right away into overcommit and the problems with it.
The article is named A fork() in the road.
I think reason is not all people have enough resources to buy memory, so in such cases all application should run properly, it help to run application with low resources.
