For years, the OOM killer of my operating system doesn't work properly and leads to a frozen system.
When the memory usage is very high, the whole system tends to "freeze" (in fact: becoming extremely slow) for hours or even days, instead of killing processes to free the memory.
The maximum that I have recorded is 7 days before resigning myself to operate a reset.
When OOM is about to be reached, the iowait is very very high (~ 70%), before becoming unmeasurable.
iotop has showed that every programs are reading at a very high throughput (per tens of MB/sec) from my hard drive.
What those programs are reading ?
- The directory hierarchy ?
- The executable code itself ?
I don't exactly now.
[edited] At the time I wrote this message (in 2017) I was using an uptodate ArchLinux (4.9.27-1-lts), but had already experienced the issue for years before.
I have experienced the same issue with various Linux distributions and different hardware configurations.
Currently (2019), I am using an uptodate Debian 9.6 (4.9.0) I have 16 GB of physical ram, a SSD on which my OS is installed, and not any swap partition.
Because of the amount of ram that I have, I don't want to enable a swap partition, since it would just delay the apparition of the issue.
Also, with SSDs swapping too often could potentially reduce the lifespan of the disk.
By the way, I've already tried with and without a swap partition, it has proved to only delay the apparition of the problem, but not being the solution.
To me the problem is caused by the fact that Linux drops essential data from the caches, which leads to a frozen system because it has to read everything, every time from the hard drive.
I even wonder if Linux wouldn't drop the executable code pages of running programs, which would explain why programs that normally don't read a lot of data, behave this way in this situation.
I have tried several things in the hope to fix this issue.
One was to set
1000000 (1 GB).
Because this 1 GB should remain free, I thought that this memory would be reserved by Linux to cache important data.
But it hasn't worked.
Also, I think useful to add that even if it could sound great in theory, restricting the size of the virtual memory to the size of the physical memory, by defining
2 isn't decently technically possible in my situation, because the kind of applications that I use, require more virtual memory than they effectively use for some reasons.
According to the file
Commited_AS value is often higher than the double of the physical ram on my system (16 GB, Commited_AS is often > 32 GB).
I have experienced this problem with
/proc/sys/vm/overcommit_memory to its default value:
0, and for a while I have defined it to:
1, because I preferred programs to be killed by the OOM killer rather than behaving wrongly because they don't check the return values of
malloc when the allocations are refused.
When I was talking about this issue on IRC, I have met other Linux users who have experienced this very same problem, so I guess that a lot of users are concerned by this.
To me this is not acceptable since even Windows deals better with high memory usage.
If you need more information, have a suggestion, please tell me.
They talk about it:
Why does linux out-of-memory (OOM) killer not run automatically, but works upon sysrq-key?
Why does OOM-killer sometimes fail to kill resource hogs?
Preloading the OOM Killer
Is it possible to trigger OOM-killer on forced swapping?
How to avoid high latency near OOM situation?