0

I ran a job that exceeded the memory, and began to write into the swap. The swap is 25 GB, and the RAM is 1TB.

I have stopped the job when the swap was full until 11GB, so 50% of it was still empty.

It did not trigger any OOM kill jobs, so everything is going as normal as before it happened.

I have only 8GB swap used now (3GB is cleaned after I have stopped the job), and it is slowly decreasing. But when I check vmstat, the si and so are both 0, so there is nothing going in or out of swap? How can this be possible?

free -lm
              total        used        free      shared  buff/cache   available
Mem:        1031757      475637       49100          63      507019      553720
Low:        1031757      982657       49100
High:             0           0           0
Swap:         25767        8272       17495

I have a free space of 40GB in RAM, so should I expect that my running jobs will be killed by OOM not now but later on? 8GB used in swap is smaller than 40GB free space in RAM, so it looks fine, but I am not sure if still tells that OOM won't be triggered..

Couple months ago, when the OOM was triggered, it killed all my jobs and used swap was 25GB (all full) and then 10 minutes later 0GB. However in this case, it takes a day to clean 2-3 GB from swap. Is it bad news for my running jobs?

Do you think my running jobs are in danger of being killed, not now but later on, when free swapspace reaches 0 GB as it would trigger OOM to kill? If that is the case, how can I prevent it from happening?

I would appreciate any help.

1

vmstat without parameters shows the average values since reboot. As the swap in / swap out is shown as blocks/second, it's not strange that those are showing 0, if you have a reasonable uptime.

Whatever is now in swap is memory that is in use, but not used since the time it was swapped out due to the overloading of the memory by your process. This is actually a good thing, as many processes have memory that is e.g. only used during startup. Having this swapped out means that you have more free RAM available for processes and buffers/cache.

The reason that in the earlier case, when you did get OOM, afterwards all swapspace was free again shortly after the event is probably because the OOM-causing process had used all the space and after it was stopped all the swapspace was free again.

The only time when you will get OOM is when there is no free swap space, and there is no RAM free (taking into account the buffers/cache, i.e. the "available" column of the free command).

For the rest you can usually trust Linux's memory management to do the right thing. Only if you have really special requirements due to the sort of load / application you're running can it be necessary to start tuning the memory management.

  • I mean when swap reaches 0GB in use space ( so it will be all free ) – bapors Jan 30 '18 at 14:31
  • OK, I've edited your question a bit to make it a bit clearer. And edited my answer to explain the free swap in the OOM case. – wurtel Jan 30 '18 at 15:16
  • thank you very much for your reply. However, if I got into the swap directly, wouldnt some jobs be flagged to be killed by OMM already? So that I will have a problem when the swap is all free? – bapors Jan 31 '18 at 12:37
  • 1
    Only if there is no free memory and no free swap will the OOM killer be invoked. Jobs aren't "flagged" to be killed, the OOM killer, when started, looks for the best candidate to kill, which is usually the job that has caused the OOM condition. – wurtel Jan 31 '18 at 13:40
  • I understand..Thank you very much for your informative replies. They helped me a lot. – bapors Feb 3 '18 at 19:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.