10

I reckon it's not such an uncommon problem: one process allocates massive amounts of memory (be it due to a memory-leak bug, because you try to process an infeasibly large input file, or whatever). The RAM fills up, and at some point Linux has to switch over to swap. Well, sometimes this is just the last resort: if I have an expensive computation going, I do not want to lose data if towards the end I run out of RAM.

Rather more often however (in my experience), the memory consumption is unbounded, by a rogue, perhaps buggy process. I.e., I do not just end up with some less urgently needed data moved to swap, but the OS is forced to panically swap loads of data. And that unfortunately does not just heavily break the offending process, but can bring the whole system to almost a standstill (it's not quite as bad anymore on machines with SSD, but OTOH it makes me worry whether writing gigabytes and gigabytes of garbage data may in the long term harm the flash cells).
Until I notice the problem and manually kill the process (once it actually took minutes until I even got myself logged into a virtual terminal!), half my running session is in swap, and I need to wait quite a while until the system runs smooth again.

There is one draconic solution to the problem: enforce a hard memory limit. But doing this system-wide would sometimes kill processes that I rather need still, and if I have to manually ulimit before starting an offending process... well, I'll often forget until it's too late.

Possible kinds of solution I'd be happier with:

  • If any process exceeds a certain memory usage, it's artificially throttled down so the rest of the system stays responsive.
  • If any process exceeds a certain memory usage, it's SIGSTOPped so I have time to figure out what to do next.
  • If a process approaches the RAM limit, I get a warning, before the great swapping starts.

Is there any way to get such a behaviour, or similar?

6

niceload --noswap yourprg is made for exactly that situation: It looks at swapping activity:

  • If swapping out: Let process run
  • If swapping in: Let process run
  • If swapping in and out: Suspend process until swapping stops and resume the process when swapping has stopped

It does not suspend the process before the swapping starts, but lets swapping run for 1 second before acting.

niceload --mem 1G yourprg works similar: If less than 1GB is free youprg is suspended. When more then 1GB is free yourprg is resumed.

0

Yes. It is quite easily done with practically any modern shell.

sh -c 'ulimit -S -m "$1"
       shift; exec your_command "$@"
' -- some_size args

You can use the -l option for locked memory limits. Your process will be signalled if the limit is exceeded.

-1

Cronjob to clear the cache: How to clear memory cache in Linux

I have actually similar problems. I have a bunch of users who runs their own custom scripts and every once in a while their scripts consumes all available memory and brings the redhat server down. The reason for the mass consumption of RAM was that their scripts can run for days just waiting on a event, thereby hogging resources when in reality it is not using any. So what I did was just simply force clear the cache with a cronjob and haven't had a problem since.

Simple and lazy.

  • Hm. Interesting suggestion, though I'm afraid it does nothing for most of the problems I'm experiencing. My main issue is with single processes (scientific applications) which can rapidly allocate many gigabytes of memory. – leftaroundabout Nov 25 '15 at 22:19
  • Clearing caches usually introduce performance problems. Very occasionally they solve performance problems. But they won't cause memory to become available when it isn't there. – Gilles Nov 25 '15 at 23:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.