25

I have a dev server on which sshd sometimes stops working because the machine runs out of RAM. Yes we are running low on memory and upgrading is not a feasible choice right now. What I want to do is to tell the machine: "Do what ever you want, but leave out 20MB and some CPU for sshd!".

How can that be done?

6
  • Sshd should not be blowing up like that, it's normally a very well behaving daemon. Make sure it's updated to the latest/greatest. What are you doing with it? Are you piping massive data through it?
    – Marcin
    Jun 7, 2011 at 12:16
  • 2
    @Marcin I don't think I'm doing anything big, text editing. Even if sshd is well behaved, I think it would stuck if there is not enough CPU/RAM to use, wouldn't it?
    – phunehehe
    Jun 7, 2011 at 15:08
  • When linux runs out of memory, it starts killing off processes to reclaim memory. You might be seeing OOM in action (out of memory killer). I don't know why would it decide to kill your sshd consistently though. Have you ever seen sshd actually leak (grow in size) horribly, before it blows up, or does it work fine one moment and is dead the next?
    – Marcin
    Jun 7, 2011 at 21:30
  • @Marcin chances are if he's OOM the system is bogged down so much he sshd just can't respond, it's not an sshd problem but an issue with his load. Unfortunately I don't know the answer to this. Jun 8, 2011 at 12:59
  • @phunehehe are you sure that the problem is no memory for ssh and not your load is through the roof? does sshd give any errors? what is the load on the box? and what is causing the system to run out of memory. Jun 8, 2011 at 13:03

4 Answers 4

9

You can probably achieve something like that by using cgroups with the Memory resource controller.

I guess you'd put all your resource-consuming tasks in a limited (CPU & RAM) cgroup, and leave sshd "outside" so that it isn't restricted.

(Adding more swap, even in the form of a swap file, might be a good option though.)

3
  • I'm still hesitant, it doesn't look easy (spoiled by Ubuntu).
    – phunehehe
    Jun 7, 2011 at 15:43
  • oh it's certainly not easy :-) Adding swap is by far easier.
    – Mat
    Jun 7, 2011 at 15:44
  • @phunehehe welcome to being a Sys Admin, it's not easy. But now you have to learn how to do not easy stuff. Jun 8, 2011 at 13:00
6

Oh but cgroups are easy :) Install the libcgroup package. Create a /etc/cgconfig.conf:

mount {
    cpu     = /cgroup/cpu_and_mem;
    cpuacct = /cgroup/cpu_and_mem;
    memory  = /cgroup/cpu_and_mem;
}

group sshd {
        cpu {
                cpu.shares="500";
        }
        cpuacct {
                cpuacct.usage="0";
        }
        memory {
                memory.limit_in_bytes="1G";
        }
}

group nosshd {
        cpu {
                cpu.shares="500";
        }
        cpuacct {
                cpuacct.usage="0";
        }
        memory {
                memory.limit_in_bytes="1G";
        }
}

Start the cgconfig process which will create the hierarchy, cgroups, and set the limits. If that succeeds, you have two cgroups, both of which have 50% of the CPU assigned and 1G of memory available (don't know what your actual amount of available memory is; assuming it's 2G in this example). Now you just need to move all the tasks (ie all processes running on the system) from the root group into the nosshd cgroup:

cgroup]# cat tasks >> nosshd/tasks
cgroup]# echo > tasks

Then you just need to get the PID of the sshd process and move it info the sshd tasks file:

cgroup]# echo $PID >> sshd/tasks

Ta-da, you're done. You can now rest assured that sshd will always have 50% of the CPU and 1G of memory.

1

Use renice to get a higher priority for sshd, or check accounting. (acct) -> with this you can set resources for users, so run sshd with s

1
  • I would add ionice to that too, and probably start sshd with nice not renicing it later. Jun 8, 2011 at 13:03
1

A more general solution to the problem of application resource usage is to run your applications in a container using Docker. You can then run containers with CPU and memory usage limits similar to cgroups.

docker run -c=10 -m=1g my-container

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .