We run a small server (Ubuntu 14.04) solely for computing purposes. From time to time, a user will manage to consume enough memory to freeze the system. Last time, the culprit was a process that spawned 30 memory eating sub-processes. The result is that I cannot log into the machine to fix it - ssh and local login both just time out. The OOM Killer did not seem to do anything. egrep -i 'killed process' /var/log/* returned nothing.

Is there a way to keep/gain command line access in such circumstances?


There is a way of limiting the use of system resources.
Check the ulimit command and it's usage. It has a conf file limits.conf where you can specify which groups can have how many resources. For example, if you specify in the conf file :

@developers        soft    nproc          20
@developers        hard    nproc          30

then the developers group can have only 30 processes and they will get a warning when reaching 20. You can also limit the number of processes globally with ulimit -u 10 - in this case users can run 10 processes.

Use ulimit -a to display current limits. To limit memory, try to use ulimit -v.

user@localhost:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 256646
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 32768
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 32768
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

You can do things to try and prevent this from happening, as mazs mentions in his answer. But to have the box recover from any situation, even the ones you don't expect, you want a watchdog.

In Linux, a watchdog is a process which sends a "ping" periodically, and if that "ping" isn't received, the system resets. When I say "ping", I don't mean this as a network ICMP echo request, but just a "hey I'm still alive" message. The ping can be sent to a physical watchdog, which might do something like power cycle the host to perform the reset, but it can also be something within the linux kernel. The latter is what you'll most likely want as hardware watchdog devices are typically only found in enterprise grade equipment.

Anyway, to get started, you first need the software watchdog enabled in your kernel. This is likely to be the biggest obstacle. I have no idea if ubuntu has the watchdog enabled or not. See if you have /dev/watchdog available. If not, try modprobe softdog. If neither works, and you still want to pursue this, you're going to need to recompile your kernel with the SOFT_WATCHDOG option.

Now, assuming you have the watchdog enabled, you need to install the watchdog package.

Once it's installed, put a script in /etc/watchdog.d that you want to use to perform your health check (can also use test-binary in watchdog.conf), and make it executable. If you wanted to make sure ssh is functional, you could do something like:

case "$1" in
  ssh testuser@localhost /bin/true
  service ssh restart

(To do exactly as above, you'll need to create testuser and set up public key auth, but thats outside the scope of this answer)

The watchdog will invoke the script with test to do the health check, and will try repair if it fails. If the repair also fails, the system is reset.

See the watchdog documentation for further details.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.