advance strategy to interrupt a linux application that has "hung" and made X desktop unresponsive

Question

When an application (thunderbird) has a problem, sometimes it goes into an unresponsive state where it has blocked the UI, is blocking login processes (at tty terminals) and also is preventing a restart of the X session using Ctrl-Alt-Backspace.

Normally on a *nix box, sending a signal to the process is enough to give me back enough access to regain control, however in this case everything was borked, even though it was only a single application using a great deal of RAM and swap.

Using Ctrl-Alt-F1 gave me a console and a login prompt, and I could enter the root username, but it wouldn't return the password prompt, so I was stuck.

Ctrl-Alt-Backspace caused a restart of the X session, but it didn't give me a login prompt and a power-cycle was necessary.

Is there some way to bind a keystroke to some sort of low level "interrupt function" (I mean that in the general sense) to suspend the bothersome hung process so I can see what it is trying to do using strace?

I am vaguely aware that some blocked processes can't be suspended until they have returned, but in this case there is no indication that the blocking process is being prevented from accessing any IO resources, it just appears to be doing something dumb.

I was originally thinking of some bash script to ionice and renice any process associated with thunderbird to the lowest priority, but I imagine that if things are as bad as they seem, then the new process would not be able to get on the CPU.

I would like to use the Ctrl-Break key for this, as it isn't used for anything else during desktop sessions on my machine.

Basically, the desktop is (slowly) responding to key and mouse, so there is some scope to get commands run to cause the bad process to suspend, I just don't know what sort of commands are available in this scenario.

Answer 1

My guess is that Thunderbird is consuming a lot of memory. If you can get a shell/console than just kill the process. Typically, thunderbird will hang if you are rebuilding the search index or loading thousands of emails in cache.

You can also set the priority on the linux oom killer in /proc so that the first thing to SIGTERM.

let's first find the process of thunderbird. (use what ever you prefer: pgrep, ps -ef|grep …, etc)

bash$ pidof thunderbird
1439

Now to temporarily stop the process, you just well, stop the process :)

bash$ kill -s SIGSTOP 1439
bash$ strace ... 

Once you are happy, you can continue the process, with well, "continue" signal. :)

bash$ kill -s SIGCONT 1439

ALso, if it is a thread you can look into using, tkill or tgkill but you may have to install that first.

Answer 2

I am don't know if it's going to answer in all of the details of your question but some hints it's going to give you.

when i am try the know what is the root cause of some program to misbehave is to run it from the console (i mean you run the GUI program but you start it from the console) so that you have some indication on what it is doing, hence you could get a clue on how to resolve the issue with thunderbird.

to be more specific to your question, what i am doing when i have situation like you mention, i ssh into the machine (yes i can login into, although i can't even get tty1 with Alt+F4), then with top, or with ps ax | grep thunderbird kill the program and the desktop (including X and every other window that was open) is back to live.

I know that it isn't touch exactly in the right spots you have draw in your question, but i give what i have.

Answer 3

I suspect there are some issues with your analysis. Are you really still running a single core CPU? I suspect that either something more fundamental than a runaway process is occurring (on a single core box) although it is possible that something is causing the system to run very slowly.

Of course that you can't get access to run any diagnostics once the problem has kicked in doesn't help.

I'd go with disabling ACPI/APM (in case there's some contribution from clock scaling) and running procmon to try to capture the event.

You might consider using watchdog in monitoring only mode (-q) to trigger diagnostics.