My server program received a SIGTERM and stopped (with exit code 0). I am surprised by this, as I am pretty sure that there was plenty of memory for it. Under what conditions does linux (busybox) send a SIGTERM to a process?

  • I can't think of any case when the kernel or a standard tool would send SIGTERM to a random process. What can you tell us about what the program is doing and how it's started? How did you find out about the program's exit status? Can you reproduce the problem? Do you have logs you can check? Mar 28 '11 at 17:57
  • It is reading and writing to a serial line, and is responding to UDP and TCP requests. I have wrapped the execution on a bash script and therefore I know the exit code. Mar 29 '11 at 8:14
  • 1
    Posix documentation indicates SIGTERM is strictly a user level event. Is it possible someone else was able to kill your server program?
    – shellter
    Mar 29 '11 at 22:48
  • 4
    You are! The return code 0 means a normal exit. If there was a SIGTERM, $? would be set to 143 (128 + signal number). Mar 30 '11 at 18:36
  • 1
    Also, ^C is SIGINT, not SIGTERM, and that would exit with a code of 130.
    – flarn2006
    Sep 1 '18 at 1:30

I'll post this as an answer so that there's some kind of resolution if this turns out to be the issue.

An exit status of 0 means a normal exit from a successful program. An exiting program can choose any integer between 0 and 255 as its exit status. Conventionally, programs use small values. Values 126 and above are used by the shell to report special conditions, so it's best to avoid them.

At the C API level, programs report a 16-bit status¹ that encodes both the program's exit status and the signal that killed it, if any.

In the shell, a command's exit status (saved in $?) conflates the actual exit status of the program and the signal value: if a program is killed by a signal, $? is set to a value greater than 128 (with most shells, this value is 128 plus the signal number; ATT ksh uses 256 + signal number and yash uses 384 + signal number, which avoids the ambiguity, but the other shells haven't followed suit).

In particular, if $? is 0, your program exited normally.

Note that this includes the case of a process that receives SIGTERM, but has a signal handler for it, and eventually exits normally (perhaps as an indirect consequence of the SIGTERM signal, perhaps not).

To answer the question in your title, SIGTERM is never sent automatically by the system. There are a few signals that are sent automatically like SIGHUP when a terminal goes away, SIGSEGV/SIGBUS/SIGILL when a process does things it shouldn't be doing, SIGPIPE when it writes to a broken pipe/socket, etc. And there are a few signals that are sent due to a key press in a terminal, mainly SIGINT for Ctrl+C, SIGQUIT for Ctrl+\ and SIGTSTP for Ctrl+Z, but SIGTERM is not one of those. If a process receives SIGTERM, some other process sent that signal.

¹ roughly speaking

  • 1
    Nice explanation of how exit status is determined upon receiving a signal. However, this answer doesn't address OP's question. May 23 '17 at 16:29
  • 1
    @codeforester I answered the question in the body, not the question in the title. Well, one of the questions in the body — given that it was based on a misunderstanding, it is a bit messy. I'll add a few words about the rest. May 23 '17 at 20:43
  • That depends on the shell. In ksh93, it's 256+signum, in yash, it's 384+signum May 23 '17 at 21:00
  • 1
    @CharlieParker SIGTERM happens because some process sends it. It is not sent automatically by the kernel. Something on your system is explicitly calling kill(…, SIGTERM). Mar 1 '21 at 18:22
  • 2
    @CharlieParker It's plausible that a HPC cluster would monitor processes and kill those that it considers to have overused resources. You'd have to look at the cluster's documentation or ask the administrators. Mar 1 '21 at 18:32

SIGTERM is the signal that is typically used to administratively terminate a process.

That's not a signal that the kernel would send, but that's the signal a process would typically send to terminate (gracefully) another process.

That's the signal that is sent by default by the kill, pkill, killall... commands.

That's the signal that is sent to daemons to stop them (like upon a service some-service stop), or sent by init before shutdown (followed by SIGKILL for those processes that have not managed to terminate in time upon SIGTERM).

Note that SIGTERM is not the signal that is sent upon ^C. The signal sent upon ^C is SIGINT.

  • but why does SIGTERM happen? how do I debug why my python script is exiting with that? Mar 1 '21 at 17:25
  • 2
    @CharlieParker, you'd need to determine what process is sending that SIGTERM to the process running the python interpreter that interprets your script. System logs may have some information. If on Linux, you could use auditd/auditctl to log invocations of kill/tkill/tgkill system calls. Mar 1 '21 at 17:30
  • could it be that the HPC/cluster is killing my processes because I am taking to much memory? (I am trying to run 112 processes in parallel and do machine learning on it). Thanks in advance for the info. I am for sure (me) not doing any killing, it must be something either in python or the cluster would be my guess. Mar 1 '21 at 18:30
  • 1
    What does 'in time' mean, usually? How much time is there between SIGTERM and SIGKILL on shutdown? Milliseconds? Seconds? Minutes? May 14 '21 at 21:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.