45

My server program received a SIGTERM and stopped (with exit code 0). I am surprised by this, as I am pretty sure that there was plenty of memory for it. Under what conditions does linux (busybox) send a SIGTERM to a process?

7
  • I can't think of any case when the kernel or a standard tool would send SIGTERM to a random process. What can you tell us about what the program is doing and how it's started? How did you find out about the program's exit status? Can you reproduce the problem? Do you have logs you can check? Commented Mar 28, 2011 at 17:57
  • It is reading and writing to a serial line, and is responding to UDP and TCP requests. I have wrapped the execution on a bash script and therefore I know the exit code. Commented Mar 29, 2011 at 8:14
  • 1
    Posix documentation indicates SIGTERM is strictly a user level event. Is it possible someone else was able to kill your server program?
    – shellter
    Commented Mar 29, 2011 at 22:48
  • 4
    You are! The return code 0 means a normal exit. If there was a SIGTERM, $? would be set to 143 (128 + signal number). Commented Mar 30, 2011 at 18:36
  • 1
    Also, ^C is SIGINT, not SIGTERM, and that would exit with a code of 130.
    – Sparkette
    Commented Sep 1, 2018 at 1:30

3 Answers 3

38

I'll post this as an answer so that there's some kind of resolution if this turns out to be the issue.

An exit status of 0 means a normal exit from a successful program. An exiting program can choose any integer between 0 and 255 as its exit status. Conventionally, programs use small values. Values 126 and above are used by the shell to report special conditions, so it's best to avoid them.

At the C API level, programs report a 16-bit status¹ that encodes both the program's exit status and the signal that killed it, if any.

In the shell, a command's exit status (saved in $?) conflates the actual exit status of the program and the signal value: if a program is killed by a signal, $? is set to a value greater than 128 (with most shells, this value is 128 plus the signal number; ATT ksh uses 256 + signal number and yash uses 384 + signal number, which avoids the ambiguity, but the other shells haven't followed suit).

In particular, if $? is 0, your program exited normally.

Note that this includes the case of a process that receives SIGTERM, but has a signal handler for it, and eventually exits normally (perhaps as an indirect consequence of the SIGTERM signal, perhaps not).


To answer the question in your title, SIGTERM is never sent automatically by the system. There are a few signals that are sent automatically like SIGHUP when a terminal goes away, SIGSEGV/SIGBUS/SIGILL when a process does things it shouldn't be doing, SIGPIPE when it writes to a broken pipe/socket, etc. And there are a few signals that are sent due to a key press in a terminal, mainly SIGINT for Ctrl+C, SIGQUIT for Ctrl+\ and SIGTSTP for Ctrl+Z, but SIGTERM is not one of those. If a process receives SIGTERM, some other process sent that signal.

¹ roughly speaking

8
  • 2
    Nice explanation of how exit status is determined upon receiving a signal. However, this answer doesn't address OP's question. Commented May 23, 2017 at 16:29
  • 1
    @codeforester I answered the question in the body, not the question in the title. Well, one of the questions in the body — given that it was based on a misunderstanding, it is a bit messy. I'll add a few words about the rest. Commented May 23, 2017 at 20:43
  • That depends on the shell. In ksh93, it's 256+signum, in yash, it's 384+signum Commented May 23, 2017 at 21:00
  • 1
    @CharlieParker SIGTERM happens because some process sends it. It is not sent automatically by the kernel. Something on your system is explicitly calling kill(…, SIGTERM). Commented Mar 1, 2021 at 18:22
  • 2
    @CharlieParker It's plausible that a HPC cluster would monitor processes and kill those that it considers to have overused resources. You'd have to look at the cluster's documentation or ask the administrators. Commented Mar 1, 2021 at 18:32
24

SIGTERM is the signal that is typically used to administratively terminate a process.

That's not a signal that the kernel would send, but that's the signal a process would typically send to terminate (gracefully) another process.

That's the signal that is sent by default by the kill, pkill, killall... commands.

That's the signal that is sent to daemons to stop them (like upon a service some-service stop), or sent by init before shutdown (followed by SIGKILL for those processes that have not managed to terminate in time upon SIGTERM).

Note that SIGTERM is not the signal that is sent upon ^C. The signal sent upon ^C is SIGINT.

4
  • but why does SIGTERM happen? how do I debug why my python script is exiting with that? Commented Mar 1, 2021 at 17:25
  • 2
    @CharlieParker, you'd need to determine what process is sending that SIGTERM to the process running the python interpreter that interprets your script. System logs may have some information. If on Linux, you could use auditd/auditctl to log invocations of kill/tkill/tgkill system calls. Commented Mar 1, 2021 at 17:30
  • could it be that the HPC/cluster is killing my processes because I am taking to much memory? (I am trying to run 112 processes in parallel and do machine learning on it). Thanks in advance for the info. I am for sure (me) not doing any killing, it must be something either in python or the cluster would be my guess. Commented Mar 1, 2021 at 18:30
  • 1
    What does 'in time' mean, usually? How much time is there between SIGTERM and SIGKILL on shutdown? Milliseconds? Seconds? Minutes? Commented May 14, 2021 at 21:54
1

In addition to the other answers, SIGTERM is commonly sent to processes running under a job scheduler in an HPC cluster. The scheduler will often send SIGTERM before it kills the process with SIGKILL.

  • Slurm will send SIGTERM before SIGKILL for jobs that run over the time limit.
  • LSF will send SIGINT then SIGTERM before SIGKILL for jobs that run over the time or memory limit.

Depending on the configured time limit, it may be more or less likely for the process to have completed between SIGTERM and SIGKILL. Defaults fall in the range of tens of seconds.

1
  • While we appreciate personal experience, we prefer authoritative references.  Can you link to a document that corroborates this?  Please do not respond in comments; edit your answer to make it more complete. Commented Feb 9 at 23:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .