I'm trying to debug an issue with my shell (specifically, fish) sending signals to my background processes. I'd like to be able to recognize what signals a process is receiving.

Ideally, I'd like some program that would do something like this:

$ log_signals > signals.txt &

Which would then write the signals it received to its output. However, I don't believe any such program readily exists.

What's the simplest way to trap incoming signals and inspect them, preferably without needing to write my own program?

(I'm running OS X, not Linux, so I'd prefer a more platform-agnostic answer if possible.)


Since Stéphane posted this information as a comment, but I found this information to be more useful than any existing answers, I'm reposting the comment as an answer.

For Linux only: strace prints signals by default, so you can use the -e flag to silence all system calls so that signals are clearer:

strace -e 'trace=!all' cmd

In my case, I was trying to figure out what process was killing a daemon I was running, so I got output that looks like this:

[hendrenj@underling02 ~]$ strace -p 171869 -e 'trace=!all'
Process 171869 attached
--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=151513, si_uid=1000} ---
+++ killed by SIGTERM +++

Here you can see that I attached to PID 171869, and that process was sent SIGTERM by the process with PID 151513.

For further reference, you can check out this blog post, which suggests tools such as auditd (which I believe comes with SELinux if you are on a distro that uses it) or stap (System Tap) if strace isn't powerful enough.

  • To clarify, it's either -p PID or -f CMD, providing command without -f flag doesn't seem to attach strace to the running command. – Dmitry Verkhoturov Apr 11 '20 at 14:03
  • 1
    @DmitryVerkhoturov providing a command without -f flag executes the command (i.e. starts a new process) with strace attached. On the other hand, -f and -p are for attaching to processes that are already running. – jayhendren Apr 13 '20 at 16:18
  • The "this blog post" link is broken. – Luke Hutchison Dec 3 '20 at 9:35

You can use gdb to do this:

gdb --batch -ex 'handle all print' -ex 'handle all nostop' -ex 'handle all pass' -ex 'run' cat will run cat under GDB and print all signals, pass them to the program and not stop execution.

Doing this for a background program is harder, as GDB will try and go into the background instead. You might be better off starting it with cat &, and then attaching the debugger in another window: gdb -ex 'handle all print' -ex 'handle all nostop' -ex 'handle all pass' -ex 'c' (pgrep cat)[1]. Push q[ENTER] three times to get past all the spaff, and then you can watch cat run.

  • This is a good idea, but when I attach GDB to another process, it doesn't seem to catch the signals. The program simply quits, and GDB displays a message like "[Inferior 1 (process 44735) exited normally]". – Alexis King Feb 22 '15 at 19:20

Use killsnoop

killsnoop traces the kill() syscall, to show signals sent via this method. This may be useful to troubleshoot failing applications, where an unknown mechanism is sending signals.

strace -o logfile -e signal=all cmdline

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.