19

I'm trying to debug an issue with my shell (specifically, fish) sending signals to my background processes. I'd like to be able to recognize what signals a process is receiving.

Ideally, I'd like some program that would do something like this:

$ log_signals > signals.txt &

Which would then write the signals it received to its output. However, I don't believe any such program readily exists.

What's the simplest way to trap incoming signals and inspect them, preferably without needing to write my own program?

(I'm running OS X, not Linux, so I'd prefer a more platform-agnostic answer if possible.)

0

5 Answers 5

22

Since Stéphane posted this information as a comment, but I found this information to be more useful than any existing answers, I'm reposting the comment as an answer.

For Linux only: strace prints signals by default, so you can use the -e flag to silence all system calls so that signals are clearer:

strace -e 'trace=!all' cmd

In my case, I was trying to figure out what process was killing a daemon I was running, so I got output that looks like this:

[hendrenj@underling02 ~]$ strace -p 171869 -e 'trace=!all'
Process 171869 attached
--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=151513, si_uid=1000} ---
+++ killed by SIGTERM +++

Here you can see that I attached to PID 171869, and that process was sent SIGTERM by the process with PID 151513.

For further reference, you can check out this blog post, which suggests tools such as auditd (which I believe comes with SELinux if you are on a distro that uses it) or stap (System Tap) if strace isn't powerful enough.

3
  • To clarify, it's either -p PID or -f CMD, providing command without -f flag doesn't seem to attach strace to the running command. Apr 11, 2020 at 14:03
  • 1
    @DmitryVerkhoturov providing a command without -f flag executes the command (i.e. starts a new process) with strace attached. On the other hand, -f and -p are for attaching to processes that are already running.
    – jayhendren
    Apr 13, 2020 at 16:18
  • The "this blog post" link is broken. Dec 3, 2020 at 9:35
2

You can use gdb to do this:

gdb --batch -ex 'handle all print' -ex 'handle all nostop' -ex 'handle all pass' -ex 'run' cat will run cat under GDB and print all signals, pass them to the program and not stop execution.

Doing this for a background program is harder, as GDB will try and go into the background instead. You might be better off starting it with cat &, and then attaching the debugger in another window: gdb -ex 'handle all print' -ex 'handle all nostop' -ex 'handle all pass' -ex 'c' (pgrep cat)[1]. Push q[ENTER] three times to get past all the spaff, and then you can watch cat run.

1
  • 1
    This is a good idea, but when I attach GDB to another process, it doesn't seem to catch the signals. The program simply quits, and GDB displays a message like "[Inferior 1 (process 44735) exited normally]". Feb 22, 2015 at 19:20
2

Rather than have some dependency, I realised you can just hook into the trap command to detect any signals. The script below sets up some "traps" on signals (which execute a function when the relevant signal comes in) which then outputs the signal as an exit code. You can change the exit to a console output if you wish.

#!/bin/sh

trap cleanup_1 1 # SIGHUP
trap cleanup_2 2 # SIGINT
trap cleanup_3 3 # SIGQUIT
trap cleanup_6 6 # SIGABRT
trap cleanup_15 15 # SIGTERM

cleanup_1()
{
  exit 1
}

cleanup_2()
{
  exit 2
}

cleanup_3()
{
  exit 3
}

cleanup_6()
{
  exit 6
}

cleanup_15()
{
  exit 15
}

echo "Started"
cat
1

Use killsnoop

killsnoop traces the kill() syscall, to show signals sent via this method. This may be useful to troubleshoot failing applications, where an unknown mechanism is sending signals.

-1
strace -o logfile -e signal=all cmdline
2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.