PID of process causing error

Question

I have this script:

#!/bin/bash
# -> dump.$pid
ulimit -c unlimited
# trap ERR
set -o errexit
# also trap error in the middle of a pipe (1)
# otherwise it will only trap the error on (2)
set -o pipefail

trap 'echo "ERR $?"' ERR

echo "a"
echo "1" | ./crash | cat
echo "b"
echo "2" | ./crash
echo "c"

Which works as intended:

$ ./script.sh >& script.log
$ echo "$?"
139
$ cat script.log
a
Segmentation fault (core dumped)
ERR 139

(without pipefail it would print "a b", without errexit it would print "a b c")

However, since this is part of a larger pipeline and run on a batch system, a number of crashes might occur and the argv information in the dump.????? file is not enough.

So: How do I print the PID (and therefore the expected core dump filename) of the process that caused the error from my script, so that it can be logged with all the rest?

The crashing programs may be part of larger pipelines (especially they may not always be at the end of a pipeline, therefore the use of pipefail), and I'd like to avoid having to surround each invocation with wrapper code, or not being able to pipe data directly between programs.

Answer 1

It turns out enabling job control does the trick — but only if there's no SIGSEGV trap, setting that prevents the verbose output.

set -o errexit
set -o pipefail
set -o monitor
trap 'echo "ERR $?"' ERR
echo "hi" | ./docrash | cat
echo "not reached"

Running this gives exactly the output I was looking for, and more. It prints the PID for each child and shows the error:

$ ./err.sh >& err.log
$ cat err.log
./err.sh: line 21: 25110 Done                               echo "hi"
                   25111 Segmentation fault (core dumped) | ./docrash
                   25112                                  | cat
ERR  139
$ gdb docrash core.25111
...

The settings can be combined, if you want to see pipe failures but not abort the script this works:

set -o pipefail
set -o monitor
echo "hi" | ./docrash | cat
echo "pipe returned $?"

Prints the trace and continues:

$ ./err.sh
./err.sh: line 21: 25110 Done                               echo "hi"
                   25111 Segmentation fault (core dumped) | ./docrash
                   25112                                  | cat
pipe returned 139