Sometimes I need to maintain programs that invoke shell scripts which invokes other programs and scripts. Therefore, when the main shell script ends with exit code 126, it is a struggle to find out which of the invoked scripts and commands set that exit code.

Is there a way to see which command was the reason for the exit code to make it easier to check for its permissions?


If you set the option set -x, then you see the commands that execute in the shell in a trace.


If on Linux, you could run the command under strace -fe process to know which process did an exit_group(126) and what command it (or any of its parent if it didn't execute anything itself) executed last before doing that:

$ strace -fe process sh -c 'env sh -c /; exit'
execve("/bin/sh", ["sh", "-c", "env sh -c /; exit"], [/* 53 vars */]) = 0
arch_prctl(ARCH_SET_FS, 0x7f24713b1700) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f24713b19d0) = 26325
strace: Process 26325 attached
[pid 26324] wait4(-1,  <unfinished ...>
[pid 26325] execve("/usr/bin/env", ["env", "sh", "-c", "/"], [/* 53 vars */]) = 0
[pid 26325] arch_prctl(ARCH_SET_FS, 0x7fbdb4e2c700) = 0
[pid 26325] execve("/bin/sh", ["sh", "-c", "/"], [/* 53 vars */]) = 0
[pid 26325] arch_prctl(ARCH_SET_FS, 0x7fef90b3b700) = 0
[pid 26325] clone(strace: Process 26326 attached
child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fef90b3b9d0) = 26326
[pid 26325] wait4(-1,  <unfinished ...>
[pid 26326] execve("/", ["/"], [/* 53 vars */]) = -1 EACCES (Permission denied)
sh: 1: /: Permission denied
[pid 26326] exit_group(126)             = ?
[pid 26326] +++ exited with 126 +++
[pid 26325] <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 126}], 0, NULL) = 26326
[pid 26325] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=26326, si_uid=10031, si_status=126, si_utime=0, si_stime=0} ---
[pid 26325] exit_group(126)             = ?
[pid 26325] +++ exited with 126 +++
<... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 126}], 0, NULL) = 26325
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=26325, si_uid=10031, si_status=126, si_utime=0, si_stime=0} ---
exit_group(126)                         = ?
+++ exited with 126 +++

Above, that was process 26326 that first exited with 126, that was because it attempted to execute /. It was a child of process 26325 that last executed sh -c /.

If those scripts are bash scripts or if they are sh scripts and sh happens to be bash on your system, you could do:

$ env SHELLOPTS=xtrace \
      BASH_XTRACEFD=7 7>&2 \
    sh -c 'env sh -c /; exit'
[0][30625|env sh -c /; exit|0]+ env sh -c /
[0][30626|/|0]+ /
sh: /: Is a directory
[126][30625|env sh -c /; exit|0]+ exit

That doesn't tell us exactly what process exited with 126 but could give you enough clue.

We use BASH_TRACEFD=7 7>&2 so that the traces are output on the original stderr, even when stderr is redirected within the scripts. Otherwise those trace messages could affect the behaviour of the scripts if they do things like (....) 2>&1 | .... That assumes those scripts don't explicitly use or close fd 7 themselves (that's would be unlikely, a lot more unlikely than them redirecting stderr).

  • You can use BASH_XTRACEFD= to make trace output to go to standard error. – cuonglm Sep 8 '16 at 18:05
  • 1
    @cuonglm, the xtrace goes on stderr by default, the point here is to make sure that is not affected by stderr redirections. See edit. – Stéphane Chazelas Sep 8 '16 at 18:51

It's a bit of a hack, but you can preload a bit of C code as a shim to trap the call to exit(126) and have it issue a SIGSTOP signal to the process group, which will pause the process (and its parents in the same group).

For example, if we trap exit code 2 in our shim then run ls on a non-existant file:

LD_PRELOAD=/home/meuh/shim_exit.so bash -c ' sh -c "ls -l xxx; echo"; echo '

it will background itself with the message

[1]+  Stopped  ...

and you can see the processes in status T wait:

~ $ ps f
30528 pts/3    T      0:00  \_ bash -c sh -c "ls -l xxx;echo";echo
30529 pts/3    T      0:00  |   \_ sh -c ls -l xxx;echo
30530 pts/3    T      0:00  |       \_ ls -l xxx

At this stage you could attach to the processes if they are debuggable, or simply foreground or SIGCONT the processes to continue.

Here's the shim_exit.c code, see the C comment for compiling.

 * capture calls to a routine and replace with your code
 * http://unix.stackexchange.com/a/308694/119298
 * gcc -Wall -O2 -fpic -shared -ldl -o shim_exit.so shim_exit.c
 * LD_PRELOAD=/home/meuh/shim_exit.so ./test
#define _GNU_SOURCE /* needed to get RTLD_NEXT defined in dlfcn.h */
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <signal.h>
#include <unistd.h>
#include <dlfcn.h>
/* doesnt work for syscall exit_group() */
void exit(int status){
    static void (*real_exit)(int status) = NULL;

    if (!real_exit) {
        real_exit = dlsym(RTLD_NEXT, "exit");
        char *error = dlerror();
        if (error != NULL) {
            fprintf(stderr, "%s\n", error);
    if (status==126/* || status==2*/)kill(0,SIGSTOP);
  • Clever, though probably doesn't work for commands that call _exit() instead of exit().(you'd want to wrap that one as well). – Stéphane Chazelas Sep 8 '16 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.