0

I'm trying to create a general-purpose function or script that I can pipe data into, and prepend the output showing which fd the data "arrived" on (stdout or stderr). My Bash skills are intermediate at best, and I find myself in over my head trying to wrap my head around statements like { foo 2>&1 >&3 3>&- | bar 3>&-; } 3>&1.

Can anyone suggest if this is possible, and a way to do it?

Essentially I want to do something like this:

$ { echo "foo" ; echo "bar" 1>&2 ; } | my_output_processor.sh
stdout: foo
stderr: bar 

I've read Piping STDERR vs. STDOUT but that didn't answer it for me. I think the answer at https://unix.stackexchange.com/a/440439/307184 might contain a clue, but it seems that would work within the running script only, not with piped in data.

My Bash version is 5.0 on macOS.

edit: @Glenn's answer below is working! I made a couple of small helper functions to save a bit of typing and prevent errors, and tested like this:

$ out(){ sed "s/^/out: /" ;} ; err(){ sed "s/^/err: /" 1>&2 ;}
$ { echo "good" ; echo "bad" >&2; } 2> >(err) 1> >(out)
err: bad
out: good

Followup question about the ordering of the redirection: why does 2> >(err) 1> >(out) work great, while 1> >(out) 2> >(err) does not? (update: answered by @RudiC below - helper functions updated so they work in any order now)

1
  • 1
    Keep in mind that the pipe ONLY connects stdout (file descriptor 1) of the left-hand side to the stdin of the right-hand side. Feb 26 '20 at 23:41
2

Use Process Substitution not pipes:

{ echo "foo" ; echo "bar" >&2; } 2> >(sed "s/^/err: /") > >(sed "s/^/out: /")
err: bar
out: foo

If you're doing this in a script, use the exec command to set up the redirection for the whole duration of the script:

bash -c '
    exec 2> >(sed "s/^/err: /") > >(sed "s/^/out: /")
    echo foo
    echo bar >&2
'
err: bar
out: foo

If you do

exec > >(sed "s/^/out: /") 2> >(sed "s/^/err: /")

then the output is

out: foo
out: err: bar

I assume this is what you mean by "doesn't work". That's because the output of the "err" process substitution by default goes to stdout, and that's now pointing at the "out" process sub. You would have to do this to get around it:

exec 3>&1 > >(sed "s/^/out: /") 2> >(sed "s/^/err: /" >&3)

Create another file descriptor (3) that points to the default stdout, then the stderr redirection prints to fd3.

1
  • Brilliant! This is exactly what I was after. I updated my OP to reflect a followup question on the ordering of the redirects and why it produces different results.
    – luckman212
    Feb 27 '20 at 0:19
1

Follow up question answers itself when executed:

The err function prints to stdout as well. Obviously the first form has the original stdout, while the second already uses the "redirected to out() function" one.

1
  • Thanks @RudiC - I modified the helper functions a bit so that the redirection order is interchangeable now.
    – luckman212
    Feb 28 '20 at 5:37
0

This should work in any standard shell, not just bash:

{ cmd 2>&1 >&3 | sed 's/^/err: /' >&2; } 3>&1 | sed 's/^/out: /'

This could be made into a function, for convenience:

# usage louterr cmd [args ...]
louterr(){ { "$@" 2>&1 >&3 | sed 's/^/err: /' >&2; } 3>&1 | sed 's/^/out: /'; }

cmd(){ echo ERR >&2; echo OUT; }
louterr cmd
out: OUT
err: ERR
louterr louterr louterr cmd
err: err: err: ERR
out: out: out: OUT

or, with configurable "filters":

# usage: louterrx out_filter err_filter cmd [args ...]
louterrx(){
   oflt=$1; eflt=$2; shift; shift
   eval "{ \"\$@\" 2>&1 >&3 | $eflt >&2; } 3>&1 | $oflt"
}

louterrx "sed 's/^/out: /'" "sed 's/^/err: /'" cmd

A disadvantage of this is that the exit status of cmd will be hidden (as with any right side of a pipeline). To work around that, you could use a subshell and set -o pipefail (supported in bash, zsh, ksh, etc but not in dash yet):

louterr()(
   set -o pipefail
   { "$@" 2>&1 >&3 | sed 's/^/err: /' >&2; } 3>&1 | sed 's/^/out: /'
)
cmd(){ echo ERR >&2; echo OUT; return 13; }
louterr cmd; echo $?
out: OUT
err: ERR
13

I do not recommend using the > >(...) zsh/bash-ism in this case, since it relies on quirks related to the order in which expansions and redirections are performed in different shells (and is subject to a lot of other quirks and bugs). For example, in

ls no_such_file >/dev/null 2> >(cat)

the output of cat (the error message of ls) will be redirected to /dev/null in bash but not in zsh.

~the only situation where you have to use the cmd > >(...) form is when cmd is a function whose purpose is to modify the current environment and which cannot be run in a subshell.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.