How to save stdout to file, stderr to file, stdout+stderr to file and get stdout + stderr to terminal like normal for a shell script

Question

How can I save stdout to one file, stderr to another file, stdout+stderr to a third file and also get stdout + stderr to terminal like normal for a shell script?

I found this elsewhere:

exec > >(tee std_out) 2> >(tee err_out >&2)
ls # Should got to std_out
fsdfs # Command not found goes to err_out

Which is really close. If I run bash test.sh 2>&1 | tee output then it works, but I don't have access to how my script is run. It's a cicd system. I need to be able to do the "combined output" from inside the script using exec.

I'm creating a CI/CD library and I'm unable to know what the clients would use the library for, so I want to account for each use case.

Answer 1

Simply expanding on your approach:

exec 2> >(tee -a stderr stdall) 1> >(tee -a stdout stdall)

Standard error will be written to the file named stderr, standard output to stdout and both standard error and standard output will also be written to the console (or whatever the two file descriptors are pointing at the time exec is run) and to stdall.
tee -a (append) is required to prevent stdall from being overwritten by the second tee that starts writing to it.

Note that the order in which redirections are performed is relevant: the second process substitution is affected by the first redirection, i.e. the errors it emitted would be sent to >(tee -a stderr stdall). You can, of course, redirect the second process substitution's standard error to /dev/null to avoid this side effect. Redirecting standard output before standard error would send every error to stdout and stdall too.

Since the commands in Bash's process substitutions are executed asynchronously, there is no way to guarantee that their output will be displayed in the order it was generated. Worse, fragments from standard output and standard error are likely to end up appearing on the same line.

Answer 2

Your script can run itself all over via $0 (with setting and checking an environment variable in order to avoid infinite recursion) instead of relying on bash's > >(...) construct, which IMLE is capricious and unreliable.

if [ "$REDIRECTED" != 1 ]; then
        export REDIRECTED=1
        set -o pipefail
        { { "$0" | tee stdout >&3; } 2>&1 | tee stderr; } 3>&1 | tee stdboth
        exit
fi
# rest of your script here

Since tee does not use line buffering (nor could it be forced to do so with stdbuf(1)), the order of the data written to stdout and stderr will not be respected in the final output. With a command which is using full buffering and writing to both stdout and stderr, even a line-buffering tee won't help and, worse, you may get in the output lines which are half stdout and half stderr.

I don't think there's any fix for that using just the shell language and readily available command-line utilites.

Answer 3

I'm creating a CI/CD library and I'm unable to know what the clients would use the library for, so I want to account for each use case.

I question the need of bash for processing the outputs, given this is the scenario. Ideally in this context you would want to timestamp the output and give it a id for the standard output type and that the application should be the one that decides what to do with the messages.