4

How can I save stdout to one file, stderr to another file, stdout+stderr to a third file and also get stdout + stderr to terminal like normal for a shell script?

I found this elsewhere:

exec > >(tee std_out) 2> >(tee err_out >&2)
ls # Should got to std_out
fsdfs # Command not found goes to err_out

Which is really close. If I run bash test.sh 2>&1 | tee output then it works, but I don't have access to how my script is run. It's a cicd system. I need to be able to do the "combined output" from inside the script using exec.

I'm creating a CI/CD library and I'm unable to know what the clients would use the library for, so I want to account for each use case.

7
  • 1
    Do you want stdout to one file, stderr to another file, stdout+stderr to a third file and both also on the terminal? So a total of 3 files + terminal output, or can some of those files be the same?
    – terdon
    Apr 14 '20 at 15:11
  • @terdon As you described. stdout to its own file, stderr to its own file and combined to its own file and terminal. 3 total files.
    – Levi
    Apr 14 '20 at 15:13
  • Why is the combined file necessary?
    – Braiam
    Apr 14 '20 at 17:59
  • @Braiam This is for a CI/CD tool. If you are in the dashboard you will want to see the output to the terminal like normal. If you are using this for your build you might want the stdout file for success notifications and stderr for failure notifications. The problem is not all linux programs obey these rules. Some print errors to stdout and some print useful information to stderr. I can't combine stdout and stderr because I wouldn't beable to combine them to match the way it looked in the terminal. So I would like the combined file as well. Doable in zsh but cant figure it out in bash =/
    – Levi
    Apr 14 '20 at 18:31
  • 1
    Shouldn't your CI tool already manage that? AFAIK, is trivial for the CI to listen to the output and process it accordingly (save files, online viewing, etc.)
    – Braiam
    Apr 15 '20 at 2:06
5

Simply expanding on your approach:

exec 2> >(tee -a stderr stdall) 1> >(tee -a stdout stdall)

Standard error will be written to the file named stderr, standard output to stdout and both standard error and standard output will also be written to the console (or whatever the two file descriptors are pointing at the time exec is run) and to stdall.
tee -a (append) is required to prevent stdall from being overwritten by the second tee that starts writing to it.

Note that the order in which redirections are performed is relevant: the second process substitution is affected by the first redirection, i.e. the errors it emitted would be sent to >(tee -a stderr stdall). You can, of course, redirect the second process substitution's standard error to /dev/null to avoid this side effect. Redirecting standard output before standard error would send every error to stdout and stdall too.

Since the commands in Bash's process substitutions are executed asynchronously, there is no way to guarantee that their output will be displayed in the order it was generated. Worse, fragments from standard output and standard error are likely to end up appearing on the same line.

0
2

Your script can run itself all over via $0 (with setting and checking an environment variable in order to avoid infinite recursion) instead of relying on bash's > >(...) construct, which IMLE is capricious and unreliable.

if [ "$REDIRECTED" != 1 ]; then
        export REDIRECTED=1
        set -o pipefail
        { { "$0" | tee stdout >&3; } 2>&1 | tee stderr; } 3>&1 | tee stdboth
        exit
fi
# rest of your script here

Since tee does not use line buffering (nor could it be forced to do so with stdbuf(1)), the order of the data written to stdout and stderr will not be respected in the final output. With a command which is using full buffering and writing to both stdout and stderr, even a line-buffering tee won't help and, worse, you may get in the output lines which are half stdout and half stderr.

I don't think there's any fix for that using just the shell language and readily available command-line utilites.

2
  • I got a command not found error for this line { { "$0" | tee stdout >&3; } 2>&1 | tee stderr; } 3>&1 | tee stdboth I do have tee so it's not that?
    – Levi
    Apr 15 '20 at 8:55
  • 1
    That's probably because your run your script as bash script instead of bash ./script or such. That snippet assumes that it's an executable script, which is called as /path/to/script or ./script. If not, you could adapt it by changing "$0" to bash "$0".
    – mosvy
    Apr 15 '20 at 9:00
1

I'm creating a CI/CD library and I'm unable to know what the clients would use the library for, so I want to account for each use case.

I question the need of bash for processing the outputs, given this is the scenario. Ideally in this context you would want to timestamp the output and give it a id for the standard output type and that the application should be the one that decides what to do with the messages.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.