22

How can I use BASH magic to achieve this?
I want to only see stderr output on the screen,
but I want both stdout and stderr to be written to a file.

Clarification: I want both stdout and stderr to end up in the same file. In the order they happen.
Unfortunately none of the answers below does this.

14

Even without any redirection, or with nothing but >logfile 2>&1, you're not guaranteed to see output in order of generation.

For starters, the stdout from the application will be line-buffered (to tty) or buffered (to a pipeline) but stderr is unbuffered, so relationships between the order of output are broken as far as a reader is concerned. Subsequent stages in any pipeline you could concoct will not get deterministically ordered access to the two streams (they are conceptually things happening in parallel, and you're always subject to the scheduler - if by the time your reader gets a slice the writer has already written to both pipes, you cannot tell which came first).

"[T]he order they happen" is only really known to the application. Ordering of output across stdout/stderr is a well-known - classic, perhaps - problem.

6

When you use the construction: 1>stdout.log 2>&1 both stderr and stdout get redirected to the file because the stdout redirection is set up before the stderr redirection.

If you invert order you can get stdout redirected to a file and then copy stderr to stdout so you can pipe it to tee.

$ cat test
#!/bin/sh
echo OUT! >&1
echo ERR! >&2

$ ./test 2>&1 1>stdout.log | tee stderr.log
ERR!

$ cat stdout.log
OUT!

$ cat stderr.log
ERR!
  • That doesn't let you log the two streams to the same file. – artistoex Mar 19 '11 at 16:19
  • 1
    @artistoex: You just can use tee -a with the same file used in 1> to sum both outputs in the same file. Something like: ./test 2>&1 1>out+err.log | tee -a out+err.log – mmoya Mar 19 '11 at 17:41
  • I have no idea why, but this does not work with wget -O - www.google.de for me. (compare with solution below) – artistoex Mar 19 '11 at 17:56
  • 4
    Using tee -a won't work reliably to have the same output that would be on the console if nothing was redirected. The output that goes via tee might be slightly delayed compared with the output that comes directly from the command, and so might appear later in the log. – Gilles Mar 19 '11 at 20:33
  • This does not work for me, out+err.log is empty. And yes, I want stderror and standard out in the same file – Andreas Mar 19 '11 at 20:41
6

I have been able to accomplish this by placing the following line at the top of a bash script:

exec 1>>log 2> >(tee -a log >&2)

This will redirect stdout to the file log (1>>log), then tee stderr to the file log (2> >(tee -a log) and direct it back to stderr (>&2). In this way, I get a single file, log, which shows both stdout and stderr in order, and stderr is also displayed on the screen as usual.

The catch is that it only seems to work when I append to the file. If I don't append, it appears that the two redirections clobber each other, and I only get whichever one outputs last.

  • Any way to get this to "modify" the STDERR lines to include a custom string so you can easily grep on it? – IMTheNachoMan Oct 18 '18 at 2:35
1

You want to duplicate the error stream so that it appears both on the console and in the log file. The tool for that is tee, and all you need to do is apply it to the error stream. Unfortunately, there's no standard shell construct to pipe a command's error stream into another command, so a little file descriptor rearrangement is required.

{ { echo out; echo err 1>&2; } 2>&1 >&3 | tee /dev/tty; } >log 3>&1
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^   ^^^^^^^^^^^^    ^^^^^^^^^
  command produces output      stdout→3                    →log
  command produces error       stderr→1   dup to terminal  →log
  • This also, does not quite work for me, I get stdout lines first, and then all stderr lines in 'log' file. example: { { echo out; echo err 1>&2;echo out2;echo err2 1>&2; } 2>&1 >&3 | tee /dev/tty; } >log 3>&1 – Andreas Mar 19 '11 at 20:56
  • 1
    @Andreas: Oh, duh, tee introduces a delay here too. I don't think there is a pure shell solution. In fact I wonder whether there is a solution without modifying the application in some way. – Gilles Mar 19 '11 at 21:28
1

To write stderr to screen and write BOTH stderr and stdout to a file -- AND have the lines for stderr and stdout come out in the same sequence they would if both were written to the screen:

Turns out to be a difficult problem, especially the part about having "same sequence" you would expect if you simply wrote them to the screen. In simple terms: Write each to its own file, do some background-process magic to mark each line (in each file) with the exact time the line was produced, and then: "tail --follow" the stderr file to the screen, but to see BOTH "stderr" and "stdout" together -- in sequence -- sort the two files (with exact-time markings on each line) together.

Code:

# Set the location of output and the "first name" of the log file(s)
pth=$HOME
ffn=my_log_filename_with_no_extension
date >>$pth/$ffn.out
date >>$pth/$ffn.err
# Start background processes to handle 2 files, by rewriting each one line-by-line as each line is added, putting a label at front of line
tail -f $pth/$ffn.out | perl -nle 'use Time::HiRes qw(time);print substr(time."0000",0,16)."|1|".$_' >>$pth/$ffn.out.txt &
tail -f $pth/$ffn.err | perl -nle 'use Time::HiRes qw(time);print substr(time."0000",0,16)."|2|".$_' >>$pth/$ffn.err.txt &
sleep 1
# Remember the process id of each of 2 background processes
export idout=`ps -ef | grep "tail -f $pth/$ffn.out" | grep -v 'grep' | perl -pe 's/\s+/\t/g' | cut -f2`
export iderr=`ps -ef | grep "tail -f $pth/$ffn.err" | grep -v 'grep' | perl -pe 's/\s+/\t/g' | cut -f2`
# Run the command, sending stdout to one file, and stderr to a 2nd file
bash mycommand.sh 1>>$pth/$ffn.out 2>>$pth/$ffn.err
# Remember the exit code of the command
myexit=$?
# Kill the two background processes
ps -ef | perl -lne 'print if m/^\S+\s+$ENV{"idout"}/'
echo kill $idout
kill $idout
ps -ef | perl -lne 'print if m/^\S+\s+$ENV{"iderr"}/'
echo kill $iderr
kill $iderr
date
echo "Exit code: $myexit for '$listname', list item# '$ix', bookcode '$bookcode'"

Yes, this seems elaborate, and it results in 4 output files (2 of which you can delete). It does appear this is a hard problem to solve, so it took several mechanisms.

In the end, to see the results from BOTH stdout and stderr in the sequence you would expect, run this:

cat $pth/$ffn.out.txt $pth/$ffn.err.txt | sort

The only reason the sequence is at least very close to what it would have been had both stdout and stderr simply gone to the screen is: Every single line is marked with a timestamp down to the millisecond.

To see stderr on screen as the process goes along, use this:

tail -f $pth/$ffn.out

Hope that helps someone out, who arrived here long after the original question was asked.

  • +1 for the sheer effort, I had a similar idea (timestamps both streams via perl) but was struggling to implement it . I'll investigate if it works for me (ie, if it really keeps the order of outputs, as others solutions here shift things a bit due to asynchronicity between tagging of stderr and stdout) – Olivier Dulac Dec 21 '16 at 16:50
0

Let f be the command you'd like be executed, then this

( exec 3>/tmp/log; f 2>&1 1>&3 |tee >(cat)>&3 )

should give you what you wish. For example wget -O - www.google.de would look like this:

( exec 3>/tmp/googlelog; wget -O - www.google.de 2>&1 1>&3 |tee >(cat)>&3 )
  • 1
    This almost works for me, but in the log file I first get all the stdout lines, and then all stderror lines. I want them to be interleaved in the natural order as they happen. – Andreas Mar 19 '11 at 20:44
0

Given what I've read here, the only solution I can see would be to prepend each line whether STOUT or STERR with a timeslice/timestamp. date +%s would get to seconds, but thats to slow to maintain order. Also the log would somehow have to be sorted. More work than I'm capable of.

0

I've struggled with this exact requirement, and in the end couldn't find a simple solution. What I ended up doing instead was this:

TMPFILE=/tmp/$$.log
myCommand > $TMPFILE 2>&1
[ $? != 0 ] && cat $TMPFILE
date >> myCommand.log
cat $TMPFILE >> myCommand.log
rm -f $TMPFILE

It appends stdout and stderr, in the proper interleaved order, to the log file. And if any errors occurred (as determined by the exit status of the command), it sends the entire output (stdout and stderr) to stdout, again in proper interleaved order. I found this to be a reasonable compromise for my needs. If you want a single-run log file rather than a growing multi-run one, it's even simpler:

myCommand > myCommand.log 2>&1
[ $? != 0 ] && cat myCommand.log
0

stderr to command-substituted tee -a, and stdout appending to same file:

./script.sh 2> >(tee -a outputfile) >>outputfile

and note: too ensure correct order (but without stderr-show) there is 'unbuffer' command from 'expect' tools, that simulates tty and keeps stdout/err in order as it would show on terminal:

unbuffer ./script.sh > outputfile

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.