Show only stderr on screen but write both stdout and stderr to file

Question

How can I use BASH magic to achieve this?
I want to only see stderr output on the screen,
but I want both stdout and stderr to be written to a file.

Clarification: I want both stdout and stderr to end up in the same file. In the order they happen.
Unfortunately none of the answers below does this.

Answer 1

Even without any redirection, or with nothing but >logfile 2>&1, you're not guaranteed to see output in order of generation.

For starters, the stdout from the application will be line-buffered (to tty) or buffered (to a pipeline) but stderr is unbuffered, so relationships between the order of output are broken as far as a reader is concerned. Subsequent stages in any pipeline you could concoct will not get deterministically ordered access to the two streams (they are conceptually things happening in parallel, and you're always subject to the scheduler - if by the time your reader gets a slice the writer has already written to both pipes, you cannot tell which came first).

"[T]he order they happen" is only really known to the application. Ordering of output across stdout/stderr is a well-known - classic, perhaps - problem.

Answer 2

When you use the construction: 1>stdout.log 2>&1 both stderr and stdout get redirected to the file because the stdout redirection is set up before the stderr redirection.

If you invert order you can get stdout redirected to a file and then copy stderr to stdout so you can pipe it to tee.

$ cat test
#!/bin/sh
echo OUT! >&1
echo ERR! >&2

$ ./test 2>&1 1>stdout.log | tee stderr.log
ERR!

$ cat stdout.log
OUT!

$ cat stderr.log
ERR!

Answer 3

I have been able to accomplish this by placing the following line at the top of a bash script:

exec 1>>log 2> >(tee -a log >&2)

This will redirect stdout to the file log (1>>log), then tee stderr to the file log (2> >(tee -a log) and direct it back to stderr (>&2). In this way, I get a single file, log, which shows both stdout and stderr in order, and stderr is also displayed on the screen as usual.

The catch is that it only seems to work when I append to the file. If I don't append, it appears that the two redirections clobber each other, and I only get whichever one outputs last.

Answer 4

To write stderr to screen and write BOTH stderr and stdout to a file -- AND have the lines for stderr and stdout come out in the same sequence they would if both were written to the screen:

Turns out to be a difficult problem, especially the part about having "same sequence" you would expect if you simply wrote them to the screen. In simple terms: Write each to its own file, do some background-process magic to mark each line (in each file) with the exact time the line was produced, and then: "tail --follow" the stderr file to the screen, but to see BOTH "stderr" and "stdout" together -- in sequence -- sort the two files (with exact-time markings on each line) together.

Code:

# Set the location of output and the "first name" of the log file(s)
pth=$HOME
ffn=my_log_filename_with_no_extension
date >>$pth/$ffn.out
date >>$pth/$ffn.err
# Start background processes to handle 2 files, by rewriting each one line-by-line as each line is added, putting a label at front of line
tail -f $pth/$ffn.out | perl -nle 'use Time::HiRes qw(time);print substr(time."0000",0,16)."|1|".$_' >>$pth/$ffn.out.txt &
tail -f $pth/$ffn.err | perl -nle 'use Time::HiRes qw(time);print substr(time."0000",0,16)."|2|".$_' >>$pth/$ffn.err.txt &
sleep 1
# Remember the process id of each of 2 background processes
export idout=`ps -ef | grep "tail -f $pth/$ffn.out" | grep -v 'grep' | perl -pe 's/\s+/\t/g' | cut -f2`
export iderr=`ps -ef | grep "tail -f $pth/$ffn.err" | grep -v 'grep' | perl -pe 's/\s+/\t/g' | cut -f2`
# Run the command, sending stdout to one file, and stderr to a 2nd file
bash mycommand.sh 1>>$pth/$ffn.out 2>>$pth/$ffn.err
# Remember the exit code of the command
myexit=$?
# Kill the two background processes
ps -ef | perl -lne 'print if m/^\S+\s+$ENV{"idout"}/'
echo kill $idout
kill $idout
ps -ef | perl -lne 'print if m/^\S+\s+$ENV{"iderr"}/'
echo kill $iderr
kill $iderr
date
echo "Exit code: $myexit for '$listname', list item# '$ix', bookcode '$bookcode'"

Yes, this seems elaborate, and it results in 4 output files (2 of which you can delete). It does appear this is a hard problem to solve, so it took several mechanisms.

In the end, to see the results from BOTH stdout and stderr in the sequence you would expect, run this:

cat $pth/$ffn.out.txt $pth/$ffn.err.txt | sort

The only reason the sequence is at least very close to what it would have been had both stdout and stderr simply gone to the screen is: Every single line is marked with a timestamp down to the millisecond.

To see stderr on screen as the process goes along, use this:

tail -f $pth/$ffn.out

Hope that helps someone out, who arrived here long after the original question was asked.

Answer 5

You want to duplicate the error stream so that it appears both on the console and in the log file. The tool for that is tee, and all you need to do is apply it to the error stream. Unfortunately, there's no standard shell construct to pipe a command's error stream into another command, so a little file descriptor rearrangement is required.

{ { echo out; echo err 1>&2; } 2>&1 >&3 | tee /dev/tty; } >log 3>&1
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^   ^^^^^^^^^^^^    ^^^^^^^^^
  command produces output      stdout→3                    →log
  command produces error       stderr→1   dup to terminal  →log

Answer 6

I've struggled with this exact requirement, and in the end couldn't find a simple solution. What I ended up doing instead was this:

TMPFILE=/tmp/$$.log
myCommand > $TMPFILE 2>&1
[ $? != 0 ] && cat $TMPFILE
date >> myCommand.log
cat $TMPFILE >> myCommand.log
rm -f $TMPFILE

It appends stdout and stderr, in the proper interleaved order, to the log file. And if any errors occurred (as determined by the exit status of the command), it sends the entire output (stdout and stderr) to stdout, again in proper interleaved order. I found this to be a reasonable compromise for my needs. If you want a single-run log file rather than a growing multi-run one, it's even simpler:

myCommand > myCommand.log 2>&1
[ $? != 0 ] && cat myCommand.log

Answer 7

stderr to command-substituted tee -a, and stdout appending to same file:

./script.sh 2> >(tee -a outputfile) >>outputfile

and note: too ensure correct order (but without stderr-show) there is 'unbuffer' command from 'expect' tools, that simulates tty and keeps stdout/err in order as it would show on terminal:

unbuffer ./script.sh > outputfile

Answer 8

Let f be the command you'd like be executed, then this

( exec 3>/tmp/log; f 2>&1 1>&3 |tee >(cat)>&3 )

should give you what you wish. For example wget -O - www.google.de would look like this:

( exec 3>/tmp/googlelog; wget -O - www.google.de 2>&1 1>&3 |tee >(cat)>&3 )

Answer 9

Given what I've read here, the only solution I can see would be to prepend each line whether STOUT or STERR with a timeslice/timestamp. date +%s would get to seconds, but thats to slow to maintain order. Also the log would somehow have to be sorted. More work than I'm capable of.