32

How can I achieve

cmd >> file1 2>&1 1>>file2

That is, the stdout and stderr should redirect to one file (file1) and only stdout (file2) should redirect to another (both in append mode)?

41

Problem is that when you redirect your output, it's not available anymore for the next redirect. You can pipe to tee in a subshell to keep the output for the second redirection:

( cmd | tee -a file2 ) >> file1 2>&1

or if you like to see the output in terminal:

( cmd | tee -a file2 ) 2>&1 | tee -a file1

To avoid adding the stderr of the first tee to file1, you should redirect the stderr of your command to some file descriptor (e.g. 3), and later add this to stdout again:

( 2>&3 cmd | tee -a file2 ) >> file1 3>&1
# or
( 2>&3 cmd | tee -a file2 ) 3>&1 | tee -a file1

(thanks @fra-san)

16

With zsh:

cmd >& out+err.log > out.log

In append mode:

cmd >>& out+err.log >> out.log

In zsh, and provided the mult_ios option has not been disabled, when a file descriptor (here 1) is redirected several times for writing, then the shell implements a built-in tee to duplicate the output to all targets.

  • I can't figure out what out+err and out mean here. File names? Streams to be redirected? – gronostaj May 13 at 20:27
  • @gronostaj Think that the command reads cmd >& file1 > file2 – Isaac May 13 at 22:00
  • This solution will preserve the order in which the output was generated. To actually store the stdout and stderr (in that order) you need a different approach. – Isaac May 13 at 22:02
  • 1
    @Isaac, the order will not necessarily be preserved as the stdout output will go through a pipe (to a process that forwards it to each file) while the stderr output will go directly to the file. In any case, it doesn't look like the OP asked for the stderr output to come after the stdout one. – Stéphane Chazelas May 14 at 11:07
3

You could : tag stdout (using an UNBUFFERED sed, ie: sed -u ...), have stderr also go to stdout (untagged, as it didn't go through that tagging sed), and thus be able to differentiate the 2 in the resulting logfile.

The following: is slow (It can be seriously optimized, by using for exemple a perl script instead of the while ... ; do ... ;done, for exemple, that will spawning subshells & commands at every lines!), weird (it seems I need the 2 {} stages to in one rename stdout and then in the other one add the "falled through" stderr to it), etc. But it is : a "proof of concept", that will try to keep the output's order the most of stdout & stderr as much as possible:

#basic principle (some un-necessary "{}" to visually help see the layers):
# { { complex command ;} | sed -e "s/^/TAGstdout/" ;} 2>&1 | read_stdin_and_redispatch

#exemple:
# complex command = a (slowed) ls of several things (some existing, others not)
#  to see if the order of stdout&stderr is kept

#preparation, not needed for the "proof of concept", but needed for our specific exemple setup:
\rm out.file out_AND_err.file unknown unknown2 
touch existing existing2 existing3

#and the (slow, too many execs, etc) "proof of concept":
uniquetag="_stdout_" # change this to something unique, that will NOT appear in all the commands outputs... 
                     # avoid regexp characters ("+" "?" "*" etc) to make it easy to remove with another sed later on.

{
   { for f in existing unknown existing2 unknown2 existing3 ; do ls -l "$f" ; sleep 1; done ;
   } | sed -u -e "s/^/${uniquetag}/" ;
} 2>&1 | while IFS="" read -r line ; do
    case "$line" in
       ${uniquetag}*) printf "%s\n" "$line" | tee -a out_AND_err.file | sed -e "s/^${uniquetag}//" >> out.file ;; 
        *)            printf "%s\n" "$line"       >> out_AND_err.file ;;   
    esac; 
done;

# see the results:
grep "^" out.file out_AND_err.file
  • This is really difficult to understand. (1) Why do you use such complicated use case (ls unknown) to print something on stderr? >&2 echo "error" would be fine. (2) tee can append to multiple files at once. (3) Why not just cat instead of grep "^" ? (4) your script will fail when stderr output begins with _stdout_. (5) Why? – pLumo May 13 at 13:39
  • @RoVo : the first 2 commented lines shows the algorithm, simpler than the proof of concept example. 1) : this ls loop will both output on stdout and stderr, mixed (alternatively), in a controlled order,;so that we can check we kept that stderr/stdout ordering despite the tagging of stdout 2) : gnu tail, maybe, but not regular tail (ex, on aix.). 3) : grep "^" also shows both filenames. 4) : this can be changed by the variable. 5) : the convoluted exemple works on old oses (ex, old aix) where I tested it (no perl available). – Olivier Dulac May 13 at 13:48
  • (1) multiple echo to stderr and stout would be fine, but okay, not important, would just be easier to read. (3) agree, (4) sure, but it will fail if it starts with whatever the variable contains. (5) I see. – pLumo May 13 at 13:51
  • @RoVo I agree with your 1). for 4), the variable could be as complex as needed to make the pb disappear (ex: uniquetag="banaNa11F453355B28E1158D4E516A2D3EDF96B3450406 ... ) – Olivier Dulac May 13 at 13:54
  • 1
    Sure sure, it's not very likely, but anyways it might introduce a security issue later on. And then you might want to remove that string before printing to file ;-) – pLumo May 13 at 13:55
2

If order of output must be: stdout then stderr; there is no solution with redirection only.
The stderr must be stored to a temporal file

cmd 2>>file-err | tee -a file1 >>file2
cat file-err >> file1
rm file-err

Description:

The only way to redirect one output (an fd like stdout or stderr) to two files is to reproduce it. The command tee is the correct tool to reproduce a file descriptor content. So, an initial idea to have one output on two files would be to use:

... |  tee file1 file2

That reproduces the stdin of tee to both files (1 & 2) leaving the output of tee still unused. But we need to append (use -a) and only need one copy. This solve both issues:

... | tee -a file1 >>file2

To supply tee with stdout (the one to repeat) we need to consume stderr directly out of the command. One way, if order is not important (order of output will (most probably) be preserved as generated, whichever is output first will be stored first). Either:

  1. cmd 2>>file1 | tee -a file2 >>file1
  2. cmd 2>>file1 > >( tee -a file2 >>file1 )
  3. ( cmd | tee -a file2 ) >> file1 2>&1

Option 2 works only in some shells. Option 3 uses an additional subshell (slower) but use the file names only once.

But if stdout must be first (whichever order output is generated) we need to store stderr to append it to file at the end (first solution posted).

  • 1
    Or store in memory like sponge does: (cmd | tee -a out >> out+err) 2>&1 | sponge >> out+err – Stéphane Chazelas May 14 at 11:15
1

In the interests of diversity:

If your system supports /dev/stderr, then

(cmd | tee -a /dev/stderr) 2>> file1 >> file2

will work.  The standard output of the cmd is sent to both the stdout and the stderr of the pipeline. The standard error of the cmd bypasses the tee and comes out the stderr of the pipeline.

So

  • the stdout of the pipeline is just the stdout of the cmd, and
  • the stderr of the pipeline is the stdout and stderr of the cmd, intermixed.

It's then a simple matter of sending those streams to the correct files.

As with almost any approach like this (including Stéphane’s answer), file1 may get lines out of order.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.