19

When you want to redirect both stdout and stderr to the same file, you can do it by using command 1>file.txt 2>&1, orcommand &>file.txt. But why is the behavior of command 1>file.txt 2>file.txt different from the above two commands?

The following is a verification command.

$ cat redirect.sh
#!/bin/bash

{ echo -e "output\noutput" && echo -e "error" 1>&2; } 1>file.txt 2>&1
{ echo -e "output\noutput" && echo -e "error" 1>&2; } 1>file1.txt 2>file1.txt
{ echo -e "error" 1>&2 && echo -e "output\noutput"; } 1>file2.txt 2>file2.txt
{ echo -e "output" && echo -e "error\nerror" 1>&2; } 1>file3.txt 2>file3.txt
{ echo -e "error\nerror" 1>&2 && echo -e "output"; } 1>file4.txt 2>file4.txt

$ ./redirect.sh

$ echo "---file.txt---"; cat file.txt;\
echo "---file1.txt---"; cat file1.txt; \
echo "---file2.txt---"; cat file2.txt; \
echo "---file3.txt---"; cat file3.txt; \
echo "---file4.txt----"; cat file4.txt;
 ---file.txt---
output
output
error
---file1.txt---
error

output
---file2.txt---
output
output
---file3.txt---
error
error
---file4.txt----
output
rror

As far as the results are seen, it looks like that the second echo string overwrites the first echo string when you run command 1>file.txt 2>file.txt, but I do not know why it will. (Is there a reference somewhere?)

42

You need to know two things:

  • An open file descriptor known to the application-mode side of a process references an internal kernel object known as a file description, which is an instance of an open file. There can be multiple file descriptions per file, and multiple file descriptors sharing a file description.
  • The current file position is an attribute of a file description. So if multiple file descriptors map to a single file description, they all share the same current file position, and a change to the file position enacted using one such file descriptor affects all of the other such file descriptors.

    Such changes are enacted by processes calling the read()/readv(), write()/writev(), lseek(), and suchlike system calls. The echo command calls write()/writev() of course.

So what happens is this:

  • command 1>file.txt 2>&1 only creates one file description, because the shell only opens a file once. The shell makes both the standard output and standard error file descriptors map to that single file description. It duplicates standard output onto standard error. So a write via either file descriptor will move the shared current file position: each write goes after the previous write the common file description. And as you can see the results of the echo commands do not overwrite one another.
  • command 1>file.txt 2>file.txt creates two file descriptions, because the shell opens the same file twice, in response to the two explicit redirections. The standard output and standard error file descriptors map to two different file descriptions, which then in turn map to the same single file. The two file descriptions have entirely independent current file positions, and each write goes immediately the previous write on the same file description. And as you can see the result is that what is written via one can overwrite what is written via the other, in various different ways according to what order you execute the writes in.

Further reading

  • 1
    Should be open file description rather than file description. It's more about the record of how the file was opened more than the file itself. That's the terminology used by POSIX, Linux, Solaris and GNU documentation at least. – Stéphane Chazelas Sep 11 '17 at 14:26
16

Using > tells it to overwrite the file. Since you have stdout and stderr writing to the file in two different operations, the last one to write will overwrite the first.

You can do:

command 1>>file.txt 2>>file.txt

or

command &>file.txt Only bash v4 and above.

>> tells it to append the file so it won't replace the previous operations' output.

&> is just an easier way to write 2>&1

  • 2
    why does ls 1>&0 and ls 0>&0 still show the output of ls ? – Yvain Sep 10 '17 at 21:48
  • I'm surprised that using >> works. Why doesn't this have the problem of two file descriptions with independent offsets? @JdeBP, do you know? I thought that opening a file in append mode was equivalent to opening in write mode, seeking to the final position, then disallowing further seeking. – JoL Sep 10 '17 at 23:57
  • 4
    @jlmg: Append-mode files are seekable, but every write is prefixed with an implied seek to the end. Whether that implied seek is atomic is less clear to me. – Kevin Sep 11 '17 at 0:40
  • 1
    That entirely depends on the follow-on question. Ones like this which are this related indicate that the answer is incomplete. And it is unlikely that people would look up the follow-on questions without also wanting to know the full answer. Hence it is quite likely such follow on questions will have answers that duplicate information, and thus get closed as duplicates. – trlkly Sep 11 '17 at 6:33
  • 1
    @Kevin, on fully POSIX-compliant filesystems, the implicit O_APPEND seek is atomic. That said, not every filesystem implements the relevant semantics properly -- for example, NFS (at least v3 and prior -- I'm not clear on v4) doesn't have the relevant support baked into the wire protocol, so the server has no way of knowing if a client opened a file with O_APPEND. – Charles Duffy Sep 11 '17 at 17:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.