38

If we use echo 1234 >> some-file then Documentation says that the output is appended.

My guess is that, if some-file does not exist, then O_CREAT will make a new file. If > was used, then O_TRUNC will truncate existing file.

In case of >> : Will the file be opened as O_WRONLY (or O_RDWR) and seeked to end and write operation is done , simulating O_APPEND ? Or will the file be opened as O_APPEND , leaving it to the kernel to make sure appending happens ?

I am asking this because a conserver process is overwriting some markers inserted by echo, when the output file is from NFS mount point, & NFS Documentation says O_APPEND is not supported on server, so client kernel will have to handle it. I guess conserver process is using O_APPEND , but not sure of bash >> on linux, hence asking the question here.

| improve this question | |
  • 12
    The problem on NFS is not that O_APPEND isn't supported; the problem is it's emulated. On a local file system, several processes writing to the same file opened with O_APPEND will never overwrite each other's data; on NFS, O_APPEND is emulated by seeking to the end before writing, which leaves the possibility of race conditions. There's no way around this on NFS; each parallel writer needs to write its own file. The only way to work around this is setup a server process on the NFS server, have the loggers log to |nc server port, and have the server append incoming data to the log. – Guntram Blohm May 11 '15 at 22:05
  • @GuntramBlohm , +1 , thanks for the confirmation. Basically, your suggestion is to use only one writer process to the file, and all other writer processes will go through this process. – Prem May 12 '15 at 2:41
  • So many good answers, Not sure which answer I should accept. First Bruce Ediger showed that O_APPEND is used. Next Random832 showed that this is given in the standards. Finally, Eric Renouf showed the source code with the same answer. All three perspectives add to the final complete picture. – Prem May 12 '15 at 2:55
  • 6
    In short, NFS is a load of bugs and should not be used. – R.. GitHub STOP HELPING ICE May 12 '15 at 13:21
  • 2
    Yeah, but we already learned that way back when O_EXCL was invented. – Kevin May 12 '15 at 16:18
61

I ran this: strace -o spork.out bash -c "echo 1234 >> some-file" to figure out your question. This is what I found:

open("some-file", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3

No file named "some-file" existed in the directory in which I ran the echo command.

| improve this answer | |
51

This is not only done in Bash, it's required by the standard.

From the Single Unix Specification:

Appended output redirection shall cause the file whose name results from the expansion of word to be opened for output on the designated file descriptor. The file is opened as if the open() function as defined in the System Interfaces volume of POSIX.1-2008 was called with the O_APPEND flag. If the file does not exist, it shall be created.

Any POSIX-compliant shell therefore must do it. On some Unix systems, /bin/sh may be a non-POSIX Bourne shell (The Bourne shell was originally written before O_APPEND was invented), and the available POSIX shell will typically be ksh, which will be available as sh in a different path location such as Solaris's /usr/xpg4/bin.

| improve this answer | |
  • 2
    Interestingly, one shell that doesn't do it is the Bourne shell. The Bourne shell opens without O_TRUNC and lseek()s to the end. That would be because it was written before the O_APPEND flag was added to open(). >> itself was introduced by its predecessor the Thomson shell. – Stéphane Chazelas May 11 '15 at 21:00
  • 1
    @StéphaneChazelas Also, I looked up the C shell source for various versions, and the O_APPEND flag wasn't introduced until 4.3BSD-Reno. – Random832 May 11 '15 at 21:15
  • It says "as if", so couldn't it be implemented differently (but producing the same observable effect)? It doesn't seem like the standard requires the use of O_APPEND, just something that behaves "as if". – Thomas May 11 '15 at 23:36
  • 1
    @Thomas It does mean you're going to get all of the behavior documented for O_APPEND, meaning repositioning at the end for every write. The "as if" is just standards verbiage that is meant to allow for e.g. it to be opened by some means other than actually calling the open() function on non-traditional-Unix platforms. – Random832 May 11 '15 at 23:38
  • +1 , for showing that this behaviour is in the standards. – Prem May 12 '15 at 2:51
32

Looking in the source, it does use O_APPEND. For bash 4.3.30 in make_cmd.c line 710-713 read:

case r_appending_to:                /* >>foo */
case r_append_err_and_out:          /* &>> filename */
  temp->flags = O_APPEND | O_WRONLY | O_CREAT;
  break;
| improve this answer | |
  • +1 , for showing the answer from the source code perspective. – Prem May 12 '15 at 2:52
19

Let's investigate that using strace on a local (non-NFS) filesystem:

$ strace -eopen -- bash -c "echo foo >> /tmp/testfile000" 2>&1 | grep /tmp/testfile000
open("/tmp/testfile000", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3

$ strace -eopen -- bash -c "echo foo > /tmp/testfile000" 2>&1 | grep /tmp/testfile000
open("/tmp/testfile000", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3

Other shells, namely dash, dash, sh of busybox' and mksh behave the same way.

The option -e open means -e trace=open to trace only the open() system call.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.