2

I am piping the output from multiple commands into the same file simultaneously using terminal like this:

cmd1 | tee -a /tmp/file
cmd2 | tee -a /tmp/file
cmd3 | tee -a /tmp/file

Is it safe to use? Any data loss or read/write permission problem with this method?

Edit: I am fine with output getting intermixed, I just want to make sure that everything get written to the file. What if two commands tries to write the output to the file exactly same time, Will it crash or not?

  • 1
    Assuming these commands run in parallel, it is not safe. – FedonKadifeli Sep 20 at 8:01
  • 1
    To prevent data loss you can use the "append" (-a) option of tee, but this will not guarantee that the outputs will not be garbled (=not intermixed) in the output file. – FedonKadifeli Sep 20 at 8:08
  • @FedonKadifeli , Yes these commands run in parallel. I missed to add -a option in the question. i have edited the question now, please check edit. – Sanjay Prajapat Sep 20 at 11:33
  • I think there is a possibility that you can lose data. See also this: stackoverflow.com/questions/7842511 – FedonKadifeli Sep 20 at 11:56
  • I would prefer something like: { cmd1 & cmd2 & cmd3 & } | tee /tmp/file. I think that this version will never lose data.. – FedonKadifeli Sep 20 at 13:36
4

If you use tee with append (-a) mode, then there's no risk of data loss, with the exception of filesystems that don't have an append operation (e.g., NFS). In append mode, tee opens the file with the O_APPEND flag.

$ echo 1 | strace tee -a /tmp/file 2>&1 | grep open | grep /tmp/file
openat(AT_FDCWD, "/tmp/file", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3

From man 2 openat:

O_APPEND
     The file is opened in append mode.  Before each write(2), the
     file offset is positioned at the end of the file, as if with
     lseek(2).  The modification of the file offset and the write
     operation are performed as a single atomic step.

The key sentence there is The modification of the file offset and the write operation are performed as a single atomic step. Every call to write() from whichever instance of tee is guaranteed to atomically position the file offset to the end of the file, then write the data.

The openat man page does note that this pattern of use is not safe on NFS filesystems (as I mentioned earlier):

     O_APPEND may lead to corrupted files on NFS filesystems if more
     than one process appends data to a file at once.  This is be‐
     cause  NFS does not support appending to a file, so the client
     kernel has to simulate it, which can't be done without a race
     condition.
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.