4

Can stdout be overwritten by two bourne (or bash, if that matters) subshell commands running asynchronously?

(tail -f ./file1 & tail -f ./file2) | cat

I don’t care about the line order, just that every output line is made up of exactly one input line. I’m worried that some lines could be partially overwritten or interleaved.

I’ve tested this by running four commands that each output a unique line 15 million times each. It appears to work, but I kind of expected it to fail.

Can someone explain how this doesn’t break? Is each subshell buffered and only one subshell gets to write to stdout at a time? or how is this managed.

Is there a better way to do this?

(Never mind that I’m using tail in the above subshells for illustrative purposes. I actually want to run two other commands that output continuously one line at the time to stdout.)

5

The shells have little involvement there. All they do is create the pipe and start those 3 commands which then run in parallel independently from the shell.

What matters here is that both tail commands write to a file descriptor to the same writing end of the same pipe.

If you do:

printf foo1 >> file1; sleep 1
printf foo2 >> file2; sleep 1
printf 'bar1\n' >> file1; sleep 1
printf 'bar2\n' >> file2

You'll see:

foo1foo2bar1
bar2

Because that's how those were written. You'll want to make sure your commands output one full line at a time, and that those lines be smaller than PIPE_BUF (4096 bytes on Linux) for the write() to be guaranteed to be atomic (it could also write more than one full line at a time provided they're all full and their cumulative size is less than PIPE_BUF).

With GNU grep, you can do that by piping your commands to grep --line-buffered '^'

(tail -f ./file1 | grep --line-buffered '^' &
 tail -f ./file2 | grep --line-buffered '^') | cat

That would guarantee that you get one write() system call for each line of the output of both commands (In the cases where the commands don't terminate their last line of output, grep would add the missing newline)

0

1. Bad solution

In the default configuration, stderr is not buffered, but stdout is.

Thus, the simplest solution of your problem is

  1. provide that the tools used write everything line-by-line
  2. redirect their output to the stderr (>&2)

However, it doesn't work, because this buffering happens in the C library, inside the process. If you redirect their stdout to the stderr, also this stderr will be unbuffered.

2. Better solution

Pipe their output into process what reads everything in, and then writes them out by line. A most easy way would be

tool1 | while read; do echo "$REPLY"; done & tool2 | while read; do echo "$REPLY"; done

For the "beautiful" parallel executions of multiple commands/scripts, here you can read my other answer.

3. The real solution

Unfortunately, how the process outputs are mainly buffered by libc, i.e. it is their internal things, how they map their outputs to write(1, ...) calls on the kernel level. It depends on them. Without changing them, we don't have control over their output what they didn't write yet.

In the case if the writing happens with the FILE* mechanism of the libc, the setbuf (3) and fflush (3) functions may be your friend.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.