Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Sometimes I want to insert something in a pipeline for reporting or some other secondary use. It might be as simple as wc -l, or a more complex beast like awk or even a python script. It'd be nice to run a pipeline like this:

zcat my_data_file.gz \
| wc -l > /tmp/linecount
| process_data.py

The problem is that most utilities won't right the data to stdout. tee can write the data to a temporary file, but then I have to wait until everything is done:

zcat my_data_file.gz \
| tee /tmp/f \
| process_data.py && \
wc -l /tmp/f > /tmp/linecount && rm /tmp/f

This is not optimal: it may be a very long-running pipeline; I may want to see intermediat results from the analogue to wc sooner; and I may not want to store all the data in a temporary file.

share|improve this question
up vote 3 down vote accepted

You can use tee and process substitution >(...) for this:

zcat my_data_file.gz |

# Count number of lines in stream
tee >(wc -l > /tmp/linecount) |

# Further processing

Note that pipes can be used for line continuation and that comments may be interspersed between commands, a nice feature when building complicated pipelines.

share|improve this answer
I discovered this just this afternoon, by coincidence. Thanks! – Cera Apr 15 '13 at 12:27

It's not entirely efficient, but you can achieve this with named pipes, which you can create with mkififo(1)

For the example in the question:

mkfifo /tmp/f

wc -l /tmp/f > /tmp/linecount &

zcat my_data_file.gz \
| tee /tmp/f \
| process_data.py &


rm /tmp/f

Note the & appended to both wc and the pipeline; this means that the shell will push to the tasks to the background. The call to wait then waits for all background tasks to conclude. Both processes will finish at around the same time.

Beware that if one of your processes is substantially slower, it could slow the whole thing down significantly, as tee can block on either its stdout pipe or the named pipe that it is writing to. Edit: Also, it's now got more failure modes, as tee will exit due to a broken pipe if your secondary process fails.

share|improve this answer
If performance is crucial and the process you want to run is fairly trivial, you could reimplement (for example) a version of wc that prints its result to stderr, and ensures that every byte from stdin comes out of its stdout. Beware, though, that an architecturally more 'efficient' implementation may be slower anyway as coreutils and sed/awk are heavily optimised for many tasks. – Cera Apr 14 '13 at 5:54

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.