Sometimes I want to insert something in a pipeline for reporting or some other secondary use. It might be as simple as
wc -l, or a more complex beast like
awk or even a python script. It'd be nice to run a pipeline like this:
zcat my_data_file.gz \ | wc -l > /tmp/linecount | process_data.py
The problem is that most utilities won't right the data to stdout.
tee can write the data to a temporary file, but then I have to wait until everything is done:
zcat my_data_file.gz \ | tee /tmp/f \ | process_data.py && \ wc -l /tmp/f > /tmp/linecount && rm /tmp/f
This is not optimal: it may be a very long-running pipeline; I may want to see intermediat results from the analogue to
wc sooner; and I may not want to store all the data in a temporary file.