Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Sometimes I want to insert something in a pipeline for reporting or some other secondary use. It might be as simple as wc -l, or a more complex beast like awk or even a python script. It'd be nice to run a pipeline like this:

zcat my_data_file.gz \
| wc -l > /tmp/linecount
| process_data.py

The problem is that most utilities won't right the data to stdout. tee can write the data to a temporary file, but then I have to wait until everything is done:

zcat my_data_file.gz \
| tee /tmp/f \
| process_data.py && \
wc -l /tmp/f > /tmp/linecount && rm /tmp/f

This is not optimal: it may be a very long-running pipeline; I may want to see intermediat results from the analogue to wc sooner; and I may not want to store all the data in a temporary file.

share|improve this question
add comment

2 Answers 2

up vote 2 down vote accepted

You can use tee and process substitution >(...) for this:

zcat my_data_file.gz |

# Count number of lines in stream
tee >(wc -l > /tmp/linecount) |

# Further processing
process_data.py

Note that pipes can be used for line continuation and that comments may be interspersed between commands, a nice feature when building complicated pipelines.

share|improve this answer
    
I discovered this just this afternoon, by coincidence. Thanks! –  Cerales Apr 15 '13 at 12:27
add comment

It's not entirely efficient, but you can achieve this with named pipes, which you can create with mkififo(1)

For the example in the question:

mkfifo /tmp/f

wc -l /tmp/f > /tmp/linecount &

zcat my_data_file.gz \
| tee /tmp/f \
| process_data.py &

wait

rm /tmp/f

Note the & appended to both wc and the pipeline; this means that the shell will push to the tasks to the background. The call to wait then waits for all background tasks to conclude. Both processes will finish at around the same time.

Beware that if one of your processes is substantially slower, it could slow the whole thing down significantly, as tee can block on either its stdout pipe or the named pipe that it is writing to. Edit: Also, it's now got more failure modes, as tee will exit due to a broken pipe if your secondary process fails.

share|improve this answer
    
If performance is crucial and the process you want to run is fairly trivial, you could reimplement (for example) a version of wc that prints its result to stderr, and ensures that every byte from stdin comes out of its stdout. Beware, though, that an architecturally more 'efficient' implementation may be slower anyway as coreutils and sed/awk are heavily optimised for many tasks. –  Cerales Apr 14 '13 at 5:54
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.