Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

This is what I usually do to run grep and wc on a file without having to scan it twice

<file.txt  tee >(grep LITERAL) >(wc -l) >/dev/null

However, this yields

EXEC LITERAL
32

sometimes and

32
EXEC LITERAL

at other times. (The output from grep precedes the output from wc in the first instance and vice versa in the second.)

On the other hand, with redirections and file descriptors

{ { <file.txt tee /dev/fd/3 | grep LITERAL >&4; } 3>&1 | wc -l ;} 4>&1 

I always seem to get

EXEC LITERAL
32

I prefer that the output order be predictable but is it guaranteed with the second approach?

share|improve this question
add comment

2 Answers 2

up vote 4 down vote accepted

In both

<file.txt  tee >(grep LITERAL) >(wc -l) >/dev/null

And:

{ { <file.txt tee /dev/fd/3 | grep LITERAL >&4; } 3>&1 | wc -l ;} 4>&1

All of tee, grep and wc are started concurrently. What matters then is what happens at the end.

wc will only print the result when it sees end-of-file on its standard input. In the first case, that's when tee exits, because then tee will close its fd on the other end of the pipe that wc is reading from (started by process substitution). There's no guarantee that grep will have read all its input by that time, let alone written its output (given that pipes can hold quite a large amount of data and that wc will likely be faster than grep)

In the second case, wc will see end-of-file when all the writers to the pipe it is reading from have closed their end of the pipe. In that case though, there are several writers. tee (via its fd open on /dev/fd/3 and via its fd 3) and grep which also has its fd 3 open to the pipe to wc (though it is not making any use of it, let alone write to it). The inner { will likely cause an extra subshell process that will also have a fd 3 open and will wait for both tee and grep.

That means that wc will only write its line number after grep has exited.

Had you written it the proper way, that is by closing the fds that didn't need open:

{ { <file.txt tee /dev/fd/3 4>&- | 
   grep LITERAL >&4 3>&- 4>&-; } 3>&1 | wc -l 4>&-;} 4>&1

Then the order would not have been guaranteed in shells that optimise out the subshell process. However, the only shell that I know that does is ksh93 but ksh93 uses socket pairs for pipes, so /dev/fd/3 won't work there on Linux at least.

To see what processes are running, you can replace grep with ps:

$ { { <file.txt tee /dev/fd/3 4>&- | ps -H >&4 3>&- 4>&-; } 3>&1 | wc -l 4>&-;} 4>&1
  PID TTY          TIME CMD
 8727 pts/5    00:00:00 bash
 8815 pts/5    00:00:00   bash
 8817 pts/5    00:00:00     tee
 8818 pts/5    00:00:00     ps
 8816 pts/5    00:00:00   wc

With bash, you can see that extra shell process, and you can see it also has the pipe opened on fd 3 with:

$ (p=$BASHPID; { { <file.txt tee /dev/fd/3 4>&- | lsof -ag "$p" -d3 >&4 3>&- 4>&-; } 3>&1 | wc -l 4>&-;} 4>&1)
COMMAND  PID PGID     USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
bash    9843 9842 chazelas    3w  FIFO    0,8      0t0 153304 pipe
tee     9845 9842 chazelas    3w  FIFO    0,8      0t0 153304 pipe
lsof    9846 9842 chazelas    3r   DIR    0,3        0      1 /proc
share|improve this answer
    
Thanks. In your "proper example", what does grep LITERAL >&4 3>&- 4>&- mean, the fd 4 appears to be both used and closed? –  1_CR Dec 20 '13 at 2:59
    
@1_CR, after >&4, short for 1>&4, grep's fd 1 and 4 point to the same resource (the shell's initial stdout). grep does not need to have its fd 4 open to anything. It does not do anything with it, so we close it with 4>&- –  Stéphane Chazelas Dec 20 '13 at 8:10
    
That last command-line is cryptic magic. –  Herman Torjussen Dec 20 '13 at 13:02
    
Great answer as always. Accepted. –  1_CR Dec 20 '13 at 20:24
add comment

To get a predictable order use

(<file.txt  tee >(grep LITERAL) >(wc -l) >/dev/null)|sort
share|improve this answer
    
Perhaps i wasn't clear enough. I meant predictable order in terms of order of command outputs (ie output from grep before output from wc). I don't need the combined output sorted –  1_CR Dec 19 '13 at 16:47
    
just found gnu.org/software/bash/manual/bashref.html#Command-Grouping, it tells me that with the {} operators you make sure (in this case) that you first do <file.txt tee /dev/fd/3 | grep LITERAL >&4; and when this is done you call wc, so to answer your original question, yes it is guaranteed to my understanding –  Thorsten Staerk Dec 19 '13 at 16:49
1  
@ThorstenStaerk could you add the extra info you found to your answer? –  terdon Dec 19 '13 at 16:52
    
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.