11

This is what I usually do to run grep and wc on a file without having to scan it twice

<file.txt  tee >(grep LITERAL) >(wc -l) >/dev/null

However, this yields

EXEC LITERAL
32

sometimes and

32
EXEC LITERAL

at other times. (The output from grep precedes the output from wc in the first instance and vice versa in the second.)

On the other hand, with redirections and file descriptors

{ { <file.txt tee /dev/fd/3 | grep LITERAL >&4; } 3>&1 | wc -l ;} 4>&1 

I always seem to get

EXEC LITERAL
32

I prefer that the output order be predictable but is it guaranteed with the second approach?

4

In both

<file.txt  tee >(grep LITERAL) >(wc -l) >/dev/null

And:

{ { <file.txt tee /dev/fd/3 | grep LITERAL >&4; } 3>&1 | wc -l ;} 4>&1

All of tee, grep and wc are started concurrently. What matters then is what happens at the end.

wc will only print the result when it sees end-of-file on its standard input. In the first case, that's when tee exits, because then tee will close its fd on the other end of the pipe that wc is reading from (started by process substitution). There's no guarantee that grep will have read all its input by that time, let alone written its output (given that pipes can hold quite a large amount of data and that wc will likely be faster than grep)

In the second case, wc will see end-of-file when all the writers to the pipe it is reading from have closed their end of the pipe. In that case though, there are several writers. tee (via its fd open on /dev/fd/3 and via its fd 3) and grep which also has its fd 3 open to the pipe to wc (though it is not making any use of it, let alone write to it). The inner { will likely cause an extra subshell process that will also have a fd 3 open and will wait for both tee and grep.

That means that wc will only write its line number after grep has exited.

Had you written it the proper way, that is by closing the fds that didn't need open:

{ { <file.txt tee /dev/fd/3 4>&- | 
   grep LITERAL >&4 3>&- 4>&-; } 3>&1 | wc -l 4>&-;} 4>&1

Then the order would not have been guaranteed in shells that optimise out the subshell process. However, the only shell that I know that does is ksh93 but ksh93 uses socket pairs for pipes, so /dev/fd/3 won't work there on Linux at least.

To see what processes are running, you can replace grep with ps:

$ { { <file.txt tee /dev/fd/3 4>&- | ps -H >&4 3>&- 4>&-; } 3>&1 | wc -l 4>&-;} 4>&1
  PID TTY          TIME CMD
 8727 pts/5    00:00:00 bash
 8815 pts/5    00:00:00   bash
 8817 pts/5    00:00:00     tee
 8818 pts/5    00:00:00     ps
 8816 pts/5    00:00:00   wc

With bash, you can see that extra shell process, and you can see it also has the pipe opened on fd 3 with:

$ (p=$BASHPID; { { <file.txt tee /dev/fd/3 4>&- | lsof -ag "$p" -d3 >&4 3>&- 4>&-; } 3>&1 | wc -l 4>&-;} 4>&1)
COMMAND  PID PGID     USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
bash    9843 9842 chazelas    3w  FIFO    0,8      0t0 153304 pipe
tee     9845 9842 chazelas    3w  FIFO    0,8      0t0 153304 pipe
lsof    9846 9842 chazelas    3r   DIR    0,3        0      1 /proc
  • Thanks. In your "proper example", what does grep LITERAL >&4 3>&- 4>&- mean, the fd 4 appears to be both used and closed? – iruvar Dec 20 '13 at 2:59
  • @1_CR, after >&4, short for 1>&4, grep's fd 1 and 4 point to the same resource (the shell's initial stdout). grep does not need to have its fd 4 open to anything. It does not do anything with it, so we close it with 4>&- – Stéphane Chazelas Dec 20 '13 at 8:10
  • That last command-line is cryptic magic. – user13742 Dec 20 '13 at 13:02
-1

To get a predictable order use

(<file.txt  tee >(grep LITERAL) >(wc -l) >/dev/null)|sort

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.