This question was motivated by "Reverse grepping", about grepping a huge file from bottom up.
tac file | grep whateverOr a bit more effective:
grep whatever < <(tac file)
The
< <(tac filename)should be as fast as a pipe
There are also many interesting comments from other users.
My questions:
| and < <()?xargs?The construction <(tac file) causes to shell to:
/dev/fd, a regular pipe is used, and /dev/fd/<the-file-descriptor-of-the-pipe> is used as the name.tac file and connect it to one end of the pipe.After the replacement, the command line becomes:
grep whatever < /tmp/whatever-name-the-shell-used-for-the-named-pipe
And then grep is executed, and it reads its standard input (which is the pipe), reads it, and searches for its first argument in that.
So the end result is the same as with...
tac file | grep whatever
...in that the same two programs are launched and a pipe is still used to connect them. But the <( ... ) construction is more convoluted because it involves more steps and may involve a temporary file (the named pipe).
The <( ... ) construct is an extension, and is not available in the standard POSIX bourne shell nor on platforms that do not support /dev/fd or named pipes. For this reason alone, because the two alternatives being considered are exactly equivalent in functionality, the more portable command | other-command form is a better choice.
The <( ... ) construction should be slower because of the additional convolution, but it's only in the startup phase and I don't expect the difference to be easily measurable.
NOTE: On Linux SysV platforms, < ( ... ) does not use named pipes but instead uses regular pipes. Regular pipes (indeed all file descriptors) can be referred to by the special named /dev/fd/<file-descriptor-number so that's what the shell uses as a name for the pipe. In this way it avoids creating a real named pipe with a bona fide temporary filename in the real filesystem. Although the /dev/fd trick is what was used to implement this feature when it originally appears in ksh, it is an optimization: on platforms that don't support this, a regular named pipe in the real filesystem is used as described above.
ALSO NOTE: To describe the syntax as <<( ... ) is misleading. In fact it's <( ... ), which is replaced with the name of a pipe, and then the other < character which prefixes the whole thing is separate from this syntax and it's the regular well-known syntax for redirecting input from a file.
bash and zsh added support for named pipes for systems that lacked /dev/fd/n later.
Commented
Jul 24, 2014 at 7:22
ksh feature, not a standard POSIX feature! Sure enough, dash doesn't appear to support it. That's one more reason to avoid it when there's an equivalent alternative available using basic POSIX features.
< <(command) is preferable in regard to the pipe |. As explained by gnouc, | requires each command to run in a subshell while process substitution leaves the pipe open and feeds the command from the stdout. There is more here: wiki.bash-hackers.org/syntax/expansion/proc_subst and mywiki.wooledge.org/ProcessSubstitution
Commented
Jul 24, 2014 at 9:41
command | command pipe construction is about as efficient as you can get. The shell uses fork() to create each new processes in a pipeline; but then fork() is universally used to create new processes — that's not considered a subshell.
What is difference between | and <<()?
There is a difference between them:
| cause each command run in a separated subshell.
<() run the command, which is substituted in background.
For the next two question, we will do some strace:
pipe:
$ strace -fc bash -c 'tac /usr/share/dict/american-english | grep qwerty'
$ time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00 0.008120 2707 3 1 wait4
0.00 0.000000 0 352 read
0.00 0.000000 0 229 write
0.00 0.000000 0 20 2 open
0.00 0.000000 0 29 2 close
0.00 0.000000 0 40 17 stat
0.00 0.000000 0 19 fstat
0.00 0.000000 0 117 lseek
0.00 0.000000 0 38 mmap
0.00 0.000000 0 18 mprotect
0.00 0.000000 0 6 munmap
0.00 0.000000 0 25 brk
0.00 0.000000 0 22 rt_sigaction
0.00 0.000000 0 18 rt_sigprocmask
0.00 0.000000 0 1 rt_sigreturn
0.00 0.000000 0 3 2 ioctl
0.00 0.000000 0 24 12 access
0.00 0.000000 0 1 pipe
0.00 0.000000 0 2 dup2
0.00 0.000000 0 1 getpid
0.00 0.000000 0 1 1 getpeername
0.00 0.000000 0 2 clone
0.00 0.000000 0 3 execve
0.00 0.000000 0 1 uname
0.00 0.000000 0 1 getrlimit
0.00 0.000000 0 13 getuid
0.00 0.000000 0 13 getgid
0.00 0.000000 0 13 geteuid
0.00 0.000000 0 13 getegid
0.00 0.000000 0 1 getppid
0.00 0.000000 0 1 getpgrp
0.00 0.000000 0 3 arch_prctl
0.00 0.000000 0 1 time
------ ----------- ----------- --------- --------- ----------------
100.00 0.008120 1034 37 total
Process Substitution:
$ strace -fc bash -c 'grep qwerty < <(tac /usr/share/dict/american-english)'
$ time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
99.14 0.016001 4000 4 2 wait4
0.46 0.000075 0 229 write
0.24 0.000038 0 341 read
0.16 0.000026 1 24 brk
0.00 0.000000 0 21 2 open
0.00 0.000000 0 27 close
0.00 0.000000 0 40 17 stat
0.00 0.000000 0 19 fstat
0.00 0.000000 0 117 lseek
0.00 0.000000 0 38 mmap
0.00 0.000000 0 18 mprotect
0.00 0.000000 0 6 munmap
0.00 0.000000 0 35 rt_sigaction
0.00 0.000000 0 24 rt_sigprocmask
0.00 0.000000 0 2 rt_sigreturn
0.00 0.000000 0 3 2 ioctl
0.00 0.000000 0 24 12 access
0.00 0.000000 0 1 pipe
0.00 0.000000 0 3 dup2
0.00 0.000000 0 1 getpid
0.00 0.000000 0 1 1 getpeername
0.00 0.000000 0 3 clone
0.00 0.000000 0 3 execve
0.00 0.000000 0 1 uname
0.00 0.000000 0 1 1 fcntl
0.00 0.000000 0 2 getrlimit
0.00 0.000000 0 13 getuid
0.00 0.000000 0 13 getgid
0.00 0.000000 0 13 geteuid
0.00 0.000000 0 13 getegid
0.00 0.000000 0 1 getppid
0.00 0.000000 0 1 getpgrp
0.00 0.000000 0 3 arch_prctl
0.00 0.000000 0 1 time
------ ----------- ----------- --------- --------- ----------------
100.00 0.016140 1046 37 total
Why something faster than other?
And what is really faster?
You can see, process substitution is slower than pipe in this case, because it use more system call. Both spend much time to wait for child processes, but process substitution use more wait4() syscall, and use more time for each call than pipe.
Why no one suggest xargs ?
I don't think xargs can help anything here, this is not its jobs.
Update
As @Gilles's suggested, I do a test with bigger file, 2GB random data generated from /dev/urandom. It shows that pipe is really faster than process substitution.
pipe:
$ strace -fc bash -c 'tac sample.txt | grep qwerty'
$ time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
81.15 8.284959 2761653 3 1 wait4
17.89 1.825959 2 780959 read
0.91 0.092708 0 524286 write
0.05 0.005364 0 262146 lseek
0.00 0.000000 0 20 2 open
0.00 0.000000 0 29 2 close
0.00 0.000000 0 40 17 stat
0.00 0.000000 0 19 fstat
0.00 0.000000 0 38 mmap
0.00 0.000000 0 18 mprotect
0.00 0.000000 0 6 munmap
0.00 0.000000 0 25 brk
0.00 0.000000 0 22 rt_sigaction
0.00 0.000000 0 18 rt_sigprocmask
0.00 0.000000 0 1 rt_sigreturn
0.00 0.000000 0 3 2 ioctl
0.00 0.000000 0 24 12 access
0.00 0.000000 0 1 pipe
0.00 0.000000 0 2 dup2
0.00 0.000000 0 1 getpid
0.00 0.000000 0 1 1 getpeername
0.00 0.000000 0 2 clone
0.00 0.000000 0 3 execve
0.00 0.000000 0 1 uname
0.00 0.000000 0 1 getrlimit
0.00 0.000000 0 13 getuid
0.00 0.000000 0 13 getgid
0.00 0.000000 0 13 geteuid
0.00 0.000000 0 13 getegid
0.00 0.000000 0 1 getppid
0.00 0.000000 0 1 getpgrp
0.00 0.000000 0 3 arch_prctl
0.00 0.000000 0 1 time
------ ----------- ----------- --------- --------- ----------------
100.00 10.208990 1567727 37 total
process substitution:
$ strace -fc bash -c 'grep qwerty < <(tac sample.txt)'
$ time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
99.51 13.912869 3478217 4 2 wait4
0.38 0.053373 0 655269 read
0.09 0.013084 0 524286 write
0.02 0.002454 0 262146 lseek
0.00 0.000030 1 38 mmap
0.00 0.000024 1 24 12 access
0.00 0.000000 0 21 2 open
0.00 0.000000 0 27 close
0.00 0.000000 0 40 17 stat
0.00 0.000000 0 19 fstat
0.00 0.000000 0 18 mprotect
0.00 0.000000 0 6 munmap
0.00 0.000000 0 24 brk
0.00 0.000000 0 35 rt_sigaction
0.00 0.000000 0 24 rt_sigprocmask
0.00 0.000000 0 2 rt_sigreturn
0.00 0.000000 0 3 2 ioctl
0.00 0.000000 0 1 pipe
0.00 0.000000 0 3 dup2
0.00 0.000000 0 1 getpid
0.00 0.000000 0 1 1 getpeername
0.00 0.000000 0 3 clone
0.00 0.000000 0 3 execve
0.00 0.000000 0 1 uname
0.00 0.000000 0 1 1 fcntl
0.00 0.000000 0 2 getrlimit
0.00 0.000000 0 13 getuid
0.00 0.000000 0 13 getgid
0.00 0.000000 0 13 geteuid
0.00 0.000000 0 13 getegid
0.00 0.000000 0 1 getppid
0.00 0.000000 0 1 getpgrp
0.00 0.000000 0 3 arch_prctl
0.00 0.000000 0 1 time
------ ----------- ----------- --------- --------- ----------------
100.00 13.981834 1442060 37 total
/dev/fd/n, it doesn't use named pipes (though on Linux, and linux only /dev/fd/n behave like named pipes when n is a file descriptor to a pipe (named or not)). On systems that don't support /dev/fd/n, some shells fall back to using named pipes.
Commented
Jul 24, 2014 at 7:19
ls -l <(:) to verify that it doesn't use named pipes.
Commented
Jul 24, 2014 at 7:36
mkfifo as described in the Wikipedia article. A named pipe is one of those "special" types of files you can sometimes see in the filesystem, like UNIX domain sockets, character devices, and block devices.
I was not able to replicate the results shown by cuonglm. Even with a 2GB file, in Bash 5 on MacOS Mojave, I see very similar timings between process substitution and pipe. This makes sense to me as the overhead involved in a call is going to be minimal compared to the actual processing of that call for a 2GB file, so running one iteration of using process substitution vs a pipe will be down to randomness / which command was run first to cache the file content.
I was able to replicate the findings in this question which show that process substitution is faster than pipes over several thousand instances of making these calls.
Here are the commands I ran, and the output:
pipe.sh:
shopt -s lastpipe
for i in {1..5000}; do
echo foo bar |
while read; do
echo $REPLY >/dev/null
done
done
proc-sub.sh:
for i in {1..5000}; do
while read; do
echo $REPLY >/dev/null
done < <(echo foo bar)
done
pipe-no-lastpipe.sh:
for i in {1..5000}; do
echo foo bar |
while read; do
echo $REPLY >/dev/null
done
done
Test:
time ./proc-sub.sh
real 0m9.505s
user 0m1.875s
sys 0m10.705s
time ./pipe.sh
real 0m14.036s
user 0m4.583s
sys 0m14.193s
time ./pipe-no-lastpipe.sh
real 0m16.696s
user 0m3.055s
sys 0m18.057s
< <(...)short for0< <(...), not<<(...).