Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

This question was motivated by "Reverse grepping", about grepping a huge file from bottom up.

@chaos said:

tac file | grep whatever

Or a bit more effective:

grep whatever < <(tac file)

@vinc17 said:

The < <(tac filename) should be as fast as a pipe

There are also many interesting comments from other users.

My questions:

  • What is the difference between | and < <()?
  • Why is one faster than other?
  • And which is really faster?
  • Why did no one suggest xargs?
share|improve this question
1  
Note that it's < <(...) short for 0< <(...), not <<(...). –  Stéphane Chazelas Jul 24 at 7:23
1  
@Anthon Thanks for fixing "greping" to "grepping" - I came just back form english.SE to fix it; I got really curious how it's right, and why. Learned a lot about English in a short time. See 6 answers at Deriving a word for the activity of using a tool from the tool name (“grep”) –  Volker Siegel Jul 24 at 18:11
    
@VolkerSiegel I just looked at the title of the linked question, for consistent spelling ;-) –  Anthon Jul 24 at 18:25
    
I knew it was inconsistent - but I did not know who's right ;) I strongly preferred "greping", because the additional "p" in "grepping" somehow becomes part of the command name - which felt "very wrong" to me - just made no sense at all... But it turned out that the guys over at english.SE made me accept "grepping" as right in about an hour and 6 answers :) –  Volker Siegel Jul 24 at 18:56
    
@StéphaneChazelas Edited the example to match. –  Volker Siegel Jul 24 at 18:58

2 Answers 2

up vote 10 down vote accepted

The construction <(tac file) causes to shell to:

  • Create a pipe with a name
    • On systems such as Linux and SysV which have /dev/fd, a regular pipe is used, and /dev/fd/<the-file-descriptor-of-the-pipe> is used as the name.
    • On other systems, a named pipe is used, which requires creating an actual file entry on disk.
  • Launch the command tac file and connect it to one end of the pipe.
  • Replace the whole construction on the command line with the name of the pipe.

After the replacement, the command line becomes:

grep whatever < /tmp/whatever-name-the-shell-used-for-the-named-pipe

And then grep is executed, and it reads its standard input (which is the pipe), reads it, and searches for its first argument in that.

So the end result is the same as with...

tac file | grep whatever

...in that the same two programs are launched and a pipe is still used to connect them. But the <( ... ) construction is more convoluted because it involves more steps and may involve a temporary file (the named pipe).

The <( ... ) construct is an extension, and is not available in the standard POSIX bourne shell nor on platforms that do not support /dev/fd or named pipes. For this reason alone, because the two alternatives being considered are exactly equivalent in functionality, the more portable command | other-command form is a better choice.

The <( ... ) construction should be slower because of the additional convolution, but it's only in the startup phase and I don't expect the difference to be easily measurable.

NOTE: On Linux SysV platforms, < ( ... ) does not use named pipes but instead uses regular pipes. Regular pipes (indeed all file descriptors) can be referred to by the special named /dev/fd/<file-descriptor-number so that's what the shell uses as a name for the pipe. In this way it avoids creating a real named pipe with a bona fide temporary filename in the real filesystem. Although the /dev/fd trick is what was used to implement this feature when it originally appears in ksh, it is an optimization: on platforms that don't support this, a regular named pipe in the real filesystem is used as described above.

ALSO NOTE: To describe the syntax as <<( ... ) is misleading. In fact it's <( ... ), which is replaced with the name of a pipe, and then the other < character which prefixes the whole thing is separate from this syntax and it's the regular well-known syntax for redirecting input from a file.

share|improve this answer
1  
process substitution appeared on ksh and was using /dev/fd/n (a SysV feature) from the start (and was not available on systems that didn't support it). bash and zsh added support for named pipes for systems that lacked /dev/fd/n later. –  Stéphane Chazelas Jul 24 at 7:22
    
@StéphaneChazelas ah, that is interesting history, thanks. So it's originally a ksh feature, not a standard POSIX feature! Sure enough, dash doesn't appear to support it. That's one more reason to avoid it when there's an equivalent alternative available using basic POSIX features. –  Celada Jul 24 at 7:37
    
It's not only SysV (SysV generally referring to a reference implementation and also to a family of OSes) and Linux. Solaris, all modern BSDs and probably many others support /dev/fd/n. (actually, the Linux one is significantly different from that of others as it's implemented as symlinks to original resources while in other Unices /dev/fd/n are special devices which when open, work like dup (which is why I said on Linux, and Linux only, /dev/fd/n on a pipe acts as a named pipe)). –  Stéphane Chazelas Jul 24 at 9:28
1  
I think process substitution < <(command) is preferable in regard to the pipe |. As explained by gnouc, | requires each command to run in a subshell while process substitution leaves the pipe open and feeds the command from the stdout. There is more here: wiki.bash-hackers.org/syntax/expansion/proc_subst and mywiki.wooledge.org/ProcessSubstitution –  val0x00ff Jul 24 at 9:41
    
@val0x00ff The command | command pipe construction is about as efficient as you can get. The shell uses fork() to create each new processes in a pipeline; but then fork() is universally used to create new processes — that's not considered a subshell. –  Celada Jul 24 at 13:23

What is difference between | and <<()?

There is a difference between them:

  • | cause each command run in a separated subshell.

  • <() run the command, which is substituted in background.

For the next two question, we will do some strace:

pipe:

$ strace -fc bash -c 'tac /usr/share/dict/american-english | grep qwerty'
$ time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.008120        2707         3         1 wait4
  0.00    0.000000           0       352           read
  0.00    0.000000           0       229           write
  0.00    0.000000           0        20         2 open
  0.00    0.000000           0        29         2 close
  0.00    0.000000           0        40        17 stat
  0.00    0.000000           0        19           fstat
  0.00    0.000000           0       117           lseek
  0.00    0.000000           0        38           mmap
  0.00    0.000000           0        18           mprotect
  0.00    0.000000           0         6           munmap
  0.00    0.000000           0        25           brk
  0.00    0.000000           0        22           rt_sigaction
  0.00    0.000000           0        18           rt_sigprocmask
  0.00    0.000000           0         1           rt_sigreturn
  0.00    0.000000           0         3         2 ioctl
  0.00    0.000000           0        24        12 access
  0.00    0.000000           0         1           pipe
  0.00    0.000000           0         2           dup2
  0.00    0.000000           0         1           getpid
  0.00    0.000000           0         1         1 getpeername
  0.00    0.000000           0         2           clone
  0.00    0.000000           0         3           execve
  0.00    0.000000           0         1           uname
  0.00    0.000000           0         1           getrlimit
  0.00    0.000000           0        13           getuid
  0.00    0.000000           0        13           getgid
  0.00    0.000000           0        13           geteuid
  0.00    0.000000           0        13           getegid
  0.00    0.000000           0         1           getppid
  0.00    0.000000           0         1           getpgrp
  0.00    0.000000           0         3           arch_prctl
  0.00    0.000000           0         1           time
------ ----------- ----------- --------- --------- ----------------
100.00    0.008120                  1034        37 total

Process Substitution:

$ strace -fc bash -c 'grep qwerty < <(tac /usr/share/dict/american-english)'
$ time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 99.14    0.016001        4000         4         2 wait4
  0.46    0.000075           0       229           write
  0.24    0.000038           0       341           read
  0.16    0.000026           1        24           brk
  0.00    0.000000           0        21         2 open
  0.00    0.000000           0        27           close
  0.00    0.000000           0        40        17 stat
  0.00    0.000000           0        19           fstat
  0.00    0.000000           0       117           lseek
  0.00    0.000000           0        38           mmap
  0.00    0.000000           0        18           mprotect
  0.00    0.000000           0         6           munmap
  0.00    0.000000           0        35           rt_sigaction
  0.00    0.000000           0        24           rt_sigprocmask
  0.00    0.000000           0         2           rt_sigreturn
  0.00    0.000000           0         3         2 ioctl
  0.00    0.000000           0        24        12 access
  0.00    0.000000           0         1           pipe
  0.00    0.000000           0         3           dup2
  0.00    0.000000           0         1           getpid
  0.00    0.000000           0         1         1 getpeername
  0.00    0.000000           0         3           clone
  0.00    0.000000           0         3           execve
  0.00    0.000000           0         1           uname
  0.00    0.000000           0         1         1 fcntl
  0.00    0.000000           0         2           getrlimit
  0.00    0.000000           0        13           getuid
  0.00    0.000000           0        13           getgid
  0.00    0.000000           0        13           geteuid
  0.00    0.000000           0        13           getegid
  0.00    0.000000           0         1           getppid
  0.00    0.000000           0         1           getpgrp
  0.00    0.000000           0         3           arch_prctl
  0.00    0.000000           0         1           time
------ ----------- ----------- --------- --------- ----------------
100.00    0.016140                  1046        37 total

Why something faster than other?

And what is really faster?

You can see, process substitution is slower than pipe in this case, because it use more system call. Both spend much time to wait for child processes, but process substitution use more wait4() syscall, and use more time for each call than pipe.

Why no one suggest xargs ?

I don't think xargs can help anything here, this is not its jobs.

Update

As @Gilles's suggested, I do a test with bigger file, 2GB random data generated from /dev/urandom. It shows that pipe is really faster than process substitution.

pipe:

$ strace -fc bash -c 'tac sample.txt | grep qwerty'
$ time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 81.15    8.284959     2761653         3         1 wait4
 17.89    1.825959           2    780959           read
  0.91    0.092708           0    524286           write
  0.05    0.005364           0    262146           lseek
  0.00    0.000000           0        20         2 open
  0.00    0.000000           0        29         2 close
  0.00    0.000000           0        40        17 stat
  0.00    0.000000           0        19           fstat
  0.00    0.000000           0        38           mmap
  0.00    0.000000           0        18           mprotect
  0.00    0.000000           0         6           munmap
  0.00    0.000000           0        25           brk
  0.00    0.000000           0        22           rt_sigaction
  0.00    0.000000           0        18           rt_sigprocmask
  0.00    0.000000           0         1           rt_sigreturn
  0.00    0.000000           0         3         2 ioctl
  0.00    0.000000           0        24        12 access
  0.00    0.000000           0         1           pipe
  0.00    0.000000           0         2           dup2
  0.00    0.000000           0         1           getpid
  0.00    0.000000           0         1         1 getpeername
  0.00    0.000000           0         2           clone
  0.00    0.000000           0         3           execve
  0.00    0.000000           0         1           uname
  0.00    0.000000           0         1           getrlimit
  0.00    0.000000           0        13           getuid
  0.00    0.000000           0        13           getgid
  0.00    0.000000           0        13           geteuid
  0.00    0.000000           0        13           getegid
  0.00    0.000000           0         1           getppid
  0.00    0.000000           0         1           getpgrp
  0.00    0.000000           0         3           arch_prctl
  0.00    0.000000           0         1           time
------ ----------- ----------- --------- --------- ----------------
100.00   10.208990               1567727        37 total

process substitution:

$ strace -fc bash -c 'grep qwerty < <(tac sample.txt)'
$ time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 99.51   13.912869     3478217         4         2 wait4
  0.38    0.053373           0    655269           read
  0.09    0.013084           0    524286           write
  0.02    0.002454           0    262146           lseek
  0.00    0.000030           1        38           mmap
  0.00    0.000024           1        24        12 access
  0.00    0.000000           0        21         2 open
  0.00    0.000000           0        27           close
  0.00    0.000000           0        40        17 stat
  0.00    0.000000           0        19           fstat
  0.00    0.000000           0        18           mprotect
  0.00    0.000000           0         6           munmap
  0.00    0.000000           0        24           brk
  0.00    0.000000           0        35           rt_sigaction
  0.00    0.000000           0        24           rt_sigprocmask
  0.00    0.000000           0         2           rt_sigreturn
  0.00    0.000000           0         3         2 ioctl
  0.00    0.000000           0         1           pipe
  0.00    0.000000           0         3           dup2
  0.00    0.000000           0         1           getpid
  0.00    0.000000           0         1         1 getpeername
  0.00    0.000000           0         3           clone
  0.00    0.000000           0         3           execve
  0.00    0.000000           0         1           uname
  0.00    0.000000           0         1         1 fcntl
  0.00    0.000000           0         2           getrlimit
  0.00    0.000000           0        13           getuid
  0.00    0.000000           0        13           getgid
  0.00    0.000000           0        13           geteuid
  0.00    0.000000           0        13           getegid
  0.00    0.000000           0         1           getppid
  0.00    0.000000           0         1           getpgrp
  0.00    0.000000           0         3           arch_prctl
  0.00    0.000000           0         1           time
------ ----------- ----------- --------- --------- ----------------
100.00   13.981834               1442060        37 total
share|improve this answer
1  
process substitution leverages /dev/fd/n, it doesn't use named pipes (though on Linux, and linux only /dev/fd/n behave like named pipes when n is a file descriptor to a pipe (named or not)). On systems that don't support /dev/fd/n, some shells fall back to using named pipes. –  Stéphane Chazelas Jul 24 at 7:19
    
Added a note. But it seems that process subsitution is only support when system supports named pipe: gnu.org/software/bash/manual/html_node/…. And from this: en.wikipedia.org/wiki/Process_substitution, it said process substitution use named pipe? –  cuonglm Jul 24 at 7:26
    
Well, wikipedia is wrong here (and bash-centric). Simply run ls -l <(:) to verify that it doesn't use named pipes. –  Stéphane Chazelas Jul 24 at 7:36
1  
@Gnouc a named pipe is a pipe that is connected to a name in the filesystem. It is created using mkfifo as described in the Wikipedia article. A named pipe is one of those "special" types of files you can sometimes see in the filesystem, like UNIX domain sockets, character devices, and block devices. –  Celada Jul 24 at 7:50
1  
When running the strace command several times, I notice that the output changes a bit, and that both solutions are rather equivalent. This becomes even more visible on a big file (several MB's). –  vinc17 Jul 24 at 9:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.