11

This question already has an answer here:

In bash, I use arguments that look like

paste <(cat file1 | sort) <(cat file2 | sort)

or

comm <(cat file1 | sort) <(cat file2 | sort)

When I check man comm or man paste, the documentation says the args are indeed FILES.

Question:

  1. Are intermediate temporary files get created (on TEMP filesystem or elsewhere on slower disk) for <(cat file1 | sort) and <(cat file2 | sort)?

  2. What is the name for this <( ) magic? (to lookup its documentation)

  3. Is it specific to bash or does it work across other shells?

marked as duplicate by muru, Romeo Ninov, tripleee, Jeff Schaller, αғsнιη May 25 '18 at 15:57

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

18

This is called process substitution.

3.5.6 Process Substitution

Process substitution allows a process’s input or output to be referred to using a filename.

The process list is run asynchronously, and its input or output appears as a filename. This filename is passed as an argument to the current command as the result of the expansion. If the >(list) form is used, writing to the file will provide input for list. If the <(list) form is used, the file passed as an argument should be read to obtain the output of list. Note that no space may appear between the < or > and the left parenthesis, otherwise the construct would be interpreted as a redirection. Process substitution is supported on systems that support named pipes (FIFOs) or the /dev/fd method of naming open files.

It is not just a bash thing as it originally appeared in ksh but it's not in the posix standard.

Under the hood, process substitution has two implementations. On systems which support /dev/fd (most Unix-like systems) it works by calling the pipe() system call, which returns a file descriptor $fd for a new anonymous pipe, then creating the string /dev/fd/$fd, and substitutes that on the command line. On systems without /dev/fd support, it calls mkfifo with a new temporary filename to create a named pipe, and substitutes this filename on the command line.

  • 2
    This also appears in zsh. To my surprise, most other shells don't do it despite process substitution being quite useful under a variety of circumstances. – Kevin May 24 '18 at 20:13
7

You can think of <( somecommand ) as the filename of a temporary file containing the output of somecommand. In other words,

utility < <( somecommand )

is a shortcut for

somecommand >tempfile
utility <tempfile
rm -f tempfile

And

utility <( somecommand )

is a shortcut for

somecommand >tempfile
utility tempfile  # notice the lack of redirection here (utility expected to read from "tempfile")
rm -f tempfile

Likewise >( somecommand ) may be thought of as the filename of a temporary file that will be fed into somecommand on its standard input. In other words,

utility > >( somecommand )

is a shortcut for

utility >tempfile
somecommand <tempfile
rm -f tempfile

And

utility >( somecommand )

could possibly be a shortcut for

mkfifo temppipe
somecommand <temppipe &
utility temppipe  # utility is expected to write to "temppipe"
rm -f temppipe

(or something similar)

  • 2
    This is true as far as it goes, but you should be clear that pipes are used in all cases, not actual temporary files. This matters because it means that programs that try to seek around in the file won't work on process substitutions. – zwol May 25 '18 at 12:12
  • @zwol This is a good point, I will add something to this effect in a while (I'm busy right now). – Kusalananda May 25 '18 at 12:25

Not the answer you're looking for? Browse other questions tagged or ask your own question.