How to emulate Process Substitution in Dash?

Question

In bash, I can use Process Substitution and treat output of a process as if it was a file saved on disk:

$ echo <(ls)
/dev/fd/63

$ ls -lAhF <(ls)
lr-x------ 1 root root 64 Sep 17 12:55 /dev/fd/63 -> pipe:[1652825]

unfortunately, Process Substitution is not supported in dash.

What would be the best way to emulate Process Substitution in dash?

I don't want to save the output as a temporary file somewhere (/tmp/) and then have to delete it. Is there an alternative way?

Answer 1

You can reproduce what the shell does under the hood by doing the plumbing manually. If your system has /dev/fd/NNN entries, you can use file descriptor shuffling: you can translate

main_command <(produce_arg1) <(produce_arg2) >(consume_arg3) >(consume_arg4)

to

{ produce_arg1 |
  { produce_arg2 |
    { main_command /dev/fd5 /dev/fd6 /dev/fd3 /dev/fd4 </dev/fd/8 >/dev/fd/9; } 5<&0 3>&1 |
    consume_arg3; } 6<&0 4>&1; |
  consume_arg4; } 8<&0 9>&1

I've shown a more complex example to illustrate multiple inputs and outputs. If you don't need to read from standard input, and the only reason you're using process substitution is that the command requires an explicit file name, you can simply use /dev/stdin:

main_command <(produce_arg1)
produce_arg1 | main_command /dev/stdin

Without /dev/fd/NNN, you need to use a named pipe. A named pipe is a directory entry, so you need to create a temporary file somewhere, but that file is just a name, it doesn't contain any data.

tmp=$(mktemp -d)
mkfifo "$tmp/f1" "$tmp/f2" "$tmp/f3" "$tmp/f4"
produce_arg1 >"$tmp/f1" &
produce_arg2 >"$tmp/f2" &
consume_arg3 <"$tmp/f3" &
consume_arg4 <"$tmp/f4" &
main_command "$tmp/f1" "$tmp/f2" "$tmp/f3" "$tmp/f4"
rm -r "$tmp"

Answer 2

The question in the current bounty notice:

the general example is too complicated. Can somebody please explain how to implement following example? diff <(cat "$2" | xz -d) <(cat "$1" | xz -d)

seems to have an answer here.

As shown in Gilles' answer, the general idea is to send the output of "producer" commands to new device files¹ at different stages of a pipeline, making them available to "consumer" commands, which may possibly take file names as arguments (assuming that your system gives you access to file descriptors as /dev/fd/X).

The simplest way to achieve what you are looking for is probably:

xz -cd file1.xz | { xz -cd file2.xz | diff /dev/fd/3 -; } 3<&0

(Using file1.xz in place of "$1" for readability and xz -cd instead of cat ... | xz -d because a single command is enough).

The output of the first "producer" command, xz -cd file1.xz, is piped to a compound command ({...}); but, instead of being consumed immediately as the standard input of the next command, it is duplicated to file descriptor 3 and thus made accessible to everything inside the compound command as /dev/fd/3. The output of the second "producer" command, xz -cd file2.xz, which does consume neither its standard input nor anything from file descriptor 3, is then piped to the "consumer" command diff, which reads from its standard input and from /dev/fd/3.

Piping and file descriptor duplication may be added in order to provide device files for as many "producer" commands as needed, e.g.:

xz -cd file1.xz | { xz -cd file2.xz | { diff /dev/fd/3 /dev/fd/4; } 4<&0; } 3<&0

While it may be irrelevant in the context of your specific question, it is worth noting that:

1. cmd1 <(cmd2) <(cmd3), cmd2 | { cmd3 | { cmd1 /dev/fd/3 /dev/fd/4; } 4<&0; } 3<&0 and ( cmd2 | ( cmd3 | ( cmd1 /dev/fd/3 /dev/fd/4 ) 4<&0 ) 3<&0 ) have different potential effects on the initial execution environment.

2. Contrary to what happens in cmd1 <(cmd2) <(cmd3), cmd3 and cmd1 in cmd2 | { cmd3 | { cmd1 /dev/fd/3 /dev/fd/4; } 4<&0; } 3<&0 will not be able to read any input from the user. That will require further file descriptors. For instance, to match

   diff <(echo foo) <(read var; echo "$var")
   

   you will need something like

   { echo foo | { read var 0<&9; echo "$var" | diff /dev/fd/3 -; } 3<&0; } 9<&0
   

---

¹ _{More on them can be found on U&L, e.g. in Understanding /dev and its subdirs and files.}

Answer 3

Can somebody please explain how to implement following example?
diff <(cat "$2" | xz -d) <(cat "$1" | xz -d)

I think named pipes are simpler to grok than redirections, so in simplest terms:

mkfifo p1 p2               # create pipes
cat "$2" | xz -d > p1 &    # start the commands in the background
cat "$1" | xz -d > p2 &    #    with output to the pipes
diff p1 p2                 # run 'diff', reading from the pipes
rm p1 p2                   # remove them at the end

p1 and p2 are temporary named pipes, they could be named anything.

It would be smarter to create the pipes in /tmp, e.g. in a directory as Gilles' answer shows, and note that you don't need cat here, so:

tmpdir=$(mktemp -d)
mkfifo "$tmpdir/p1" "$tmpdir/p2"
xz -d < "$2" > "$tmpdir/p1" &
xz -d < "$1" > "$tmpdir/p2" &
diff "$tmpdir/p1" "$tmpdir/p2"
rm -r "$tmpdir"

(You could probably get away without the quotes, too, since mktemp is likely to create "nice" filenames.)

The pipe solution does have the caveat that if the main command (diff) dies before reading all the input, the background processes can be left hanging.

Answer 4

What about:

cat "$2" | xz -d | diff /dev/sdtin /dev/stderr 2<<EOT
`cat "$1" | xz -d`
EOT