18

In bash, I can use Process Substitution and treat output of a process as if it was a file saved on disk:

$ echo <(ls)
/dev/fd/63

$ ls -lAhF <(ls)
lr-x------ 1 root root 64 Sep 17 12:55 /dev/fd/63 -> pipe:[1652825]

unfortunately, Process Substitution is not supported in dash.

What would be the best way to emulate Process Substitution in dash?

I don't want to save the output as a temporary file somewhere (/tmp/) and then have to delete it. Is there an alternative way?

  • Depending on what you're trying to do you might be able to use pipes. – Eric Renouf Sep 17 '16 at 11:08
  • Honestly, if portability isn't your main concern here wouldn't it be easier to just install bash on your device? – Arkadiusz Drabczyk Aug 7 '19 at 10:27
  • 1
    The example you provide in the bounty notice happens to be the subject of this linked answer. As shown there, a simplified version of Gilles' answer (assuming the availability of /dev/fd and using xz -cd <file> instead of cat <file> | xz -d) could be xz -cd "$1" | { xz -cd "$2" | { diff /dev/fd/3 /dev/fd/4; } 3<&0; } 4<&0. – fra-san Aug 7 '19 at 12:26
  • 1
    @fra-san - that is actually what I needed. You can make it an answer if you want. Thanks. – Martin Vegter Aug 11 '19 at 17:06
4
+300

The question in the current bounty notice:

the general example is too complicated. Can somebody please explain how to implement following example? diff <(cat "$2" | xz -d) <(cat "$1" | xz -d)

seems to have an answer here.

As shown in Gilles' answer, the general idea is to send the output of "producer" commands to new device files1 at different stages of a pipeline, making them available to "consumer" commands, which may possibly take file names as arguments (assuming that your system gives you access to file descriptors as /dev/fd/X).

The simplest way to achieve what you are looking for is probably:

xz -cd file1.xz | { xz -cd file2.xz | diff /dev/fd/3 -; } 3<&0

(Using file1.xz in place of "$1" for readability and xz -cd instead of cat ... | xz -d because a single command is enough).

The output of the first "producer" command, xz -cd file1.xz, is piped to a compound command ({...}); but, instead of being consumed immediately as the standard input of the next command, it is duplicated to file descriptor 3 and thus made accessible to everything inside the compound command as /dev/fd/3. The output of the second "producer" command, xz -cd file2.xz, which does consume neither its standard input nor anything from file descriptor 3, is then piped to the "consumer" command diff, which reads from its standard input and from /dev/fd/3.

Piping and file descriptor duplication may be added in order to provide device files for as many "producer" commands as needed, e.g.:

xz -cd file1.xz | { xz -cd file2.xz | { diff /dev/fd/3 /dev/fd/4; } 4<&0; } 3<&0

While it may be irrelevant in the context of your specific question, it is worth noting that:

  1. cmd1 <(cmd2) <(cmd3), cmd2 | { cmd3 | { cmd1 /dev/fd/3 /dev/fd/4; } 4<&0; } 3<&0 and ( cmd2 | ( cmd3 | ( cmd1 /dev/fd/3 /dev/fd/4 ) 4<&0 ) 3<&0 ) have different potential effects on the initial execution environment.

  2. Contrary to what happens in cmd1 <(cmd2) <(cmd3), cmd3 and cmd1 in cmd2 | { cmd3 | { cmd1 /dev/fd/3 /dev/fd/4; } 4<&0; } 3<&0 will not be able to read any input from the user. That will require further file descriptors. For instance, to match

    diff <(echo foo) <(read var; echo "$var")
    

    you will need something like

    { echo foo | { read var 0<&9; echo "$var" | diff /dev/fd/3 -; } 3<&0; } 9<&0
    

1 More on them can be found on U&L, e.g. in Understanding /dev and its subdirs and files.

|improve this answer|||||
12

You can reproduce what the shell does under the hood by doing the plumbing manually. If your system has /dev/fd/NNN entries, you can use file descriptor shuffling: you can translate

main_command <(produce_arg1) <(produce_arg2) >(consume_arg3) >(consume_arg4)

to

{ produce_arg1 |
  { produce_arg2 |
    { main_command /dev/fd5 /dev/fd6 /dev/fd3 /dev/fd4 </dev/fd/8 >/dev/fd/9; } 5<&0 3>&1 |
    consume_arg3; } 6<&0 4>&1; |
  consume_arg4; } 8<&0 9>&1

I've shown a more complex example to illustrate multiple inputs and outputs. If you don't need to read from standard input, and the only reason you're using process substitution is that the command requires an explicit file name, you can simply use /dev/stdin:

main_command <(produce_arg1)
produce_arg1 | main_command /dev/stdin

Without /dev/fd/NNN, you need to use a named pipe. A named pipe is a directory entry, so you need to create a temporary file somewhere, but that file is just a name, it doesn't contain any data.

tmp=$(mktemp -d)
mkfifo "$tmp/f1" "$tmp/f2" "$tmp/f3" "$tmp/f4"
produce_arg1 >"$tmp/f1" &
produce_arg2 >"$tmp/f2" &
consume_arg3 <"$tmp/f3" &
consume_arg4 <"$tmp/f4" &
main_command "$tmp/f1" "$tmp/f2" "$tmp/f3" "$tmp/f4"
rm -r "$tmp"
|improve this answer|||||
  • 2
    </dev/fd/8 >/dev/fd/9 are not equivalent to <&8 >&9 on Linux and should be avoided there. – Stéphane Chazelas Apr 17 '19 at 12:25
  • @Gilles - I am confused by the complex example. Could you please show how to emulate: diff <(cat "$2" | xz -d) <(cat "$1" | xz -d) ? – Martin Vegter Aug 2 '19 at 5:07
3

Can somebody please explain how to implement following example?
diff <(cat "$2" | xz -d) <(cat "$1" | xz -d)

I think named pipes are simpler to grok than redirections, so in simplest terms:

mkfifo p1 p2               # create pipes
cat "$2" | xz -d > p1 &    # start the commands in the background
cat "$1" | xz -d > p2 &    #    with output to the pipes
diff p1 p2                 # run 'diff', reading from the pipes
rm p1 p2                   # remove them at the end

p1 and p2 are temporary named pipes, they could be named anything.

It would be smarter to create the pipes in /tmp, e.g. in a directory as Gilles' answer shows, and note that you don't need cat here, so:

tmpdir=$(mktemp -d)
mkfifo "$tmpdir/p1" "$tmpdir/p2"
xz -d < "$2" > "$tmpdir/p1" &
xz -d < "$1" > "$tmpdir/p2" &
diff "$tmpdir/p1" "$tmpdir/p2"
rm -r "$tmpdir"

(You could probably get away without the quotes, too, since mktemp is likely to create "nice" filenames.)

The pipe solution does have the caveat that if the main command (diff) dies before reading all the input, the background processes can be left hanging.

|improve this answer|||||
0

What about:

cat "$2" | xz -d | diff /dev/sdtin /dev/stderr 2<<EOT
`cat "$1" | xz -d`
EOT
|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.