52

Some shells, like bash, support Process Substitution which is a way to present process output as a file, like this:

$ diff <(sort file1) <(sort file2)

However, this construct isn't POSIX and, therefore, not portable. How can process substitution be achieved in a POSIX-friendly manner (i.e. one which works in /bin/sh) ?

note: the question isn't asking how to diff two sorted files - that is only a contrived example to demonstrate process substitution!

3

3 Answers 3

34

That feature was introduced by ksh (first documented in ksh86) and was making use of the /dev/fd/n feature (added independently in some BSDs and AT&T systems earlier).  In ksh and up to ksh93u, it wouldn't work unless your system had support for /dev/fd/n. zsh, bash and ksh93u+ and above can make use of temporary named pipes (named pipes added in System III) where /dev/fd/n are not available.

On systems where /dev/fd/n is available (POSIX doesn't specify those), you can do process substitution (e.g., diff <(cmd1) <(cmd2)) yourself with:

{
  cmd1 4<&- | {
    # in here fd 3 points to the reading end of the pipe
    # from cmd1, while fd 0 has been restored from the original
    # stdin (saved on fd 4, now closed as no longer needed)

    cmd2 3<&- | diff /dev/fd/3 -

  } 3<&0 <&4 4<&- # restore the original stdin for cmd2

} 4<&0 # save a copy of stdin for cmd2

However that doesn't work with ksh93 on Linux as there, shell pipes are implemented with socketpairs instead of pipes and opening /dev/fd/3 where fd 3 points to a socket doesn't work on Linux.

Though POSIX doesn't specify /dev/fd/n, it does specify named pipes. Named pipes work like normal pipes except that you can access them from the file system. The issue here is that you have to create temporary ones and clean up afterwards, which is hard to do reliably, especially considering that POSIX has no standard mechanism (like a mktemp -d as found on some systems) to create temporary files or directories, and signal handling (to clean-up upon hang-up or kill) is also hard to do portably.

You could do something like:

tmpfifo() (
  n=0
  until
    fifo=$1.$$.$n
    mkfifo -m 600 -- "$fifo" 2> /dev/null
  do
    n=$((n + 1))
    # give up after 20 attempts as it could be a permanent condition
    # that prevents us from creating fifos. You'd need to raise that
    # limit if you intend to create (and use at the same time)
    # more than 20 fifos in your script
    [ "$n" -lt 20 ] || exit 1
  done
  printf '%s\n' "$fifo"
)

cleanup() { rm -f -- "$fifo"; }

fifo=$(tmpfifo /tmp/fifo) || exit

cmd2 > "$fifo" & cmd1 | diff - "$fifo"

cleanup

(not taking care of signal handling here).

5
  • The named pipe example is completely clear to me (I guess 10 is an arbitrary limit?) but I can't figure out your /dev/fd/n example. Why is descriptor 4 closed twice? (And so is descriptor 3.) I'm lost.
    – Wildcard
    Sep 13, 2016 at 23:47
  • @Wildcard, it's closed for commands that don't need it. Here, only diff needs that fd 3. None need the fd 4, it's only used to propagate the original stdin to cmd2 (dup2(0,4) on the outer {...}, restored with dup2(4,0) for the inside {...}). Sep 14, 2016 at 7:28
  • You could use mktemp -d to help ensure you can get FIFOs, as nobody should be writing to your brand-new random temporary directory.
    – Daniel H
    Jan 27, 2017 at 16:32
  • 2
    @DanielH, I already mention mktemp -d. But that's not a standard/POSIX command. Jan 27, 2017 at 17:18
  • Huh, I didn’t realize that. Oops. Quickly searching seems to show that most systems support it, so it might still be portable, but it isn’t POSIX.
    – Daniel H
    Feb 2, 2017 at 19:48
4

If needed to avoid lost variable in the useful cmd | while read A B C, instead of:

VAR="before"
while read A B C 
do
  VAR="$A $VAR"
done < <(cmd)
echo "$VAR"

you can use:

VAR="before"
while read A B C 
do
  VAR="$A $VAR"
done << EndOfText
`cmd`
EndOfText
echo "$VAR"

So to answer the question:

sort file1 | diff /dev/stdin /dev/stdout 2<<EOT
`sort file2`
EOT
0
4

You can rely on heredoc and /dev/fd/n files to achieve that. For example, in bash you can do this:

#!/usr/bin/env bash

paste  <(echo "$SHELL") <(echo "$TERM") <(echo "$LANG")

but in POSIX sh, you have to do this:

#!/bin/sh

paste /dev/fd/3 3<<-EOF /dev/fd/4 4<<-EOF /dev/fd/5 5<<-EOF 
$SHELL
EOF
$TERM
EOF
$LANG
EOF

The output is the same.

And to answer the question, this works:

#!/bin/sh

diff /dev/fd/3 3<<-EOF /dev/fd/4 4<<-EOF
$(sort file1)
EOF
$(sort file2)
EOF

It looks clunky, but it works.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.