2

Apart from possible races that have to be controlled by using proper synchronisation, it's possible in bash to feed a data source concurrently into multiple pipelines and collect all their outputs later into a common data sink.

For example, if you wanted to pre-process header and body of an email separately by different processes before sending it, you could do this as follows:

cat email.txt \
  | { tee >(sed -ne '1,/^$/p' | process_header >&3) \
  | sed -e '1,/^$/d' | process_body; } 3>&1 \
| sendmail -oi -- test@example.org

Given that, I was looking for a way to use the output of one of these pipelines to appear on the command line of one of the other pipelines or of the final data sink. The best I could achive so far was using a named pipe and xargs's -a option that allows for having two sources of input.

For example, to append -- automatically -- the number of lines in an email's body to the email's subject line, one could use:

cat email.txt \
  | { tee >(sed -ne '1,/^$/p' >&3; : branch off the header) \
  | sed -e '1,/^$/d' \
    | tee >(wc -l >~/.fifo); : number of body lines into pipe; } 3>&1 \
| xargs -I% -a ~/.fifo sed -e '1,/^$/{/^Subject:/Is/$/ (%)/}' \
| sendmail ...

(xargs -I% -a /dev/fd/4 4<~/.fifo ... also works, cf. below.) In this example the file ~/.fifo is a named pipe, created with mkfifo ~/.fifo.

But when i try to do this without a named pipe by using only file descriptors and redirection, e.g. analogous to the 1st example,

cat email.txt \
  | { tee >(sed -ne '1,/^$/p' >&3) \
  | sed -e '1,/^$/d' | tee >(wc -l >&4); } 3>&1 \
| xargs -I% -a /dev/fd/4 sed -e '1,/^$/{/^Subject:/Is/$/ (%)/}' \
| sendmail ...

this only results in an error:

xargs: Cannot open input file ‘/dev/fd/4’: No such file or directory
bash: 4: Bad file descriptor

[Update: Replacing the -a /dev/fd/4 with -a <(cat <&4) in the xargs call doesn't work as well; the complaint about the non-existent /dev/fd/4 is just replaced by another Bad file descriptor error. It seems to me that the fd 4 that is used for output (>&4) is not connected to the fd 4 that is used for input (<&4 resp. /dev/fd/4).]

Is there any way to get rid of the named pipe by some clever combination of redirection and process expansion? And, of course, without stating the data source more than once as in

nol="$(sed -e '1,/^$/d' email.txt | wc -l)"
sed -e "1,/^$/{/^Subject:/Is/$/ ($nol)/}" email.txt | sendmail ...
0

The errors in your command are because fd 4 is not open at all.

In fact you receive two "bad file descriptor" messages, one from the wc -l and the other from the cat <&4 (or the xargs -a /dev/fd/4).

You’d need an unnamed pipe to open fd 4 onto, but the only official way to have unnamed pipes in Bash is actually through the coproc command.

However for your specific use case there is a nice short-cut

A most compact way: “fake” unnamed pipes

This trick is undocumented as of Bash v5, but works on at least v4.3 (couldn’t test v5 yet).

It takes advantage of a few standard idioms that when put together allow for arbitrary “unnamed” pipes to be obtained on systems which support them. By “unnamed pipe” I mean “a FIFO not requiring to firstly create a type p file on the file-system through mkfifo or equivalent command”. (This definition for unnamed-pipe is not the correct one, but I daresay it is what really makes sense when using a command shell).

The example use case with these “unnamed” pipes boils down to the following:

cat email.txt | ( : {pipe}<> <(:) ; tee >(sed -e '1,/^$/d' | wc -l >&${pipe}) | xargs -I% -a <({ read count ; echo $count; } <&${pipe}) sed -e '1,/^$/{/^Subject:/Is/$/ (%)/}' )

The above command line should yield the intended result as per your example case.

Broken down for explanation: (only for clarity purposes, it cannot work when copied&pasted)

cat email.txt | \  # pipe data to ...
    ( \  # a compound statement, which ...
     : {pipe}<> <(:) ; \ # ... first opens the unnamed pipe in RW mode and put its fd into the (arbitrary) variable ${pipe}
    tee \ # then mirrors the data from main stdin to ...
        >( \ # the side processing of main input ...
            sed -e '1,/^$/d' | wc -l \ # ... which counts the body lines sending the result ...
            >&${pipe} \ # ... to the unnamed pipe
         ) \
    | \ # the tee also pipes all main input to ...
    xargs -I% -a \ # an xargs that reads iterative lines from ...
        <({ read count ; echo $count ; } <&${pipe}) \ # a compound command that reads the one-single line (being the count provided by wc) from ${pipe} fd, and echoes it back to xargs -a
        sed -e \ # that finally executes the sed command which looks for Subject: line in header part
        '/1,^$/{/^Subject:/Is/$/ (%)/}' ; \ # to append it with the count number
    )

A few additional notes:

  • opening the unnamed pipe RW is required as I found no way to rather open the usual pair of pipes being one the read-end and the other its write-end
  • this means there can’t be the usual EOF event notifying the read part that no more data will come, you have to do it your own in some other way but here we can take advantage of the fact that only one line is ever of interest and so just one single read suffices. If instead you’d need to read several lines from the side-channel then you’d need some kind of in-band EOF notification, like a simple EOF string to append at the very end of output and then filter out during reading for the xargs -a and bail-out from reading. This is perfectly doable but quite a longer typing for a command line. Getting rid of the in-band EOF string would also possible but even more complex
  • the management of these unnamed pipes are completely up to you, so you may need to close them explicitly by exec {pipe}<&- ; in this example I didn’t need to do it because the fd is created in a subprocess

For completeness, here is an equivalent version using coproc, which provides a real unnamed pipe through the usual couple of interconnected file-descriptors.

The official way for unnamed pipes: coproc

There can be quite a few approaches to using coproc, however for your case I suppose the best one would be as follows:

cat email.txt | (coproc cat ; : {input}<&${COPROC[0]} {output}>&${COPROC[1]} ; tee >(sed -e '1,/^$/d' | wc -l >&${output}) | xargs -I% -a <(exec cat <&${input}) sed -e '1,/^$/{/FOO/Is/$/ (%)/}' & )

Broken down for explanation: (only for clarity purposes, it cannot work when copied&pasted)

cat email.txt | \ # pipe data to ...
    ( \ # a subcommand statement, which ...
    coproc cat ; \  # ... first spawns the coprocess, a simple cat command acting as a simple line-oriented bridge
    : {cp_output}<&${COPROC[0]} {cp_input}>&${COPROC[1]} ; \ # then copies coproc’s own fds into new ones whose number are put into (arbitrary) variables ${cp_output} and ${cp_input} 
    tee \ # and then mirrors the data from main stdin to ...
        >( \ # the side processing of main input ...
            sed -e '1,/^$/d' | wc -l \ # ... which counts the body lines sending the result ...
            >&${output} \ # ... to the (bridging) coproc
          ) \
    | \ # the tee also pipes all main input to ...
    xargs -I% -a \ # an xargs that reads iterative lines from ...
        <(exec cat <&${pipe}) \ # another cat that reads from the coproc bridging the count provided by wc, and echoes it back to xargs -a
        sed -e \ # that finally executes the sed command which looks for Subject: line in header part
        '/1,^$/{/^Subject:/Is/$/ (%)/}' ; \ # to append it with the count number
    )

Again a few additional notes:

  • a subcommand statement is recommended so that no coproc’s data (ie process and fds) leaks to the interactive bash (assuming you run this beast interactively!)
  • else the management of this coproc’s data is completely up to you, so you may need eg to close the fds explicitly by exec {cp_input}<&- or exec {COPROC[1]}<&-
  • you can use any command with coproc but I always found that using a simple cat bridging the two fds makes a handy general purpose solution; however you can optimize towards performance if you manage to embed any one worker process into the coproc itself; in this example you’d need a lot of rearranging the whole command line
  • as per Bash v4 documentation, Bash supports only one coproc at a time
  • however, at least on v4.3 onwards it does accept more coprocs, though with an explicit warning, and Bash v5 docs does not state any limit
  • in case of more coprocs you have to use explicit names for each coproc (see the docs for details)
  • moving/copying coproc’s fds to arbitrary fds is required for them to survive the pipelines and process substitutions used in this example, because the ${COPROC[*]} array does not export to child processes and its own fds are always closed on exec
  • here we can take advantage of xargs -a that actively reads from both stdin and the file indicated in -a, thus not letting the tee fill the pipes’ buffers, otherwise there would be a deadlock and you’d need some more complex approach to avoid it
  • This is another interesting solution for the specific use-case of the example with the need for a fifo being replaced by the need for a final sorting step. Unfortunately my real, more complicated use-case actually requires that the result of the 1st child process somehow get into the command line of the 2nd. (It's about reformatting a table: one process returns a list of the longest content for each column (e.g. "12:8:23:5") and the other process needs this as an option (--colwidth=12:8:23:5) on its commandline to do the work.) – serolmy Mar 22 at 8:38
  • 1
    Oh I see, you’d like a sort of side-band channel. Then you may have a use of the (albeit undocumented) unnamed-pipe in bash. For your OP example you’d do something like this: cat email.txt | { : {pipe}<> <(:) ; tee >({ sed -e '1,/^$/d' | wc -l; echo EOF ; } >&${pipe}) | sed -e "/^Subject:/c\Subject: $(sed -ne '/^EOF$/q;p' <&${pipe})" ; }. If this achieves the intended result I will update my post with a full explanation – LL3 Mar 22 at 13:46
  • Wow. This really weird looking stuff works perfectly (well almost, in the example the number of lines is appended, but anyway) on the 1st try (with echo -e "From: foo\nTo: bar\nSubject: blah\n\nbody1\nbody2\nbody3\n\nbody4" instead of cat email.txt). – serolmy Mar 22 at 15:25
  • Right you are! Of course! :D I’ll put the corrected (ie appending) sed in my Answer (though I’m sure you corrected it yourself already). Good! – LL3 Mar 22 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.