Process substitution with tee and paste

Question

I'm trying to pipe a command through the output of two other commands and then merge the results of the two process substitutions. An example that gets me close is:

command | tee >(sed -rn 's/.*foo (bar).*/1/p') >(awk '{print $3}')

However, I would like to achieve the following:

• I don't need to see the input stream of the original command
• I would like to use 'paste' to merge the results

I suppose one option is to run two separate commands and put them into files, but that isn't as elegant as I would like. What is the most elegant (single liner, clearly understood) way to do this in bash?

Answer 1

The reason you are seeing the output of the original command is because tee outputs to stdout as well as the files specified. To discard this you can put >/dev/null at the end of the command or redirect this output to one of your process substitutions by adding an extra >, eg:

command | tee >(sed -rn 's/.*foo (bar).*/1/p') > >(awk '{print $3}')

Or simpler just use another pipe:

command | tee >(sed -rn 's/.*foo (bar).*/1/p') | awk '{print $3}'

As for combining the result of the two process substitutions using paste, unless there is some obscure shell trick that I don't know about, there is no way to do this without using a named pipe. Ultimately this is two lines (formatted to more for clarity):

mkfifo /tmp/myfifo
command |
  tee >(sed -rn 's/.*foo (bar).*/1/p' >/tmp/myfifo) |
  awk '{print $3}' |
  paste /tmp/myfifo -

If you are putting this in a script, also consider using the recommendations for creating a temporary named pipe here.

Answer 2

You can do all of that just in sed.

command | 
sed '/\([^ ][^ ]*  *\)\{2\}/{h
         s///;s/^  *//;s/ .*/p
     g};s/.*foo \(bar\).*/\1/p;d'  

But as to the other thing, you have tee around the |pipe files:

cmd1 | {     
    {   tee /dev/fd/3 | 
        cmd2 >&2
    } 3>&1 | 
    cmd3
} 2>&1 |paste

But since you're already using sed you can use it like a smart tee and only dup/redirect when you have to:

cmd | { 
    sed -n 'p;s/.*foo \(bar\).*/\1/w /dev/fd/3' |
    awk ...
} 2>&1

Answer 3

IMHO it is more elegant (clearly understood) avoiding "tricks"with tee.
@mikeserv already showed a sed command, you can also "paste" inside awk.
Example with command replaced by a printf command:

$ printf "foo %s %s %s\n"  {1..30} 
foo 1 2 3
foo 4 5 6
foo 7 8 9
foo 10 11 12
foo 13 14 15
foo 16 17 18
foo 19 20 21
foo 22 23 24
foo 25 26 27
foo 28 29 30

Requirement (example):
Paste output from

1. sed -rn '/3/ s/[^ ]* ([^ ]*).*/\1/'
   Only select lines with a '3' and get the second field.
   These are the lines with the first number 1, 13, 22 and 28
2. awk 'NR>2 {print $3}'
   Show third field for all lines

The stream from requirement 1 is captured in the array bar,
the stream from requirement 2 is captured in the array secondstream.
After processing all input, in the END block the captured streams are pasted.

printf "foo %s %s %s\n"  {1..30} |
   awk '/3/  {bar[barcounter++]=$2}
        NF>2 {secondstream[streamnr++]=$3} 
        END  {for (i=0;i<streamnr;i++) print bar[i], secondstream[i];}
       '

Result:

1 2
13 5
22 8
28 11
 14
 17
 20
 23
 26
 29