6

I'm trying to pipe a command through the output of two other commands and then merge the results of the two process substitutions. An example that gets me close is:

command | tee >(sed -rn 's/.*foo (bar).*/1/p') >(awk '{print $3}')

However, I would like to achieve the following:

  • I don't need to see the input stream of the original command
  • I would like to use 'paste' to merge the results

I suppose one option is to run two separate commands and put them into files, but that isn't as elegant as I would like. What is the most elegant (single liner, clearly understood) way to do this in bash?

9

The reason you are seeing the output of the original command is because tee outputs to stdout as well as the files specified. To discard this you can put >/dev/null at the end of the command or redirect this output to one of your process substitutions by adding an extra >, eg:

command | tee >(sed -rn 's/.*foo (bar).*/1/p') > >(awk '{print $3}')

Or simpler just use another pipe:

command | tee >(sed -rn 's/.*foo (bar).*/1/p') | awk '{print $3}'

As for combining the result of the two process substitutions using paste, unless there is some obscure shell trick that I don't know about, there is no way to do this without using a named pipe. Ultimately this is two lines (formatted to more for clarity):

mkfifo /tmp/myfifo
command |
  tee >(sed -rn 's/.*foo (bar).*/1/p' >/tmp/myfifo) |
  awk '{print $3}' |
  paste /tmp/myfifo -

If you are putting this in a script, also consider using the recommendations for creating a temporary named pipe here.

| improve this answer | |
  • Thanks, I knew about the tee redirect option. I did indeed achieve what I wanted with fifos, I was just hoping there was something more elegant. – Arima Jun 12 '14 at 1:09
  • @Arima, it definitely seems like it should (there is certainly a lot you can do with process substitution). There is a connect command proposed for moreutils looks like it would come close, but it just looks like it is a proposal just now without an implementation. – Graeme Jun 12 '14 at 7:52
3

You can do all of that just in sed.

command | 
sed '/\([^ ][^ ]*  *\)\{2\}/{h
         s///;s/^  *//;s/ .*/p
     g};s/.*foo \(bar\).*/\1/p;d'  

But as to the other thing, you have tee around the |pipe files:

cmd1 | {     
    {   tee /dev/fd/3 | 
        cmd2 >&2
    } 3>&1 | 
    cmd3
} 2>&1 |paste

But since you're already using sed you can use it like a smart tee and only dup/redirect when you have to:

cmd | { 
    sed -n 'p;s/.*foo \(bar\).*/\1/w /dev/fd/3' |
    awk ...
} 2>&1
| improve this answer | |
0

IMHO it is more elegant (clearly understood) avoiding "tricks"with tee.
@mikeserv already showed a sed command, you can also "paste" inside awk.
Example with command replaced by a printf command:

$ printf "foo %s %s %s\n"  {1..30} 
foo 1 2 3
foo 4 5 6
foo 7 8 9
foo 10 11 12
foo 13 14 15
foo 16 17 18
foo 19 20 21
foo 22 23 24
foo 25 26 27
foo 28 29 30

Requirement (example):
Paste output from

  1. sed -rn '/3/ s/[^ ]* ([^ ]*).*/\1/'
    Only select lines with a '3' and get the second field.
    These are the lines with the first number 1, 13, 22 and 28
  2. awk 'NR>2 {print $3}'
    Show third field for all lines

The stream from requirement 1 is captured in the array bar,
the stream from requirement 2 is captured in the array secondstream.
After processing all input, in the END block the captured streams are pasted.

printf "foo %s %s %s\n"  {1..30} |
   awk '/3/  {bar[barcounter++]=$2}
        NF>2 {secondstream[streamnr++]=$3} 
        END  {for (i=0;i<streamnr;i++) print bar[i], secondstream[i];}
       '

Result:

1 2
13 5
22 8
28 11
 14
 17
 20
 23
 26
 29
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.