Trying to do a "lookup" in a pipeline, where the input looks like this:

alice  5
bob    7

I want to look up codes in the second column in a database and return the corresponding name, and keep on trucking with the original and looked-up data.

cat source.tab | \
  tee foo.tmp | \
  cut -f 2 | \
  dbstream ... -s "select(select name from my_lookup where code=?)" | \
  paste foo.tmp -

Result should be:

alice  5  foo
bob    7  bar

Imagine for a minute that cat source.tab is really a long pipeline that does other pre-processing. And that dbstream .. could be some other command, say wget | jq.

Important: I only want to start the lookup process once.

a) is this a terrible idea, and if so what should I be doing instead?
b) is there a better pattern than tee tmp | cut | "lookup" | paste tmp - ?


It depends on how complicated the output is and how much formatting needs to be maintained (eg is the first column always 8 characters long? etc). However a while loop might work

cat source.tab | while read -r name id
  echo "$name $id $(dbstream .... code=$id)"

You can change what happens inside the loop to format however you like eg

cat source.tab | while read -r name id
  res=$(dbstream ... code=$id)
  printf "%10s %5d %s" $name $id $res

As per comment, you only want to call dbstream once. This requires dbstream to keep output in the same order as input.

Here's a simple example dbstream program:

for a in "$@"
  echo dbstream $$ sees $a

We include the PID in the output so we can show it only gets called once.

Now we can use paste and process substitution:

$ paste source.tab <(./dbstream $(awk '{print $2}' source.tab ))
alice 1 dbstream 20671 sees 1
bob   2 dbstream 20671 sees 2

Now if the source.tab is a slow process I would recommend using a temporary file




trap '/bin/rm -f $tmp ; exit' 0 1 2 3 15

cat source.tab > $tmp
paste $tmp <(./dbstream $(awk '{print $2}' $tmp ))
  • does that start a new dbstream process per line? I want to avoid that. thx – Neil McGuigan Jul 28 '16 at 23:18
  • 1
    It does. I'll write a variation that will only call it once. – Stephen Harris Jul 28 '16 at 23:23
  • nice one. i'll try it at home this eve. cheers – Neil McGuigan Jul 28 '16 at 23:32
  • changed it to paste source.tab <( awk '{print $2}' source.tab | dbstream ) as dbstream reads from stdin. works! – Neil McGuigan Jul 28 '16 at 23:52
  • 1
    and in fact you can say while IFS= read ...; do ... done < file instead of cat file | while .... – fedorqui Jul 29 '16 at 9:25

The correct way appears to be to use a named pipe


function datastreamWrapper() {

  mypipe=$(mktemp -u)
  mkfifo -m 600 "$mypipe"

  tee >( cut -f2 | datastream ... > "$mypipe") | paste - "$mypipe"

  rm "$mypipe"

Then you can put datastreamWrapper in your pipeline:

cat source.tab | datastreamWrapper | foo

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.