How to display output of command while also parsing it?

Question

I have two commands, build and deploy. Currently I run build manually, parse its output using my own eyes, and use a value I find in the output as an argument to deploy. The process looks something like:

$ build
==> amazon-ebs: amazon-ebs output will be in this color.
... hundreds of lines of output ...
==> Builds finished. The artifacts of successful builds are:
--> amazon-ebs: AMIs were created:
us-east-1: ami-19601070

$ deploy ami-19601070
... some more output ...

(build is actually Packer, for the astute)

I would like to tie these two steps together in a script. The rough outline would include the following:

• Run build
• Make sure the exit status was 0 and that the output contained the string "AMIs were created", otherwise abort
• Extract the AMI number (ami-19601070) from the output
• Run deploy ami-19601070

I'm trying to come up with the best way to connect everything together, ideally by using a shell script, but I'm stuck on how to grep the output for two separate patterns while, ideally, still streaming all stdout/stderr messages to the terminal as the commands run. I'm wondering if I should abandon the idea of doing this in a shell script and instead write a small Python script to do it.

Answer 1

Sounds like a job for tee:

build | tee /some/file
ami_name=$(do_something_to /some/file)
deploy "$ami_name"

Answer 2

 deployArgs=`build | tee /dev/tty  | extractDeployArgs` &&
    deploy "$deployArgs" #won't run unless extractDeployArgs suceeded

tee /dev/tty will print directly to the terminal and pass the output to the next command in the pipeline at the same time.

(Feel free to replace it with some other file (or /dev/fd/"$someFileDescriptor" if you need want the side output to go to $someFileDescriptor))

In more advanced shells (ksh, bash, zsh, but not in dash) you can set -o pipefail to make sure the pipeline fails if any of its links fails (useful if extractDeployArgs can't tell from its input whether build suceeded or not).

Answer 3

tf=$(mktemp)
build | tee "$tf"
grep -Fq 'AMIs were created' "$tf" && ami=$(grep -o 'ami-[0-9]\+$' "$tf")

# you didn't say if `deploy` can handle multiple args or not.
# uncomment one of the following:
# deploy $ami
# for a in $ami ; do deploy "$a" ; done

Answer 4

If your interest is in a particular string of output you could run it like:

msg="$(build | grep -e "AMIs were created\|ami-[0-9]*")"

if [ -n "$(echo $msg | grep -e "AMIs were created")" ];then
    ami="$(echo "$msg" | grep -e "ami-[0-9]*" | cut -d ' ' -f 2)"
    deploy "$ami"
else
    exit 1
fi

The first grep selects only the lines in your output that report AMI's were created or ami-#. The output is checked for "AMIs were created", and if present, parses the ami-# out of your relevant output and uses it to deploy.