Need help structuring a nested zgrep to produce a count

Question

I've been assisting with some reporting at work, and am trying to get a unique count of different elements from some rather large log files that we are generating. So far, I've been able to run two separate commands to come up with the counts, but due to some restrictions, I'm going to have to combine them. Here is what I have been running (the names of files, directories, search items have been changed to protect the innocent):

Command 1 - filter by _transformer_ and write all these unique log entries to a new file (each item that I'm counting has multiple entries per transaction, so I do this to speed up the next step as well as remove duplicates):

zgrep -a _transformer_ /files/are/located/here/logfile_date.gz > \
     /temp/directory/count_file_date.gz

Command 2 - count the number of instances of each of several items related to the transactions:

zgrep -caE 'shockwave|starscream|megatron|prowl|blaster' \
     /temp/directory/count_file_date.gz

This has worked great, but I want to combine them into one command and skip writing the new file. This is what I thought would work, but isn't:

Single Command

zgrep -a _transformer_ | \
zgrep -acE 'shockwave|starscream|megatron|prowl|blaster' \
      /files/are/located/here/logfile_date.gz

Running the above command outputs a count of all log entries with the words between the pipes, and doesn't only count the ones with transformer in the particular line of the log.

Answer 1

zgrep -a _transformer_ /files/are/located/here/logfile_date.gz |
  grep -acE 'shockwave|starscream|megatron|prowl|blaster'

The first zgrep calls gzip -dcf on the file, then internally runs grep -a _transformer_ on the resulting stream which produces an uncompressed stream which we pass on to another grep invocation, all three (gzip, grep, grep) running concurrently, there's no intermediary data ever stored on disk.

You can also run those 3 commands manually:

gzip -dcf /files/are/located/here/logfile_date.gz |
  grep -a _transformer_ |
  grep -acE 'shockwave|starscream|megatron|prowl|blaster'

(the -f (when combined with -c and -d) is for gzip to behave like cat if the file happens not to be compressed, -d to uncompress, -c to output the result on stdout instead of producing an uncompressed /files/are/located/here/logfile_date).

It can be done in one grep if it supports -P:

gzip -dcf /files/are/located/here/logfile_date.gz |
  grep -acP '^(?=.*_transformer_).*(shockwave|starscream|megatron|prowl|blaster)'

Or portably with awk:

gzip -dcf /files/are/located/here/logfile_date.gz |
  awk -v n=0 '
    /_transformer/ && /shockwave|starscream|megatron|prowl|blaster/ {n++}
    END {print n}'