How can I duplicate a pipe stream (output from another program) so that for every line, I can check whether it matches independent regular expressions (ideally, reuse a grep command with options like -i etc.) but process results of each expression separately?

I want to define exit code from my check function as follows:

  • no line in stream matches blacklist; AND
  • at least one line in stream matches whitelist

I want to process lines in memory without storing them to file, report exit code and pass original stream to another program using pipes (logging whitelist matches to stderr or elsewhere would be useful too).

Usage would look like:

fetch-big-stream | check whitelist blacklist | cat

where fork duplicates lines, whitelist and blacklist are functions and check passess copy of stream to each separately and terminates once exitcode can be determined before EOF. There is also a problem with | implementation that does not propagate non-zero code.

  • This would me much easier to understand if you showed the actual commands you use. Are you looking through one file or many? – terdon Feb 24 '14 at 21:55

Just use '-f' and '-v' with grep:

$ cat white
$ cat black
$ cat input
foo baz
foo bar

$ grep -f white input | grep -v -f black
foo bar

I think what you are trying to do is report files that contain at least one of the patterns in good.txt and none of those in bad.txt. If you want to do this for a single file, it's trivial:

grep -Ff good.txt file | grep -vFf bad.txt 

If you want to find all files in a directory that match the criteria, you could do something like this (note that it will also match good.txt if that's in the same directory):

grep -lFf good.txt * | xargs -I{} grep -lvFf bad.txt {}

You can fine tune it some more by using find. For example, avoid matching good.txt:

find . -type f -not -name good.txt -exec grep -lFf good.txt {} \; | 
 xargs -I{} grep -l a {}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.