1

I'm working on a script that will run a checksum process which outputs to STDOUT, which I then want to grep for lines matching OK, or FAILED and do different things with those matches (i.e. output to terminal and log). I've watched a ton of Youtube videos and read a ton about redirection, but I just can't seem to wrap my head around how exactly redirection works. What I'm trying to do is chain STDOUT to multiple greps without them gobbling up the non-matched text.

Here's a concept of what I'm trying using cat instead of md5sum with a text file of animal names on each line (DOG, CAT, PONY, RHINO, DEER, FOX):

{ cat test.txt 3>&1 | tee /dev/fd/3 | grep DOG; } 3> results.txt

This does what I expect. What I understand here is I'm doing a cat on the file, and then opening fd3 which points to whatever is written to STDOUT(fd1). Since the grep will gobble up everything from fd1, I tee the STDOUT of cat explicitly to fd3 and then pipe the STDOUT to grep. Grep will print out the line matching DOG, and then all the text written to fd3 from cat will get pushed to a results.txt file.

Now, to chain another grep to look for other text I have to point the fd3 data back into STDOUT, tee it explicitly back to fd3 and then pipe STDOUT to a new grep.

{ { cat test.txt 3>&1 | tee /dev/fd/3 | grep DOG; } 3>&1 | tee /dev/fd/3 | grep PONY; } 3> results.txt

The first problem here is the STDOUT from the first grep is being pushed into fd3 a second time instead of printing to the terminal. So now my results.txt is getting duplicates and I never got anything printed to the screen for the first grep. This is where my understanding of redirections is falling apart. I sort-of get what's happening but I can't figure out a simple solution.

I want to grep STDOUT, print the results to screen, and pass the original text to another grep, and maybe a third, fourth, etc without modifying the original text I'm passing to each GREP, and without each subsequent grep eating up the previous' match that should print to screen.

I could probably do this by storing a variable and calling it on multiple lines of greps, but then I have to wait for the entire first command to complete. In the case of the application I'm working on, I want to see realtime results during a checksum, not just a blank screen for an hour until the whole process is complete. Any clarification on what I'm doing wrong would be super helpful, thanks!

EDIT

I understand this exact use of cat is pointless, I just used it to demonstrate the concept. In the script I'll be applying the concept to, the first command is actually:

md5sum -c checksum.md5

Which will read a checksum file, re-hash the the source and output to STDOUT a pass/fail line. I then want to grep this stream and send the results to separate logs and/or terminal output - but cat seemed like a simpler way to demonstrate the problem as this can be applied to filtering any command and grepping the stream, such as find, md5, ls, etc.

  • The redirection in cat test.txt 3>&1 doesn't make any sense as it is never used. The whole cat is useless: <test.txt tee ... – Hauke Laging Jan 13 '18 at 23:38
  • What do you need results.txt for? It is just a copy of test.txt – Hauke Laging Jan 13 '18 at 23:45
  • @HaukeLaging I edited my post to to clarify that the first command doesn't really matter, and I'd really be using this where the first command is generating a new stream of text, rather than a simple read of a file which I used to just demonstrate the concept. – iceblueorbitz Jan 14 '18 at 0:02
4

You can better do what you ask for with process substitution:

  1. Being as close as your original command as possible:

    cat test.txt | tee >(grep DOG) >(grep PONY) >results.txt
    
  2. Removing the useless use of cat:

    <test.txt tee >(grep DOG) >(grep PONY) >results.txt
    

    Or:

    tee >(grep DOG) >(grep PONY) <test.txt >results.txt
    
  • Useless use of cat – Hauke Laging Jan 13 '18 at 23:36
  • While a useless use case for cat, the answer seems to point out a potential solution to the concept of filtering STDOUT into separate grep processes, which is all my original question was asking, first command being a moot point, as long as it prints something to STDOUT. – iceblueorbitz Jan 14 '18 at 0:00
  • @HaukeLaging Thanks for the nitpicking. Solved. Please Remove your downvote. – Isaac Jan 14 '18 at 1:50
  • This should not be downvoted, it provided a perfect solution to my question about redirecting stdout to multiple processes, regardless what command was used to create the initial stdout. Thanks @isaac – iceblueorbitz Jan 14 '18 at 2:37
  • There is a downvote but it's not mine. I would not downvote an answer because of UUOC. I even made an upvote after the UUOC edit. – Hauke Laging Jan 14 '18 at 10:32
1

isaac's solution is better but your way would look like this:

{ <input tee results.txt /dev/fd/3 | grep DOG >&2; } 3>&1 |
    { tee /dev/fd/3 | grep PONY >&2; } 3>/dev/null

or for three

{ <input tee results.txt /dev/fd/3 | grep DOG >&2; } 3>&1 |
    { tee /dev/fd/3 | grep PONY >&2; } 3>&1 |
    { tee /dev/fd/3 | grep CAT >&2; } 3>/dev/null
  • @isaac This can be extended as much as you like. – Hauke Laging Jan 14 '18 at 10:26
  • @isaac See the edit – Hauke Laging Jan 14 '18 at 11:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.