3

I have few logs files in my machineB under this directory /opt/ptd/Logs/ as shown below - My logs files are pretty big.

david@machineB:/opt/ptd/Logs$ ls -lt
-rw-r--r-- 1 david david  49651720 Oct 11 16:23 ptd.log
-rw-r--r-- 1 david david 104857728 Oct 10 07:55 ptd.log.1
-rw-r--r-- 1 david david 104857726 Oct 10 07:50 ptd.log.2

I am trying to write a generic shell script which should try to parse all my log file in machineB for a particular pattern and print the line which has those patterns. I will be running my below shell script from machineA which has all the ssh keys setup everything meaning I need to remotely grep on the logs files on machineB from machineA.

#!/bin/bash

wordsToInclude="hello,animal,atttribute,metadata"
wordsToExclude="timeout,runner"

# now grep on the various log file for above words and print out the lines accordingly

Meaning, I will have words separated by comma in wordsToInclude variable - If my logs contain hello word then print out that line, also print out the line which contains animal word. Similarly with attribute and metadata words.

And also I will have words separated by comma in wordsToExclude variable - If any of the lines contains those words then don't print out those line.

I am going with the above format for now for storing the words but any better format is fine to me. I can have long list of words in wordsToInclude and wordsToExclude variable so that's why I am going with storing them in those variables.

I know how to do a grep on small set of variables. If I need to do grep from the command line directly on machineB, then I will do it like this -

grep -E 'hello|animal|atttribute|metadata' ptd.log | grep -v 'timeout'

But I am not sure how do I combine this in my shell script so that I can do a remote ssh grep on machineB from machineA.

6

If you are open to other formats, consider:

inc="hello|animal|atttribute|metadata"
exc="timeout|runner" 
ssh machineB "grep -E '$inc' path/ptd.log | grep -vE '$exc'"

Faster Alternative

If your log files are large and you are grepping for fixed words, as opposed to fancy regular expressions, you may want to consider this approach:

inc='hello
animal
atttribute
metadata'

exc='timeout
runner'

ssh office "grep -F '$inc' ptd.log | grep -vF '$exc'"

By putting each word on a separate line, we can use grep's -F feature for fixed strings. This turns off regex processing, making the process faster.

1

It may not seem possible but you can make use of grep's -f option to make use of that list of words, even though they're in a environment variable and not a proper file. The trick is in fooling grep into thinking that they are from a file like so:

$ ssh machineB 'grep -f <(echo $wordsToInclude|tr , "\n") file1 file2 file3'

This will run the grep ... command remotely via ssh on machineB. It will take your variable, $wordsToInclude and switch the commas to end of line characters, (, -> \n). This list of words is then fed into grep via its -f switch.

To run this through the exclude list simply add that as a 2nd grep after the initial one via a pipe.

$ ssh machineB 'grep -f <(echo $wordsToInclude|tr , "\n") \
    file1 file2 file3 | grep -vf <(echo $wordsToExclude)'
1

SSH is run with a command like so:

ssh host command

Or in your case:

ssh -t machineB "grep -E \"$wordsToInclude\" ptd.log | grep -v \"$wordsToExclude\""

The -t prevents an "ioctl error". I'd also recommend using grep's fixed words for increased speed, as specified by this answer by @John1024. Just put each word on it's own line, like:

wordsToInclude='hello
animal
atttribute
metadata'

wordsToExclude='timeout
runner'

And add -F to grep's options.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.