1

I need to extract certain strings from a log file although the strings are NOT in a standardized field/column. For example:

date="2017-01-03 08:30:02 -0500",fac=f_kernel_ipfilter,area=a_general_area,type=t_attack,pri=p_major,hostname=hostname,category=policy_violation,event="ACL deny",attackip=1.1.1.1,attackzone=internal,app_risk=low,app_categories=remote-admin,netsessid=c550e586ba75a,src_geo=US,srcip=1.1.1.1,srcport=38256,srczone=internal,protocol=6,dst_geo=US,dstip=2.2.2.2,dstport=80,dstzone=external,rule_name=Deny_All,reason="Traffic denied by policy.",application=SSH

I want to grab srcip, srczone, protocol, dstip, dstzone,dstport, and rule_name. I currently use a perl lazy match to strip OUT fields I don't want. Is there a way to only grab those 8 strings and the data within the commas like ,dstport=80, regardless of the position in the log file? many different entries positions for the same data which make this difficult.

7
  • I apologize if the question is not articulated appropriately. It is hard to describe the problem.
    – Steve
    Jan 3 '17 at 14:53
  • 3
    It would be easier to understand if you included an example of the exact output you would like to get from your input. Do you need everything in the same line? Different lines? Should the commas be included?
    – terdon
    Jan 3 '17 at 14:54
  • 1
    Output all on one line
    – Steve
    Jan 3 '17 at 14:59
  • srcip=1.1.1.1, srczone=internal, dstip=2.2.2.2, dstzone=external, dstport=53, protocol=UDP, rule_name=Deny_All
    – Steve
    Jan 3 '17 at 15:01
  • 1
    Commas can be included or stripped.
    – Steve
    Jan 3 '17 at 15:02
4

Here's a quick'n dirty approach using perl:

$ perl -F, -lane '@l = grep {/srcip|srczone|protocol|dstip|dstzone|dstport|rule_name/} @F; 
                  print join ",",@l' file 
srcip=1.1.1.1,srczone=internal,protocol=6,dstip=2.2.2.2,dstport=80,dstzone=external,rule_name=Deny_All

The -a makes perl act like awk and split its input lines on the character given by -F, saving them as elements of the array @F. Then, we grep the array and keep elements matching your target words in the array @l, and finally print @l joined with commas.

Note that this will fail if any of your patterns can be subpatterns (say you have foo=bar and foobar=baz).

For longer lists of target patterns (assuming you don't want to write an actual script), you could store them in an array and join them with | to make the regex for grep. And by adding \b around each pattern you protect from matching subpatterns as well. If we also remove the needless temp array, we get:

$ perl -F, -lane '
    BEGIN{
     $pat="\\b" . join "\\b|",qw(srcip= srczone= protocol= dstip= dstzone= dstport= rule_name=)
    } print join ",",grep {/$pat/}@F' file 
srcip=1.1.1.1,srczone=internal,protocol=6,dstip=2.2.2.2,dstport=80,dstzone=external,rule_name=Deny_All

Our resident expert said it cannot be done in the bourne shell regardless of sed awk or [. . .. ] From your commant

Sorry, but that's patently absurd. Here's one (of many) ways of doing it in each of those tools:

  1. Bourne (again) shell. Don't use this, I only show it to demonstrate it is possible.

    $ pat=(srcip= srczone= protocol= dstip= dstzone= dstport= rule_name=); 
    $ o=""; while IFS=, read -a fields; do 
                for f in "${fields[@]}"; do 
                    for pat in "${pat[@]}"; do 
                        [[ $f =~ $pat ]] && o="$f,$o"
                    done 
                done
               done < file ; echo ${o%,}
    
  2. Awk

    Save your target patterns in a file:

    $ cat patterns
    srcip
    srczone
    protocol
    dstip
    dstzone
    dstport
    rule_name
    

    Then:

    $ awk -F, '(NR==FNR){ 
                    pat[$0]++; 
                    next;
                } 
                {
                    for(i=1;i<=NF;i++){ 
                        split($i,a,"="); 
                        if(a[1] in pat){
                            printf "%s=%s,",a[1],a[2]
                        }
                    }
                    print ""
                }' patterns file | sed 's/,$//'
    srcip=1.1.1.1,srczone=internal,protocol=6,dstip=2.2.2.2,dstport=80,dstzone=external,rule_name=Deny_All
    
  3. sed (and shell)

    $ pat=(srcip= srczone= protocol= dstip= dstzone= dstport= rule_name=);
    $ for p in ${pat[@]}; do 
        sed -E "s/.*($p[^,]*).*/\1/" file; done | 
            sed ':a;N;$!ba;s/\n/,/g'
    srcip=1.1.1.1,srczone=internal,protocol=6,dstip=2.2.2.2,dstzone=external,dstport=80,rule_name=Deny_All
    
  4. Bourne shell (or any POSIX shell) + sed (as for 1., don't do this, it's possible but silly)

    $ set srcip= srczone= protocol= dstip= dstzone= dstport= rule_name=
    $ for f in "$@"; do sed "s/.*\($f[^,]*\).*/\1/" file; done | sed ':a;N;$!ba;s/\n/,/g'
    srcip=1.1.1.1,srczone=internal,protocol=6,dstip=2.2.2.2,dstzone=external,dstport=80,rule_name=Deny_All
    
3
  • For 5 years i have been stripping out text i didnt need. In 5 minutes you solved it all !!! BOTH commands work flawlessly!!! HAPPY NEW YEAR !!!!
    – Steve
    Jan 3 '17 at 15:12
  • @Steve heh, glad you liked it :) See updated answer for something to show your "expert". None of these are particularly elegant, mind you, but they are proof of concept to demonstrate that of course this is achievable using any of the mentioned tools.
    – terdon
    Jan 3 '17 at 15:44
  • I may show him, but I don't want to rub his nose in poop for CLEARLY being wrong. But this is why I signed up on this site as rarely is there text editing that cannot be done.
    – Steve
    Jan 3 '17 at 16:47
1

A solution with GNU awk:

gawk -v OFS= -v FPAT=',(srcip|srczone|protocol|dstip|dstzone|dstport|rule_name)=[^,]*' -e 'NF > 0 { $1=$1; print }'

Here, I'm using a feature specific to GNU awk: with the FPAT variable, I'm specifying the format of the fields with a regex, so that each part of the line matching the regex is assigned to $1...$n. I then assign $1 to $1 so that $0 is rebuilt from scratch using only $1...$n.

6
  • Thank you for the input but unfortunately it did not work. I do have a solution already now.
    – Steve
    Jan 3 '17 at 15:44
  • this was the output date="2017-01-0309:29:18-0500",fac=f_kernel,area=a_nil_area,type=t_netprobe,pri=p_minor,hostname=hostname,event="TCPnetprobe",src_geo=US,srcip=2.2.2.2,srcport=51753,srczone=internal,dstip=1.1.1.1,dstport=4092,protocol=6,interface=2-1,reason="ReceivedaTCPconnectionattemptdestinedforaservicethatthecurrentpolicydoesnotsupport."
    – Steve
    Jan 3 '17 at 15:45
  • @Steve I do like to propose a solution with awk ;-) Sorry that it does not work. Are you sure to use gawk? I believe your awk doesn't understand the special variable FPAT.
    – xhienne
    Jan 3 '17 at 15:45
  • I am VERY NEW with this advanced use of sed and awk. I am using the old bourne shell (not BASH) on BSD and have found conflicts like this before. I have not yet learned about gawk. Thank you very much for the help !!
    – Steve
    Jan 3 '17 at 15:49
  • This is quite good, but please add an explanation. Especially since you're using obscure GNU-only features like FPAT.
    – terdon
    Jan 3 '17 at 15:49
1

I'm a little delayed on this, but would offer a suggestion - this sort of data is quite well suited to maping into a hash:

#!/usr/bin/env perl

use strict;
use warnings;
#for debugging - can be removed;
use Data::Dumper;

my @fields = qw ( srcip srczone protocol dstip dstzone dstport rule_name );

#read STDIN or files specified on command line (just like grep/sed/awk)
while ( <> ) {

   #split commas
   #then read key-value pairs. 
   my %row = map { m/(.*)=(.*)/ } split /,/;
   #for debugging:
   print Dumper \%row;

   #print fields tab-separated and in order as above. 
   print join "\t", @row{@fields};
}

It's slightly harder to oneliner-ify because you've got a list of fields to spell out. But:

perl -lane -F, 'BEGIN { @k = qw ( srcip srczone protocol dstip dstzone dstport rule_name ) } %r = map { m/(.*)=(.*)/ } @F; print join "\t", @r{@k}'
0

Put the regex patterns you want in a file and use egrep.

Your file (we'll call it filters.txt):

srcip=(.*?),
srczone=(.*?),
....

Your command:

grep -Eof filters.txt logfile.txt
14
  • I do have a solution now but this is interesting. I am looping through multiple log files grabbing specific strings. So you are saying I should create a file with only the regex of what I need srcip=(.*?),srczone=(.*?),dstip=(.*?),dstzone=(.*?),dstport=(.*?),protocol=(.*?),rule_name(.*?), but then in the grep command, how would I run it against the log file? grep -Eof filters.txt from logfile.txt ?
    – Steve
    Jan 3 '17 at 15:28
  • Unfortunately, you missed one of the OP's comments: "Output all on one line". Moreover, grep's regex are greedy, did you test your answer?
    – xhienne
    Jan 3 '17 at 15:30
  • should I create the filters.txt regex file and put each regex on one line or individula lines
    – Steve
    Jan 3 '17 at 15:42
  • like srcip=(.*?),srczone=(.*?),dstip=(.*?),dstzone=(.*?),dstport=‌​(.*?),protocol=(.*?)‌​,rule_name(.*?), or one on each line ?
    – Steve
    Jan 3 '17 at 15:42
  • @xhienne extended regex has non-greedy patterns which is what is being used here. That's what the ? means. Did you test this answer?
    – terdon
    Jan 3 '17 at 15:46
0

bash

IFS=, read -r -a fields <<< "$date"
results=()
for keyval in "${fields[@]}"; do 
    IFS='=' read -r key value <<< "$keyval"
    case $key in 
        srcip|srczone|protocol|dstip|dstzone|dstport|rule_name) results+=("$keyval")
    esac
done
(IFS=,; echo "${results[*]}")
srcip=1.1.1.1,srczone=internal,protocol=6,dstip=2.2.2.2,dstport=80,dstzone=external,rule_name=Deny_All
0

Solution

You can use the command cut. In your case, you can give it three parameters :

  • -d ',' : This represents the delimiter of your string. In your case, it's the ,. You can replace the delimiter by what ever you want.
  • -f 4,7,8 : This represents the part of the string that you want to get (separated by comma). In this case, it is the 4th, 7th and the 8th part. You can specify a range using this form 1-8 for example.
  • file.ext : This is your log file.

Examples of use

Lets say your log file is file.log (very creative name) :

  • cut -d ',' -f 1-5 file.log : This will give you date="2017-01-03 08:30:02 -0500" fac=f_kernel_ipfilter,area=a_general_area

  • cut -d ',' -f 3,5,6 file.log : This will give you fac=f_kernel_ipfilter,type=t_attack,pri=p_major

  • cut -d ',' -f 4 file.log : This will give you type=t_attack

Note

cut will go through all the lines of your file. If you want to skip the lines that doesn't contain the delimiter you specified, use the option -s.

2
  • 1
    right on, I do have a solution already though. One problem I have is that this log has multiple entries and different positions for many entries so grabbing a static field range wont work. Thank you for the input !!
    – Steve
    Jan 3 '17 at 15:24
  • Ah, sorry. Apparently I didn't well understand the question.
    – Sidahmed
    Jan 3 '17 at 15:33
0

I have tried with only sed

sed -r 'h;
s/.*(srcip=[^,]*).*/\1/; H; g; 
s/.*(srczone=[^,]*).*/\1/; H; g; 
s/.*(protocol=[^,]*).*/\1/; H; g; 
s/.*(dstip=[^,]*).*/\1/; H; g; 
s/.*(dstport=[^,]*).*/\1/; H; g; 
s/.*(dstzone=[^,]*).*/\1/; H; g; 
s/.*(rule_name=[^,]*).*/\1/; H; g; 
s/.*\n//M; 
s/\n/,/g;
' file

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.