How do I extract multiple strings which are comma delimited from a log file?

Question

I need to extract certain strings from a log file although the strings are NOT in a standardized field/column. For example:

date="2017-01-03 08:30:02 -0500",fac=f_kernel_ipfilter,area=a_general_area,type=t_attack,pri=p_major,hostname=hostname,category=policy_violation,event="ACL deny",attackip=1.1.1.1,attackzone=internal,app_risk=low,app_categories=remote-admin,netsessid=c550e586ba75a,src_geo=US,srcip=1.1.1.1,srcport=38256,srczone=internal,protocol=6,dst_geo=US,dstip=2.2.2.2,dstport=80,dstzone=external,rule_name=Deny_All,reason="Traffic denied by policy.",application=SSH

I want to grab srcip, srczone, protocol, dstip, dstzone,dstport, and rule_name. I currently use a perl lazy match to strip OUT fields I don't want. Is there a way to only grab those 8 strings and the data within the commas like ,dstport=80, regardless of the position in the log file? many different entries positions for the same data which make this difficult.

Answer 1

Here's a quick'n dirty approach using perl:

$ perl -F, -lane '@l = grep {/srcip|srczone|protocol|dstip|dstzone|dstport|rule_name/} @F; 
                  print join ",",@l' file 
srcip=1.1.1.1,srczone=internal,protocol=6,dstip=2.2.2.2,dstport=80,dstzone=external,rule_name=Deny_All

The -a makes perl act like awk and split its input lines on the character given by -F, saving them as elements of the array @F. Then, we grep the array and keep elements matching your target words in the array @l, and finally print @l joined with commas.

Note that this will fail if any of your patterns can be subpatterns (say you have foo=bar and foobar=baz).

For longer lists of target patterns (assuming you don't want to write an actual script), you could store them in an array and join them with | to make the regex for grep. And by adding \b around each pattern you protect from matching subpatterns as well. If we also remove the needless temp array, we get:

$ perl -F, -lane '
    BEGIN{
     $pat="\\b" . join "\\b|",qw(srcip= srczone= protocol= dstip= dstzone= dstport= rule_name=)
    } print join ",",grep {/$pat/}@F' file 
srcip=1.1.1.1,srczone=internal,protocol=6,dstip=2.2.2.2,dstport=80,dstzone=external,rule_name=Deny_All

---

Our resident expert said it cannot be done in the bourne shell regardless of sed awk or [. . .. ] _{From your commant}

Sorry, but that's patently absurd. Here's one (of many) ways of doing it in each of those tools:

1. Bourne (again) shell. Don't use this, I only show it to demonstrate it is possible.

   $ pat=(srcip= srczone= protocol= dstip= dstzone= dstport= rule_name=); 
   $ o=""; while IFS=, read -a fields; do 
               for f in "${fields[@]}"; do 
                   for pat in "${pat[@]}"; do 
                       [[ $f =~ $pat ]] && o="$f,$o"
                   done 
               done
              done < file ; echo ${o%,}
   

2. Awk

   Save your target patterns in a file:

   $ cat patterns
   srcip
   srczone
   protocol
   dstip
   dstzone
   dstport
   rule_name
   

   Then:

   $ awk -F, '(NR==FNR){ 
                   pat[$0]++; 
                   next;
               } 
               {
                   for(i=1;i<=NF;i++){ 
                       split($i,a,"="); 
                       if(a[1] in pat){
                           printf "%s=%s,",a[1],a[2]
                       }
                   }
                   print ""
               }' patterns file | sed 's/,$//'
   srcip=1.1.1.1,srczone=internal,protocol=6,dstip=2.2.2.2,dstport=80,dstzone=external,rule_name=Deny_All
   

3. sed (and shell)

   $ pat=(srcip= srczone= protocol= dstip= dstzone= dstport= rule_name=);
   $ for p in ${pat[@]}; do 
       sed -E "s/.*($p[^,]*).*/\1/" file; done | 
           sed ':a;N;$!ba;s/\n/,/g'
   srcip=1.1.1.1,srczone=internal,protocol=6,dstip=2.2.2.2,dstzone=external,dstport=80,rule_name=Deny_All
   

4. Bourne shell (or any POSIX shell) + sed (as for 1., don't do this, it's possible but silly)

   $ set srcip= srczone= protocol= dstip= dstzone= dstport= rule_name=
   $ for f in "$@"; do sed "s/.*\($f[^,]*\).*/\1/" file; done | sed ':a;N;$!ba;s/\n/,/g'
   srcip=1.1.1.1,srczone=internal,protocol=6,dstip=2.2.2.2,dstzone=external,dstport=80,rule_name=Deny_All
   

Answer 2

A solution with GNU awk:

gawk -v OFS= -v FPAT=',(srcip|srczone|protocol|dstip|dstzone|dstport|rule_name)=[^,]*' -e 'NF > 0 { $1=$1; print }'

Here, I'm using a feature specific to GNU awk: with the FPAT variable, I'm specifying the format of the fields with a regex, so that each part of the line matching the regex is assigned to $1...$n. I then assign $1 to $1 so that $0 is rebuilt from scratch using only $1...$n.

Answer 3

I'm a little delayed on this, but would offer a suggestion - this sort of data is quite well suited to maping into a hash:

#!/usr/bin/env perl

use strict;
use warnings;
#for debugging - can be removed;
use Data::Dumper;

my @fields = qw ( srcip srczone protocol dstip dstzone dstport rule_name );

#read STDIN or files specified on command line (just like grep/sed/awk)
while ( <> ) {

   #split commas
   #then read key-value pairs. 
   my %row = map { m/(.*)=(.*)/ } split /,/;
   #for debugging:
   print Dumper \%row;

   #print fields tab-separated and in order as above. 
   print join "\t", @row{@fields};
}

It's slightly harder to oneliner-ify because you've got a list of fields to spell out. But:

perl -lane -F, 'BEGIN { @k = qw ( srcip srczone protocol dstip dstzone dstport rule_name ) } %r = map { m/(.*)=(.*)/ } @F; print join "\t", @r{@k}'

Answer 4

Put the regex patterns you want in a file and use egrep.

Your file (we'll call it filters.txt):

srcip=(.*?),
srczone=(.*?),
....

Your command:

grep -Eof filters.txt logfile.txt

Answer 5

bash

IFS=, read -r -a fields <<< "$date"
results=()
for keyval in "${fields[@]}"; do 
    IFS='=' read -r key value <<< "$keyval"
    case $key in 
        srcip|srczone|protocol|dstip|dstzone|dstport|rule_name) results+=("$keyval")
    esac
done
(IFS=,; echo "${results[*]}")

srcip=1.1.1.1,srczone=internal,protocol=6,dstip=2.2.2.2,dstport=80,dstzone=external,rule_name=Deny_All

Answer 6

Solution

You can use the command cut. In your case, you can give it three parameters :

• -d ',' : This represents the delimiter of your string. In your case, it's the ,. You can replace the delimiter by what ever you want.
• -f 4,7,8 : This represents the part of the string that you want to get (separated by comma). In this case, it is the 4th, 7th and the 8th part. You can specify a range using this form 1-8 for example.
• file.ext : This is your log file.

Examples of use

Lets say your log file is file.log (very creative name) :

• cut -d ',' -f 1-5 file.log : This will give you date="2017-01-03 08:30:02 -0500" fac=f_kernel_ipfilter,area=a_general_area

• cut -d ',' -f 3,5,6 file.log : This will give you fac=f_kernel_ipfilter,type=t_attack,pri=p_major

• cut -d ',' -f 4 file.log : This will give you type=t_attack

Note

cut will go through all the lines of your file. If you want to skip the lines that doesn't contain the delimiter you specified, use the option -s.

Answer 7

I have tried with only sed

sed -r 'h;
s/.*(srcip=[^,]*).*/\1/; H; g; 
s/.*(srczone=[^,]*).*/\1/; H; g; 
s/.*(protocol=[^,]*).*/\1/; H; g; 
s/.*(dstip=[^,]*).*/\1/; H; g; 
s/.*(dstport=[^,]*).*/\1/; H; g; 
s/.*(dstzone=[^,]*).*/\1/; H; g; 
s/.*(rule_name=[^,]*).*/\1/; H; g; 
s/.*\n//M; 
s/\n/,/g;
' file