3

I want to extract ip the very next to (axyz-pc). I have done this task through grep command using regex. But i need to extract through awk and sed. grep -Po '(?<='axyz-pc')[^:]+' logs | grep -oE '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'| sort -nr| uniq -c |sort -nr

Logs:

2017-04-11 15:15:00 SMTP connection from (axyz-pc) [36.32.138.106]:1236 I=[10.10.19.36]:25 closed by DROP in ACL
2017-04-11 15:15:01 H=(axyz-pc) [114.225.87.41]:3823 I=[10.10.19.36]:25 rejected EHLO or HELO axyz-pc: HELO/EHLO - HELO on heloblocks Blocklist
2017-04-11 15:15:01 SMTP connection from (axyz-pc) [114.225.87.41]:3823 I=[10.10.19.36]:25 closed by DROP in ACL
2017-04-11 15:15:02 H=(axyz-pc) [36.32.138.216]:1984 I=[10.10.19.36]:25 rejected EHLO or HELO axyz-pc: HELO/EHLO - HELO on heloblocks Blocklist
2017-04-11 15:15:02 SMTP connection from (axyz-pc) [36.32.138.216]:1984 I=[10.10.19.36]:25 closed by DROP in ACL
2017-04-11 15:15:02 H=(axyz-pc) [37.49.224.14]:51593 I=[10.10.19.36]:25 rejected EHLO or HELO axyz-pc: HELO/EHLO - HELO on heloblocks Blocklist
2017-04-11 15:15:02 SMTP connection from (axyz-pc) [37.49.224.14]:51593 I=[10.10.19.36]:25 closed by DROP in ACL
2017-04-11 15:15:02 H=(axyz-pc) [36.32.138.106]:4619 I=[10.10.19.36]:25 rejected EHLO or HELO axyz-pc: HELO/EHLO - HELO on heloblocks Blocklist

Output should be(duplicate ip not repeat) :

36.32.138.106
114.225.87.41
36.32.138.216
37.49.224.14
  • seems like for given input, extracting ips from alternate lines will be unique and sufficient... if that is not the case, please change sample input for that case – Sundeep Apr 11 '17 at 10:59
  • @Sundeep, if it's a log file, there might be multiple entries with the same IP (all with those two lines) – ilkkachu Apr 11 '17 at 11:02
  • @black, is there any particular reason to require awk or sed and not grep? – ilkkachu Apr 11 '17 at 11:02
  • @likkachu I want to extract ip from real time logging, grep got stuck while logs running too fast. – blaCkninJa Apr 11 '17 at 11:05
  • 1
    I don't see why grep would stall, but sort requires all input before it can complete and return any output - it's a much more likely bottleneck – Izkata Apr 11 '17 at 14:13
2

I am not sure why grep gets stuck, you will have to explore that further. However, there is no need for perlre here, something like this would do (at least for the sample you have given):

grep -o 'axyz-pc) \[[^]]*' | grep -o '[^[]*$'

Output:

36.32.138.106
114.225.87.41
114.225.87.41
36.32.138.216
36.32.138.216
37.49.224.14
37.49.224.14
36.32.138.106

Now sort numerically and apply uniq:

sort -t. -k1,1n -k2,2n -k3,3n -k4,4n | uniq

Output:

36.32.138.106                                                             
36.32.138.216
37.49.224.14
114.225.87.41
2

If you need to use sed, assuming that the ip is in the first brackets:

sed -n '/axyz-pc/s/[^[]*\[\([0-9.]*\).*/\1/p' x|sort -nr| uniq
2

using awk, and telling it fields are separated with ] or [ and that we just need the 2nd field:

awk -F'[][]'  '
     {uniqoccurences[$2]++;}
 END { for (i in uniqoccurences) {
            print i ":" uniqoccurences[i] 
        } 
     } '

In the above example I also print ":n", ie the number of occurences of each "$2", but there is no need to do that (In that case, just do print i in the loop withing the END section)

The regexp: [][] is using the way regexp treat ] and [ within a [...] character class ( a ] just after a [ is then treated as a character to look for, and a [ after the initial [ (and before a closing ]) is also treated as a character to look for. So [][] looks for ] or [)

Another way:

awk -F'[][]'  '{ print $2 }' | sort | uniq
2

A few options:

  1. GNU grep

    $ grep -oP '\(axyz-pc\) \[\K[^]]+' file | sort | uniq 
    114.225.87.41
    36.32.138.106
    36.32.138.216
    37.49.224.14
    
  2. perl

    $ perl -lne '/\(axyz-pc\) \[([^]]+)/ && ++$seen{$1}<2 && print $1' file 
    36.32.138.106
    114.225.87.41
    36.32.138.216
    37.49.224.14
    
  3. awk

    $ awk -F ') \\[' '{print $2}' file | awk -F '\\]' '!seen[$1]++{print $1}'
    36.32.138.106
    114.225.87.41
    36.32.138.216
    37.49.224.14
    
  4. sed

    $ sed -n 's/.*(axyz-pc) \[\([^]]*\).*/\1/p' file | sort | uniq
    114.225.87.41
    36.32.138.106
    36.32.138.216
    37.49.224.14
    
1
cat in.txt | awk '/SMTP/{print $7}'  

gives me this .

[36.32.138.106]:1236
[114.225.87.41]:3823
[36.32.138.216]:1984
[37.49.224.14]:51593

For the final step :

cat in.txt | awk '/SMTP/{print $7}' | sed -e 's/\[//; s/\]//; s/:...//' 

Edit : The above $7 will not work because the two kinds of lines have different field offsets for the IP address. A better way could be :

cat in.txt | awk -F "axyz-pc\) \[" '{print $2}' | awk -F"\]" '{print $1}'

We would use "axyz-pc" as the field delimiter in the first awk then pipe the output to the second awk.

Using sed instead, would be not complicated.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.