3

The task: find and print lines from a file that contain any (IPv4) IP address except those on the local LAN (local LAN uses class C addresses of the 192.168.1.x variety). I (having cribbed bits from various online searches) use sed and a couple of pipes to do this fairly effectively, as follows:

sed -rn '/([0-9]{1,3}\.){3}[0-9]{1,3}/p' logfile.txt | sed '/192\.168\.[[:digit:]]\.[[:digit:]]\{,3\}/d'

Question: are there other or better ways of doing this using alternate utilities, or perhaps improvements I could make to the incantation I've come up with?

Background: the purpose is, on a non-systemd (Void Linux) system, to trawl daily a system log file that contains ssh connection information to see who, other than hosts on my local LAN, has been trying to make ssh connections/queries. The file containing the target information is generated daily using svlogtail, after which the above command runs. The system then e-mails me the output.

Sample input data:

2020-06-21T08:28:04.56472 auth.err: sshd[21813]: error: Bind to port 22 on 192.168.2.16 failed: Cannot assign requested address.  
2020-06-23T11:12:59.04698 auth.info: Jun 23 06:12:59 sshd[25036]: banner exchange: Connection from 194.61.24.4 port 1565: invalid format  
2020-07-14T14:53:30.54107 auth.info: Jul 14 09:53:30 sshd[30149]: banner exchange: Connection from 31.207.47.114 port 1848: invalid format  
2020-09-06T15:25:19.32385 auth.info: Sep  6 10:25:19 sshd[18826]: banner exchange: Connection from 193.142.146.216 port 30884: invalid format  
2021-01-30T16:03:43.39251 auth.info: Jan 30 10:03:43 sshd[32208]: rexec line 80: Unsupported option GSSAPIAuthentication  
2021-02-05T12:24:30.42762 auth.info: Feb  5 06:24:30 sshd[27489]: banner exchange: Connection from 94.232.47.170 port 107: invalid format  
2021-02-19T15:48:10.29592 auth.info: Feb 19 09:48:10 sshd[2924]: Disconnected from user 192.168.1.10 port 33732

Desired output (this acceptable output is what I get after running above sed commands)

2020-06-23T11:12:59.04698 auth.info: Jun 23 06:12:59 sshd[25036]: banner exchange: Connection from 194.61.24.4 port 1565: invalid format  
2020-07-14T14:53:30.54107 auth.info: Jul 14 09:53:30 sshd[30149]: banner exchange: Connection from 31.207.47.114 port 1848: invalid format  
2020-09-06T15:25:19.32385 auth.info: Sep  6 10:25:19 sshd[18826]: banner exchange: Connection from 193.142.146.216 port 30884: invalid format  
2021-02-05T12:24:30.42762 auth.info: Feb  5 06:24:30 sshd[27489]: banner exchange: Connection from 94.232.47.170 port 107: invalid format

Later summary The most elegant way to do this sort of thing definitely seems to be grepcidr. Had I known about it prior to trying to accomplish this task I definitely would have used it. I am going to mark the response that suggests using it as the answer, even though there are mutliple ways, including the sed commands and pipes I posted, to accomplish the same goal. The grepcidr command does have much greater adaptability for these sorts of tasks, as multiple address ranges can be specified far more easily than would be the case if attempting to use sed or grep.

Btw, I've also discovered a couple of grep commands that, when piped together, also give the same result, as follows:

grep -E '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' logfile.txt | grep -v 192.168.[[:digit:]].[[:digit:]]
4
  • 1
    Does this answer your question? How to filter IPv4 and IPv6 addresses? – αғsнιη Feb 26 at 20:35
  • My post seeks alternate, perhaps better, ways of doing what the commands listed do. Namely finding all lines in a file that contain ipv4 address but excluding lines containing IP's in the class C range specified. Proposals at the link are simply for finding IP's without excluding some. – MJiller Feb 26 at 20:59
  • edit your question to include concise, testable sample input and expected output so we can help you. – Ed Morton Feb 26 at 21:25
  • This is a horrible hack, but a little easier. sed -i '/192.168.*/d' ./file_name.txt - which gives you the output you're already getting in a shorter format. It removes all lines that have 192.168.* in them. – KGIII Feb 27 at 0:29
4

You seem to want to filter out the lines that contain an address in the 192.168.0.0/16 block. You may do this with grepcidr:

$ grepcidr -v '192.168.0.0/16' file
2020-06-23T11:12:59.04698 auth.info: Jun 23 06:12:59 sshd[25036]: banner exchange: Connection from 194.61.24.4 port 1565: invalid format
2020-07-14T14:53:30.54107 auth.info: Jul 14 09:53:30 sshd[30149]: banner exchange: Connection from 31.207.47.114 port 1848: invalid format
2020-09-06T15:25:19.32385 auth.info: Sep 6 10:25:19 sshd[18826]: banner exchange: Connection from 193.142.146.216 port 30884: invalid format
2021-02-05T12:24:30.42762 auth.info: Feb 5 06:24:30 sshd[27489]: banner exchange: Connection from 94.232.47.170 port 107: invalid format

Or, if you want to use a specific range, such as "between 192.168.1.0 and 192.168.2.255":

$ grepcidr -v '192.168.1.0-192.168.2.255' file
2020-06-23T11:12:59.04698 auth.info: Jun 23 06:12:59 sshd[25036]: banner exchange: Connection from 194.61.24.4 port 1565: invalid format
2020-07-14T14:53:30.54107 auth.info: Jul 14 09:53:30 sshd[30149]: banner exchange: Connection from 31.207.47.114 port 1848: invalid format
2020-09-06T15:25:19.32385 auth.info: Sep 6 10:25:19 sshd[18826]: banner exchange: Connection from 193.142.146.216 port 30884: invalid format
2021-02-05T12:24:30.42762 auth.info: Feb 5 06:24:30 sshd[27489]: banner exchange: Connection from 94.232.47.170 port 107: invalid format

To filter out all private IPv4 blocks, use

grepcidr -v '10.0.0.0/8,172.16.0.0/12,192.168.0.0/16' file

grepcidr should also understand IPv6, but since I don't, I won't show an example.

1
  • Very interesting. I was not aware of this utility but do see it in Void repositories. It does look like the ideal tool to use for a task like this. Thanks for pointing it out. – MJiller Feb 27 at 13:39
2

I think that grep and PCRE are the best tool for such jobs:

grep -Po "(?<=from )(\d{1,3}\.){3}\d{1,3}(?= port)" file | grep -Ev "^192\.168\.1\.[[:digit:]]{1,3}$"

Assuming your network have default C class mask /24.

A better approach should be to skip logging for the desired hosts, such as lan hosts in your case.

Furthermore the output of your logger are not ideally for parsing. A better logger, like logging module of iptables firewall show entry for parsing in a more sophisticated way:

Feb 27 00:32:33 hostname kernel: [181663.827178] [HTTPS-ATTEMPT]IN=eno1 OUT= MAC=aa:bb:cc:dd:ee:ff:ff:ee:dd:cc:bb:aa:08:00 SRC=X.X.X.X DST=X.X.X.X LEN=40 TOS=0x00 PREC=0x00 TTL=242 ID=54321 PROTO=TCP SPT=42717 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0 

If you can skip logging for local lan host then the grep could stop at the first without pipe for the second round.

@Praveen Kumar BS make me think to a good python module for doing this job: ipaddrress

#!/usr/bin/python3
import ipaddress
import re

lan = ipaddress.IPv4Network("192.168.1.0/24")

with open("stack.log", 'r') as log:
    for line in log:
        curr_ip = re.search(r'(?<=from )(\d{1,3}\.){3}\d{1,3}\b', line)
        if curr_ip and not ipaddress.IPv4Address(curr_ip.group()) in lan:
            print(curr_ip.group())

with ipaddress you can build an address from a regexp capture and check if it is in a given network, lan, in my example.

1
  • The grep command given in this answer captures only IP's, not surrounding context from the line containing the IP as I'd stipulated. – MJiller Feb 28 at 14:05
1

With GNU grep or any grep with perl-like regexp support, you can use the negative look-ahead operator:

grep -P '\b(?!192\.168\.)(\d{1,3})(\.(?1)){3}\b'

Or directly with perl:

perl -ne 'print if /\b(?!192\.168\.)(\d{1,3})(\.(?1)){3}\b/'

Those report the lines that contain a quad-decimal representation of an IP address other than those starting with 192.168., even if those lines also contain a 192.168.x.y IP address.

For a stricter matching of IP addresses, you could use the Regexp::Common::net module:

perl -MRegexp::Common=net -ne '
  print if m{
    \b$RE{net}{IPv4}{-keep}\b
    (?(?{$2 == 192 && $3 == 168}) (*FAIL))
  }x'
0

command:

awk '!/192.168.*.*/{for(i=1;i<=NF;i++){if($i ~ /^[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*$/){print $i}}}' filename

output

194.61.24.4
31.207.47.114
193.142.146.216
94.232.47.170

Python

#!/usr/bin/python
import re
k=re.compile(r'192.168.*.*')
c=re.compile(r'^[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*')
m=open('p.txt','r')
for i in m:
    if not re.search(k,i):
        z=i.split(' ')
        for v in range(0,len(z),1):
            if re.search(c,z[v]):
                print z[v]
            

output

194.61.24.4
31.207.47.114
193.142.146.216
94.232.47.170
1
  • This is an interesting proposal but seems to exclude important context attending the target IP's. As mentioned, I want the whole line where the IP appears, not just the IP itself. The context around the IP is important because it tells me when the attempt happened and a little about what sort of attempt was made. – MJiller Feb 27 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.