Find IP addresses in file excluding local IPs

Question

The task: find and print lines from a file that contain any (IPv4) IP address except those on the local LAN (local LAN uses class C addresses of the 192.168.1.x variety). I (having cribbed bits from various online searches) use sed and a couple of pipes to do this fairly effectively, as follows:

sed -rn '/([0-9]{1,3}\.){3}[0-9]{1,3}/p' logfile.txt | sed '/192\.168\.[[:digit:]]\.[[:digit:]]\{,3\}/d'

Question: are there other or better ways of doing this using alternate utilities, or perhaps improvements I could make to the incantation I've come up with?

Background: the purpose is, on a non-systemd (Void Linux) system, to trawl daily a system log file that contains ssh connection information to see who, other than hosts on my local LAN, has been trying to make ssh connections/queries. The file containing the target information is generated daily using svlogtail, after which the above command runs. The system then e-mails me the output.

Sample input data:

2020-06-21T08:28:04.56472 auth.err: sshd[21813]: error: Bind to port 22 on 192.168.2.16 failed: Cannot assign requested address.  
2020-06-23T11:12:59.04698 auth.info: Jun 23 06:12:59 sshd[25036]: banner exchange: Connection from 194.61.24.4 port 1565: invalid format  
2020-07-14T14:53:30.54107 auth.info: Jul 14 09:53:30 sshd[30149]: banner exchange: Connection from 31.207.47.114 port 1848: invalid format  
2020-09-06T15:25:19.32385 auth.info: Sep  6 10:25:19 sshd[18826]: banner exchange: Connection from 193.142.146.216 port 30884: invalid format  
2021-01-30T16:03:43.39251 auth.info: Jan 30 10:03:43 sshd[32208]: rexec line 80: Unsupported option GSSAPIAuthentication  
2021-02-05T12:24:30.42762 auth.info: Feb  5 06:24:30 sshd[27489]: banner exchange: Connection from 94.232.47.170 port 107: invalid format  
2021-02-19T15:48:10.29592 auth.info: Feb 19 09:48:10 sshd[2924]: Disconnected from user 192.168.1.10 port 33732

Desired output (this acceptable output is what I get after running above sed commands)

2020-06-23T11:12:59.04698 auth.info: Jun 23 06:12:59 sshd[25036]: banner exchange: Connection from 194.61.24.4 port 1565: invalid format  
2020-07-14T14:53:30.54107 auth.info: Jul 14 09:53:30 sshd[30149]: banner exchange: Connection from 31.207.47.114 port 1848: invalid format  
2020-09-06T15:25:19.32385 auth.info: Sep  6 10:25:19 sshd[18826]: banner exchange: Connection from 193.142.146.216 port 30884: invalid format  
2021-02-05T12:24:30.42762 auth.info: Feb  5 06:24:30 sshd[27489]: banner exchange: Connection from 94.232.47.170 port 107: invalid format

Later summary The most elegant way to do this sort of thing definitely seems to be grepcidr. Had I known about it prior to trying to accomplish this task I definitely would have used it. I am going to mark the response that suggests using it as the answer, even though there are mutliple ways, including the sed commands and pipes I posted, to accomplish the same goal. The grepcidr command does have much greater adaptability for these sorts of tasks, as multiple address ranges can be specified far more easily than would be the case if attempting to use sed or grep.

Btw, I've also discovered a couple of grep commands that, when piped together, also give the same result, as follows:

grep -E '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' logfile.txt | grep -v 192.168.[[:digit:]].[[:digit:]]

Answer 1

You seem to want to filter out the lines that contain an address in the 192.168.0.0/16 block. You may do this with grepcidr:

$ grepcidr -v '192.168.0.0/16' file
2020-06-23T11:12:59.04698 auth.info: Jun 23 06:12:59 sshd[25036]: banner exchange: Connection from 194.61.24.4 port 1565: invalid format
2020-07-14T14:53:30.54107 auth.info: Jul 14 09:53:30 sshd[30149]: banner exchange: Connection from 31.207.47.114 port 1848: invalid format
2020-09-06T15:25:19.32385 auth.info: Sep 6 10:25:19 sshd[18826]: banner exchange: Connection from 193.142.146.216 port 30884: invalid format
2021-02-05T12:24:30.42762 auth.info: Feb 5 06:24:30 sshd[27489]: banner exchange: Connection from 94.232.47.170 port 107: invalid format

Or, if you want to use a specific range, such as "between 192.168.1.0 and 192.168.2.255":

$ grepcidr -v '192.168.1.0-192.168.2.255' file
2020-06-23T11:12:59.04698 auth.info: Jun 23 06:12:59 sshd[25036]: banner exchange: Connection from 194.61.24.4 port 1565: invalid format
2020-07-14T14:53:30.54107 auth.info: Jul 14 09:53:30 sshd[30149]: banner exchange: Connection from 31.207.47.114 port 1848: invalid format
2020-09-06T15:25:19.32385 auth.info: Sep 6 10:25:19 sshd[18826]: banner exchange: Connection from 193.142.146.216 port 30884: invalid format
2021-02-05T12:24:30.42762 auth.info: Feb 5 06:24:30 sshd[27489]: banner exchange: Connection from 94.232.47.170 port 107: invalid format

To filter out all private IPv4 blocks, use

grepcidr -v '10.0.0.0/8,172.16.0.0/12,192.168.0.0/16' file

grepcidr should also understand IPv6, but since I don't, I won't show an example.

Answer 2

I think that grep and PCRE are the best tool for such jobs:

grep -Po "(?<=from )(\d{1,3}\.){3}\d{1,3}(?= port)" file | grep -Ev "^192\.168\.1\.[[:digit:]]{1,3}$"

Assuming your network have default C class mask /24.

A better approach should be to skip logging for the desired hosts, such as lan hosts in your case.

Furthermore the output of your logger are not ideally for parsing. A better logger, like logging module of iptables firewall show entry for parsing in a more sophisticated way:

Feb 27 00:32:33 hostname kernel: [181663.827178] [HTTPS-ATTEMPT]IN=eno1 OUT= MAC=aa:bb:cc:dd:ee:ff:ff:ee:dd:cc:bb:aa:08:00 SRC=X.X.X.X DST=X.X.X.X LEN=40 TOS=0x00 PREC=0x00 TTL=242 ID=54321 PROTO=TCP SPT=42717 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0 

If you can skip logging for local lan host then the grep could stop at the first without pipe for the second round.

@Praveen Kumar BS make me think to a good python module for doing this job: ipaddrress

#!/usr/bin/python3
import ipaddress
import re

lan = ipaddress.IPv4Network("192.168.1.0/24")

with open("stack.log", 'r') as log:
    for line in log:
        curr_ip = re.search(r'(?<=from )(\d{1,3}\.){3}\d{1,3}\b', line)
        if curr_ip and not ipaddress.IPv4Address(curr_ip.group()) in lan:
            print(curr_ip.group())

with ipaddress you can build an address from a regexp capture and check if it is in a given network, lan, in my example.

Answer 3

With GNU grep or any grep with perl-like regexp support, you can use the negative look-ahead operator:

grep -P '\b(?!192\.168\.)(\d{1,3})(\.(?1)){3}\b'

Or directly with perl:

perl -ne 'print if /\b(?!192\.168\.)(\d{1,3})(\.(?1)){3}\b/'

Those report the lines that contain a quad-decimal representation of an IP address other than those starting with 192.168., even if those lines also contain a 192.168.x.y IP address.

For a stricter matching of IP addresses, you could use the Regexp::Common::net module:

perl -MRegexp::Common=net -ne '
  print if m{
    \b$RE{net}{IPv4}{-keep}\b
    (?(?{$2 == 192 && $3 == 168}) (*FAIL))
  }x'

Answer 4

command:

awk '!/192.168.*.*/{for(i=1;i<=NF;i++){if($i ~ /^[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*$/){print $i}}}' filename

output

194.61.24.4
31.207.47.114
193.142.146.216
94.232.47.170

Python

#!/usr/bin/python
import re
k=re.compile(r'192.168.*.*')
c=re.compile(r'^[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*')
m=open('p.txt','r')
for i in m:
    if not re.search(k,i):
        z=i.split(' ')
        for v in range(0,len(z),1):
            if re.search(c,z[v]):
                print z[v]
            

output

194.61.24.4
31.207.47.114
193.142.146.216
94.232.47.170