-1

I am trying to extract IPs from a huge web generated list which contains a mix of text, IPv4 & IPv6 IPs, newlines, IP ranges etc.

Below is a snippet from the list:

; Spamhaus DROP List 2016/07/03 - (c) 2016 The Spamhaus Project
; http://www.spamhaus.org/drop/drop.txt
; Last-Modified: Sun,  3 Jul 2016 21:18:32 GMT
; Expires: Sun, 03 Jul 2016 23:26:45 GMT

1.0.1.0/24
223.223.176.0
129.130.100.100
1.160.118.30
91.121.120.228 # 2016-07-05, ns350944.ip-91-121-120.eu, FRA, 1                        
62.210.111.59 # 2016-07-05, sender9p2.offresduweb.fr, FRA, 1                            
52.90.253.169 # 2016-07-05, ec2-52-90-253-169.compute-1.amazonaws.com, USA, 13                  
2a01:4f8:200:2153::2 # 2016-06-27, 2a01:4f8:200:2153::2, DEU, 2                                
2601:1c1:8801:618c:9864:3f33:7569:38c4  # 2016-06-28, 2601:1c1:8801:618c:9864:3f33:7569:38c4, USA, 2
#last updated 2016.07.04 1733 UTC

1.0.1.0/24  China
1.0.2.0/23  China
1.0.8.0/21  China
1.0.32.0/19  China
1.1.0.0/24  China

For a bigger snippet, please see pastebin This is not the full list btw, as the actual one has over 44k lines.

What I'm trying to do is to get only regular IPs (IPv4) from the list.

Here is what I ran against the above:

grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}'

and it gives me:

1.0.1.0
223.223.176.0
129.130.100.100
1.160.118.3
198.55.103.144
etc
etc

Now, this is fine but I don't want anything that ends in '.0' ( eg 1.0.1.0 or 223.223.176.0) as they are IP ranges and not actual IPs. So I piped the output of the above grep to an awk statement which strips off all the IPs ending in 0.

The grep (IPs) | awk (remove those that end in 0) solution works, but I'd like to know if there is a better way to do it and minimize the use of multiple piped greps (or sed/awk).

  • 2
    Note that some hosts do have addresses that end in .0 . What I think you don't want is the network blocks, i.e. the ones with a "/n" after them, they specify networks rather than hosts. For example the entry 223.223.176.0 in that sample specifies a single host (which as it turns out, doesn't legitimately exist). – MAP Jul 5 '16 at 10:39
1

You can do the entire task with awk (assumes the pathname, of course):

#!/usr/bin/awk -f

/^[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\.[1-9][0-9]*$/ {
        print;
        next;
}
/^[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\.[1-9][0-9]*[^0-9\.:].*$/ {
        sub("[^0-9.].*$","");
        print;
}

The first pattern matches just an IPv4 (no following text), and the second allows for some other text to match (and excludes the lines with colons).

By the way, the patterns should be anchored using "^" and "$" to skip unwanted matches.

This is shown as a script, which could then be run just like any other command (such as a pipe with grep):

./foo <foo.in

gives

129.130.100.100
1.160.118.30
91.121.120.228
62.210.111.59
52.90.253.169

I split the match into two expressions to simplify handling stray text after the IP-address. That range [^0-9:\.:] ensures that there is at least one stray character to deal with.

An awk program need not be a script, but is free-format (and the line-breaks could be discarded when making a single command-string). However, the result would be hard to read as a single-line.

Unlike the suggestions to use the -o option of grep -E or the -E option of sed, this awk solution should work on any POSIX system.

For reference (POSIX):

  • Your awk loses on the lines that have comments. Probably just add ([\w]*#.*)? before the $. – MAP Jul 5 '16 at 10:46
  • @Thomas: Thank you. I ran this against the pastebin list which I posted above and it runs fine. The only thing it did not pick up was 223.223.176.0 – Dave Jul 5 '16 at 11:20
  • @MAP: The ones with comment came out fine, eg: 36.77.166.158 in the pastebin list. Not sure which one you are referring to? Thanks. – Dave Jul 5 '16 at 11:21
  • @Thomas: 223.223.176.0 appears to be network block and not an IP, as per the source web list: 223.223.176.0/20 Sorry for the confusion! – Dave Jul 5 '16 at 11:28
  • @Thomas: Thank you. On a set with 44K lines, it took just under a sec to process the list. – Dave Jul 6 '16 at 0:17
1

Just specify that you don't allow 0s at the end in your regex:

$ grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[1-9][0-9]*' file 
129.130.100.100
1.160.118.30
91.121.120.228
62.210.111.59
52.90.253.169

The trick is the \.[1-9][0-9]*, which means match a ., then any number greater than 0 once (you can't have IPs ending in 019 or similar numbers) and then 0 or more numbers from 0 to 9.

I would also use grep -E to simplify the syntax:

grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[1-9][0-9]*' file 

Or, even simpler:

grep -Eo '([0-9]{1,3}\.){3}[1-9]\d*' file 

And, if your grep supports it, grep -P to simplify further:

grep -Po '(\d{1,3}\.){3}[1-9]\d*' file 
  • Thank you. Interesting to note that on a line which had this entry: 5.9.83.211 # 2016-07-03, static.211.83.9.5.clients.your-server.de, DEU, 11 , it pulled both 5.9.83.211 and 211.83.9.5 – Dave Jul 6 '16 at 0:20
  • @Dave if the IPs will always be at the beginning of the line, you can use grep -Po '^(\d{1,3}\.){3}[1-9]\d*' file to avoid that. – terdon Jul 6 '16 at 0:28
  • Thanks, that fixed it. I'm trying to understand your regex at the end which omits 0s. .[1-9][0-9]* . So that means it will match 10-99*? – Dave Jul 6 '16 at 3:11
  • @Dave no, it will match one digit between 1 and 9 ([1-9]) then 0 or more digits between 0 and 9 ([0-9]*), effectively matching 1 to infinity. – terdon Jul 6 '16 at 8:52
1
$ sed -E -e 's/[[:space:];#\/].*//;
             /\.0$+|[0-9a-f]{1,4}:|^[[:space:]]*$/d' spamhaus.txt 
129.130.100.100
1.160.118.30
91.121.120.228
62.210.111.59
52.90.253.169

(newline and indentation added to improve readability)

  • remove comments and everything from the first whitespace on a line (i.e. replace with the empty string)
  • delete lines containing:
    • .0 followed by a slash or end-of-line
    • 1-4 hex digits followed by a `:'
    • empty and whitespace-only lines
  • print everything else.

Same algorithm in perl:

perl -lne 's/[[:space:];#\/].*//;
           next if (m/\.0$|[0-9a-f]{1,4}:|^\s*$/o);
           print'

Output of a timing test script that ran each method 10 times in a row, using the full files downloaded from their respective hosts:

$ ./timing.sh 
input file sizes:
24K drop.txt
72K base_90days.txt
120K    sinokoreacidr.txt
216K    total

input file line count:
   793 drop.txt
  4997 base_90days.txt
  5400 sinokoreacidr.txt
 11190 total

tdickey.awk: real 0m0.367s  user 0m0.305s   sys 0m0.027s
terdon.grep: real 0m0.550s  user 0m0.514s   sys 0m0.029s
cas.sed    : real 0m0.531s  user 0m0.484s   sys 0m0.035s
cas.perl   : real 0m0.379s  user 0m0.341s   sys 0m0.036s

output line counts:
  4990 out.cas.perl
  4990 out.cas.sed
  4990 out.tdickey.awk
  4990 out.terdon.grep

output differences (if any):

(BTW, the timing.sh test script found a bug in my original sed script. some lines were being printed with the trailing /CIDR. fixed)

The all produced exactly the same output, which is good :)

I ran this multiple times, on an AMD Phenom II 1090T. The sed and grep versions had relatively stable timings with almost no difference between runs, a ms or two at the most.

The awk and perl versions had slightly greater variation between runs - up to 20ms or so...almost always within a few ms of each other. Sometimes perl was slightly faster, usually awk was slightly faster. probably because my system runs lots of other stuff at the same time.

On this CPU, given the tiny amounts of time to run any of these versions, there's no significant difference between them. On a slower CPU, the differences may be more significant. I've included the timing script below so you can test on your own system:

#!/bin/bash

export TIMEFORMAT=$'real %3lR\tuser %3lU\tsys %3lS'
files=(drop.txt base_90days.txt sinokoreacidr.txt)

function timetest() {
  # first arg is title string, remaining args are executed.

  # prime the cache
  cat "${files[@]}" > /dev/null

  title="$1" ; shift
  printf '%-11s' "$title" >&2

  # 10 runs for each
  time for i in {1..10} ; do
    "$@" "${files[@]}" > "out.$title"
  done
  # unique sort the output, but don't include sort in timings
  sort -u "out.$title" > "out.tmp" ; mv -f out.tmp "out.$title"
}

echo 'input file sizes:'
du -sch "${files[@]}"
echo
echo 'input file line count:'
wc -l "${files[@]}"
echo

rm -f out.*

timetest tdickey.awk ./tdickey.awk
timetest terdon.grep grep -h -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[1-9][0-9]*'
timetest cas.sed sed -E -e 's/[[:space:];#\/].*//; /\.0$|[0-9a-f]{1,4}:|^[[:space:]]*$/d'
timetest cas.perl perl -lne 's/[[:space:];#\/].*//; next if (m/\.0$|[0-9a-f]{1,4}:|^\s*$/o); print'

echo
echo "output line counts:"
wc -l out.* | grep -v total

# check if they all produce exactly the same output
echo
echo "output differences (if any):"
diff -u out.cas.sed out.cas.perl
diff -u out.cas.sed out.tdickey.awk
diff -u out.cas.sed out.terdon.grep
  • Thank you. This works well, but is slightly slower compared to the grep or awk solutions above. – Dave Jul 6 '16 at 22:21
  • BTW, what kind of system/CPU do you have that it takes just under a second to process 44KB? all versions took ~ 30-60ms for an individual run against 216KB of files on my AMD 1090T. effectively instantaneous. – cas Jul 7 '16 at 2:46
  • I'm running the scripts on an Edgerouter. There is some additional sorting/filtering done after the initial extract as well, which is probably why is takes that time. And thank you for your detailed explanation and test script. – Dave Jul 11 '16 at 3:43
  • Ran this on the router: 24.0K drop.txt 72.0K base_90days.txt 120.0K sinokoreacidr.txt 216.0K total input file line count: 792 drop.txt 4984 base_90days.txt 5401 sinokoreacidr.txt 11177 total tdickey.awkreal 0m1.345s terdon.grepreal 0m1.150s cas.sed real 0m1.681s cas.perl real 0m5.778s output line counts: 4978 out.cas.perl 4978 out.cas.sed 4978 out.tdickey.awk 4978 out.terdon.grep – Dave Jul 11 '16 at 3:56
  • Highest time was the perl @ 5.778s . Then the sed @ 1.681s . Grep was the quickest @ 1.150s. The Edgerouter has a MIPS processor btw. – Dave Jul 11 '16 at 3:59
0

Well, just add an appropriate pattern to your regular expression. Instead of [0-9]\{1,3\}, you want something like [0-9]*[1-9] at the end of your expression.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.